Pedestrian detection method, device, medium and product for top-mounted fisheye camera

Abstract

This invention discloses a method, apparatus, medium, and product for pedestrian detection using a top-mounted fisheye camera. The method includes: inputting an image to be detected acquired by the top-mounted fisheye camera into a first processing module for feature extraction; the first processing module splits the input channel of the feature map into a first subset and a second subset; performing conventional convolution on the first subset and dilated convolution on the second subset; and merging the results to output a fused feature image; inputting the fused feature image into a target detection network containing a P2 detection head, outputting pedestrian detection targets of different sizes; performing Transformer processing on the pedestrian detection targets through a second processing module to obtain global features; and outputting the position information of each pedestrian detection target based on the global features. This invention solves the problems of low pedestrian detection recall rates caused by difficulties in detecting small targets, insufficient edge receptive fields, and head-to-foot disproportion in existing fisheye distortion-reduced images.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, medium, and product for pedestrian detection using a top-mounted fisheye camera. Background Technology

[0002] In scenarios such as indoor surveillance, customer flow analysis, and smart retail, top-mounted fisheye cameras are widely used due to their advantages such as wide field of view, concealed installation, and blind-spot-free monitoring. However, fisheye images suffer from severe radial distortion, which usually requires distortion correction processing to obtain a perspective projection image that is more intuitive for the human eye. After distortion correction processing, pedestrians located at the edges and corners of the image will appear severely stretched, distorted, and tilted, no longer appearing as standard upright figures, but rather with disproportionate head-to-toe ratios and varying orientations.

[0003] Existing technologies for pedestrian detection using top-mounted fisheye cameras mainly face the following technical challenges: First, there's the issue of missed detections due to insufficient receptive fields for pedestrians at the edges. In fisheye distortion-resolved images, pedestrians at the edges are severely stretched, requiring a larger receptive field to effectively cover the entire pedestrian, while pedestrians in the central area do not require such a large receptive field. Existing detection networks use convolutional designs with fixed receptive fields, which cannot simultaneously meet the detection needs of pedestrians in different areas, leading to the easy missed detection of stretched pedestrians at the edges.

[0004] Second, there is the problem of detecting small pedestrians. Top-mounted fisheye cameras are typically installed at heights between 1.9 meters and 6 meters, and the size difference of pedestrians in the image can be several times or even tens of times. Small pedestrians at a distance occupy only a very small pixel area on the feature map. Existing detection networks have a limited number of feature pyramid layers, resulting in a low detection rate for small targets.

[0005] Third, there is the issue of missed detections due to head-to-foot disproportion. In fisheye distortion-corrected images, the head-to-foot ratio of pedestrians is severely disproportionate, with the head and feet being far apart in the image. Conventional convolutional neural networks have difficulty covering the head and foot regions simultaneously and establishing the correlation features between them, resulting in a low recall rate for the detector when detecting pedestrians with severe distortion. Summary of the Invention

[0006] The embodiments of the present invention provide a pedestrian detection method, device, medium and product for a top-mounted fisheye camera, which aims to solve the problems of low pedestrian detection recall rate caused by the difficulty in detecting small targets, insufficient edge receptive field and head-to-foot ratio imbalance in existing fisheye distortion-reduced images.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a pedestrian detection method for a top-mounted fisheye camera, comprising the following steps: The image to be inspected, captured by the top-mounted fisheye camera, is input into a predetermined first processing module for feature extraction. The first processing module splits the input channel of the feature map into a first subset and a second subset according to the channel dimension. Perform conventional convolution on the first subset, perform dilated convolution on the second subset, and merge the two convolution results to output a fused feature image; The fused feature image is input into a predetermined target detection network, which outputs pedestrian detection targets of different sizes; the target detection network includes a P2 detection head. The pedestrian detection target is processed by Transformer through a predetermined second processing module to obtain the global features of the pedestrian detection target; The location information of each pedestrian detection target is obtained based on the global features.

[0008] Furthermore, the first subset and the second subset have the same number of channels.

[0009] Furthermore, the target detection network also includes a P3 detection head, a P4 detection head, and a P5 detection head.

[0010] Furthermore, the target detection network includes a classification branch and a regression branch, the second processing module is a Transformer module, the Transformer module is added to the regression branch, and the classification branch uses a CNN structure to classify the pedestrian detection target.

[0011] Furthermore, it also includes a third processing module. During the training phase, the third processing module splits the input channels into a third subset and a fourth subset according to the channel dimension, performs traditional convolution on the third subset, performs high-efficiency convolution with added noise interference on the fourth subset, and merges the two convolution results for output. During the inference phase, the third processing module performs high-efficiency convolution on the fourth subset without adding noise interference.

[0012] Furthermore, the location information includes the center coordinates, width, height, and rotation angle around the center point of the detection frame where the pedestrian detection target is located.

[0013] Secondly, the present invention provides a pedestrian detection device for a top-mounted fisheye camera, comprising a memory and a processor, wherein the memory stores at least one program, which is executed by the processor to implement the pedestrian detection method for a top-mounted fisheye camera as described above.

[0014] Thirdly, the present invention provides a computer-readable storage medium storing at least one program that is executed by a processor to implement the pedestrian detection method for a top-mounted fisheye camera as described above.

[0015] Fourthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the pedestrian detection method for a top-mounted fisheye camera as described above.

[0016] The above technical solution has the following technical effects: The invention employs a first processing module to extract features from images captured by a top-mounted fisheye camera. This module splits the input channels of the feature map into a first subset and a second subset. Traditional convolution is performed on the first subset, and dilated convolution is performed on the second subset. The resulting merged feature image is then input into a target detection network containing a P2 detection head, which outputs pedestrian detection targets of different sizes. A second processing module then performs Transformer processing on the pedestrian detection targets to obtain global features. Based on these global features, the location information of each pedestrian detection target is output. This invention addresses the problem of insufficient receptive field for edge pedestrians through channel splitting and parallel convolution, solves the problem of difficulty in detecting small targets through the P2 detection head, and addresses the problem of missed detections caused by head-to-foot disproportion through the Transformer. This effectively improves the recall rate and reduces the false detection rate of pedestrian detection in fisheye distortion-free images. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a pedestrian detection method using a top-mounted fisheye camera according to an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of a pedestrian detection device for a top-mounted fisheye camera according to an embodiment of the present invention. Detailed Implementation

[0019] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments and to explain the operating principles of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0020] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0021] Example 1: Figure 1 This is a flowchart illustrating a pedestrian detection method using a top-mounted fisheye camera according to an embodiment of the present invention. Figure 1 As shown, this embodiment provides a pedestrian detection method for a top-mounted fisheye camera, including the following steps: First, an image of the indoor scene to be inspected is acquired using a top-mounted fisheye camera. The installation height of the top-mounted fisheye camera ranges from 1.9 meters to 6 meters, with the specific installation height determined based on the actual application scenario. Due to severe radial distortion in fisheye images, this embodiment performs distortion correction processing on the acquired fisheye images to obtain a distortion-corrected image. The distortion correction processing employs fisheye image correction algorithms, such as polynomial fitting-based correction methods or spherical projection-based correction methods.

[0022] In the distortion-reduced image, pedestrians located at the edges and corners of the image will be severely stretched, distorted, and tilted. These pedestrians no longer appear in a standard upright position, but rather exhibit varying degrees of tilting, stretching, and even distortion, with severely disproportionate head-to-foot ratios, posing a challenge to subsequent pedestrian detection.

[0023] The distortion-corrected image is input into a predefined first processing module (HalfDilateConv module) for feature extraction. The first processing module employs a convolutional neural network structure to extract pedestrian features from the image.

[0024] To accommodate the varying receptive field requirements of pedestrians in different regions of fisheye distortion-corrected images—that is, pedestrians stretched at the edges require a larger receptive field, while pedestrians in the center require a smaller receptive field—the first processing module in this embodiment employs a channel-split parallel convolution design. This design transforms traditional convolution by channel into a combination of half traditional convolution and half dilated convolution.

[0025] Specifically, the first processing module splits the channels of the input feature map into a first subset and a second subset according to the channel dimension. The input feature map is an intermediate feature representation obtained after several layers of convolution, and its number of channels is usually 64, 128, or 256, etc. The first subset and the second subset have the same number of channels, that is, each accounts for half of the channels. For example, if the input feature map has 64 channels, then the first subset contains 32 channels, and the second subset contains 32 channels.

[0026] Perform traditional convolution operations on the first subset. Traditional convolution uses a standard convolution kernel, such as 3×3, to extract features within a local neighborhood. It has a small receptive field and is suitable for pedestrians with relatively normal morphology in the central region of the image.

[0027] Dilated convolution is performed on the second subset. Dilated convolution expands the receptive field by inserting holes (zero values) between the kernel elements. Its dilation rate parameter d can be configured according to the degree of stretching of pedestrians at the edges of the fisheye-dedistorted image. Dilated convolution can expand the receptive field without increasing the number of parameters and computation, thus effectively covering the entire body of the stretched pedestrian in the edge region.

[0028] The traditional convolution results of the first subset and the dilated convolution results of the second subset are concatenated along the channel dimension to output a fused feature image. This fused feature image contains both local detail features from a small receptive field and global contextual features from a large receptive field, enabling the network to adaptively process pedestrian targets at different locations and with different degrees of deformation in the image.

[0029] In this embodiment, the HalfDilateConv module is specifically used at feature maps downsampled by 16 times and 32 times, replacing the original traditional convolution module, giving the feature maps a larger receptive field, increasing the detection of large and super-large targets, while not affecting the detection of shallow small targets.

[0030] The fused feature image is input into a predetermined object detection network. The object detection network employs an Oriented Bounding Box (OBB) detection network based on the YOLOv8 architecture to output pedestrian detection targets of different sizes.

[0031] In the standard configuration of YOLOv8, three detection heads are typically used: P3, P4, and P5, with downsampling magnifications of 8x, 16x, and 32x, respectively. However, since the installation height of the top-mounted fisheye camera ranges from 1.9 meters to 6 meters, the size of pedestrians in the image varies drastically, requiring the simultaneous detection of both large and small targets. The standard P3 detection head (8x downsampling) is still not precise enough for small pedestrians at a distance.

[0032] To address this, the object detection network in this embodiment adds a P2 detection head. The P2 detection head has a 4x downsampling rate, performing detection on a larger feature map, which significantly improves the detection rate of small pedestrians. For an input image with a resolution of 640×640, the P2 feature map size is 160×160, effectively preserving the detailed features of small pedestrians at a distance. Thus, the object detection network's detection heads have been changed from the standard YOLO three detection heads (P3, P4, P5) to four detection heads (P2, P3, P4, P5).

[0033] The four detection heads, P2, P3, P4, and P5, together form a feature pyramid, which is used to detect extremely small, medium-sized, medium-sized, and large targets, respectively, thus realizing multi-scale pedestrian detection.

[0034] Each head of the object detection network includes a classification branch and a regression branch. The classification branch uses a CNN structure to classify pedestrian detection targets, that is, to determine whether the detection area is a pedestrian and output the class probability.

[0035] The regression branch outputs the location information of the detected pedestrian. In a conventional YOLOv8 OBB detection network, the regression branch uses a CNN to output a quintuple of parameters. However, for fisheye distortion-corrected images, due to the severe imbalance in the head-to-foot ratio of pedestrians, the head and feet are far apart in the image. The local receptive field of a conventional CNN cannot simultaneously cover both the head and feet, causing the detector to be unable to establish a correlation between the head and feet, which easily leads to missed detections.

[0036] Therefore, this embodiment includes a second processing module in the regression branch, which is a Transformer module. The Transformer module is added to the regression branch, replacing the original OBB detection module. The input to the Transformer module is the feature vector output by each detector head.

[0037] The Transformer module employs a self-attention mechanism to capture long-distance pixel relationships between different parts of a pedestrian detection target. Specifically, through multi-head self-attention computation, pixels at the head location and pixels at the foot location in the feature map are directly associated, thereby obtaining global features of the pedestrian detection target. This ensures the model has complete feature responses regardless of orientation, effectively solving the problem of missed detections caused by head-to-foot ratio imbalance. Meanwhile, the classification branch maintains the CNN structure unchanged.

[0038] The global features output by the second processing module are enhanced feature vectors, which contain long-distance correlation information between different parts of a pedestrian's body, such as the head and feet. This feature vector is input into the fully connected layer of the regression branch, and through linear mapping, five regression parameters are output: the x-coordinate (cx), y-coordinate (cy), width (w), height (h), and the rotation angle (angle) of the longer side relative to the horizontal axis of the center point of the rotated detection box. Based on these five-tuple parameters, the coordinates of the four vertices of the rotated rectangle can be further calculated to characterize the precise position and orientation of the pedestrian detection target in the image. Based on the global features output by the second processing module, the position information of each pedestrian detection target is output. The position information uses an Oriented Bounding Box (OBB) format, including the center coordinates, width, height, and rotation angle (cx, cy, w, h, angle) of the detection box containing the pedestrian detection target.

[0039] Where cx and cy are the x and y coordinates of the center point of the rotation detection box, w and h are the width and height of the rotation detection box, and angle is the rotation angle of the longer side of the rotation detection box relative to the horizontal axis. Based on the above quintuple parameters, the coordinates of the four vertices of the rotation rectangle can be calculated using the rotation matrix: First, calculate the coordinates of the four vertices of the rectangle before rotation: P1(x1,y1)=(cx-w / 2, cy-h / 2), P2(x2,y2)=(cx+w / 2, cy-h / 2), P3(x3,y3)=(cx+w / 2, cy+h / 2), P4(x4,y4)=(cx-w / 2, cy+h / 2), Then, a rotation matrix is ​​applied for rotation. Taking P1(x1,y1) as an example, the point (x1,y1) is rotated by an angle angle around the center point (cx,cy), and the rotated coordinates (x1_rot, y1_rot) are calculated using the following formula: x1_rot = cos(angle)·(x1-cx) - sin(angle)·(y1-cy) + cx, y1_rot = sin(angle)·(x1-cx) + cos(angle)·(y1-cy) + cy, The coordinates of the four vertices can be obtained in the same way and used for visualization or subsequent processing.

[0040] This embodiment also includes a third processing module. The purpose of the third processing module is to improve the generalization ability of the model and reduce the false detection rate by adding noise interference during the training phase, while removing noise during the inference phase to maintain detection accuracy and reduce computational load.

[0041] During the training phase, the third processing module splits the input channels into a third subset and a fourth subset along the channel dimension. Traditional convolution is performed on the third subset, while high-efficiency convolution with added noise interference is performed on the fourth subset. The two convolution results are then combined and output. High-efficiency convolution refers to convolution operations that reduce computation and parameter count while maintaining the same kernel size, using techniques such as grouped convolution. The added noise interference can be Gaussian noise or random deactivation noise. By adding noise interference to the high-efficiency convolution, the computational cost and parameter count of the model are reduced, while the model's generalization ability is improved, its detection rate is increased, and its false detection rate is reduced.

[0042] During the inference phase, the third processing module performs efficient convolutions on the fourth subset without adding noise interference. Removing the noise module during inference allows the model to maintain high detection accuracy while further reducing computational and parameter requirements, facilitating real-time inference on embedded devices.

[0043] In this embodiment, the system successfully achieved high-precision detection of pedestrians in the distortion-free images from the top-mounted fisheye camera.

[0044] The HalfDilateConv module, a channel-split parallel convolution in the first processing module, solves the problem of missed detections caused by insufficient receptive field for pedestrians at the edge. This module is used at feature maps downsampled by 16x and 32x, giving the feature maps a larger receptive field, increasing the detection of large and super-large targets, while not affecting the detection of small targets in shallow layers.

[0045] By employing a multi-scale target detection network incorporating a P2 detector head, the problem of detecting small pedestrians was solved. Since the camera installation height ranges from 1.9 meters to 6 meters, the size of pedestrians in the image varies drastically. The P2 detector head, through its high-resolution feature map with 4x downsampling, significantly improves the recall rate of small targets.

[0046] The TransOBB module, a global feature capture module of the second processing module Transformer, solves the problem of missed detections caused by head-to-foot disproportion. The Transformer module captures the relationship between distant pixels through a self-attention mechanism to obtain global features, allowing the model to have complete features in any direction, effectively adapting to the severe head-to-foot disproportion of pedestrians in fisheye-corrected images.

[0047] The training / inference differentiation design of the third processing module (HalfNoiseConv module) improves the model's generalization ability and reduces the false detection rate by adding noise interference during training; and removes noise during inference to reduce the computational load, making it easier to perform real-time inference on embedded devices.

[0048] The four technical features mentioned above, such as HalfDilateConv, P2 detection head, TransOBB, and HalfNoiseConv, work together to significantly improve the recall rate of pedestrian detection, reduce the false detection rate, and meet the real-time inference requirements of embedded devices.

[0049] This solution can be deployed in applications such as indoor monitoring, passenger flow analysis, and smart retail, and can also be integrated into embedded devices to achieve real-time pedestrian detection.

[0050] Example 2: Figure 2 This is a schematic diagram of the pedestrian detection device for a top-mounted fisheye camera according to an embodiment of the present invention, as shown below. Figure 2 As shown, the device includes a processor 201, a memory 202, a bus 203, and a computer program stored in the memory 202 and executable on the processor 201. The processor 201 includes one or more processing cores. The memory 202 is connected to the processor 201 via the bus 203. The memory 202 is used to store program instructions. When the processor executes the computer program, it implements the steps in the above-described method embodiment of Embodiment 1 of the present invention.

[0051] Furthermore, as an executable solution, the pedestrian detection device for the top-mounted fisheye camera can be a computer unit, which can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer unit may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described structure of the computer unit is merely an example and does not constitute a limitation on the computer unit. It may include more or fewer components, or combine certain components, or use different components. For example, the computer unit may also include input / output devices, network access devices, buses, etc., and this embodiment of the invention does not limit this.

[0052] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, connecting various parts of the entire computer unit via various interfaces and lines.

[0053] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0054] Example 3: The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the embodiments of the present invention.

[0055] If the modules / units integrated in the computer unit are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0056] Example 4: The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the pedestrian detection method for a top-mounted fisheye camera as described above.

[0057] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A pedestrian detection method for a top-mounted fisheye camera, characterized in that, Includes the following steps: The image to be inspected, captured by the top-mounted fisheye camera, is input into a predetermined first processing module for feature extraction. The first processing module splits the input channel of the feature map into a first subset and a second subset according to the channel dimension. Perform conventional convolution on the first subset, perform dilated convolution on the second subset, and merge the two convolution results to output a fused feature image; The fused feature image is input into a predetermined target detection network, which outputs pedestrian detection targets of different sizes; the target detection network includes a P2 detection head. The pedestrian detection target is processed by Transformer through a predetermined second processing module to obtain the global features of the pedestrian detection target; The location information of each pedestrian detection target is obtained based on the global features.

2. The pedestrian detection method for a top-mounted fisheye camera according to claim 1, characterized in that, The first subset and the second subset have the same number of channels.

3. The pedestrian detection method for a top-mounted fisheye camera according to claim 1, characterized in that, The target detection network also includes a P3 detection head, a P4 detection head, and a P5 detection head.

4. The pedestrian detection method for a top-mounted fisheye camera according to claim 1, characterized in that, The target detection network includes a classification branch and a regression branch. The second processing module is a Transformer module, which is added to the regression branch. The classification branch uses a CNN structure to classify the pedestrian detection target.

5. The pedestrian detection method for a top-mounted fisheye camera according to claim 1, characterized in that, It also includes a third processing module. During the training phase, the third processing module splits the input channels into a third subset and a fourth subset according to the channel dimension, performs traditional convolution on the third subset, performs high-efficiency convolution with added noise interference on the fourth subset, and merges the two convolution results for output. During the inference phase, the third processing module performs efficient convolution on the fourth subset without adding noise interference.

6. The pedestrian detection method for a top-mounted fisheye camera according to claim 1, characterized in that, The location information includes the center coordinates, width, height, and rotation angle around the center point of the detection frame where the pedestrian detection target is located.

7. The pedestrian detection method for a top-mounted fisheye camera according to claim 1, characterized in that, The installation height of the top-mounted fisheye camera ranges from 1.9 meters to 6 meters.

8. A pedestrian detection device for a top-mounted fisheye camera, characterized in that, The method includes a memory and a processor, the memory storing at least one program, which is executed by the processor to implement the pedestrian detection method for a top-mounted fisheye camera as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, which is executed by a processor to implement the pedestrian detection method for a top-mounted fisheye camera as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the pedestrian detection method for a top-mounted fisheye camera as described in any one of claims 1-7.

Images