Pedestrian detection method and device based on video sequence, and computer storage medium

Abstract

The present invention provides a method and device for pedestrian detection based on video sequences, as well as a computer storage medium. The method comprises extracting moving target regions from a video sequence and defining them as moving candidate frames; concatenating moving candidate frames of the same target in different sequences to establish a target ID; performing pedestrian detection on moving candidate frames with successfully established target IDs in turn, regressing the pedestrian target position and defining it as a pedestrian frame; setting the target ID of a successfully detected pedestrian as an active target, and setting the target ID of an undetected pedestrian as a dormant target; and inferring the position of pedestrians in other frames using the relative position of the active target and the pedestrian in the detected frame within the moving candidate frame. High-quality candidate regions are screened from the spatial and temporal domains using motion information and target tracking information in the video sequence, respectively, allowing pedestrian targets to be located with minimal computational and memory overhead. This method addresses the problems of existing pedestrian detection methods, such as high computational complexity, poor detection result reliability, and low detection efficiency, that plague existing methods.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent video analysis, and in particular to a pedestrian detection method and device based on video sequences, and a computer storage medium. Background Art

[0002] With the advancement of science and technology, the application demand for intelligent video analysis is becoming increasingly strong. Among them, pedestrian detection technology plays an important role in application scenarios such as video surveillance and unmanned driving. Pedestrian detection technology (Pedestrian Detection) uses computer vision technology to determine whether there are pedestrians in an image or video sequence and accurately locate them. This technology can be combined with technologies such as pedestrian tracking and pedestrian re-identification, and is applied to artificial intelligence systems, vehicle-assisted driving systems, intelligent robots, intelligent video surveillance, human behavior analysis, intelligent transportation and other fields. However, since pedestrians have the characteristics of both rigid and flexible objects, their appearance is easily affected by clothing, scale, occlusion, posture and viewing angle. Therefore, extremely high requirements are placed on the robustness and reliability of the detection method, making pedestrian detection a hot topic in the field of computer vision that is both valuable and challenging.

[0003] Currently, the academically recognized, highly accurate convolutional neural network pedestrian detection method requires a significant amount of computation, making it difficult to strike a balance between detection accuracy and computational cost. Some embedded video processing devices only offer limited computing power and memory. To conserve computing resources, leveraging information from previous and subsequent frames in a video sequence can more effectively improve pedestrian detection efficiency. Based on this, the most commonly used method currently uses a motion detection algorithm to extract moving areas, then concentrating high-quality computing power on pedestrian detection within the moving areas of the current frame. However, due to the shadows of moving targets and the adhesion of multiple targets, the candidate areas obtained by motion detection are of poor quality, making it difficult to accurately locate the target. Light changes and minor disturbances in real scenes can also make the moving areas fragmented, forcing the use of additional computing power to detect these redundant areas.

[0004] Therefore, the current pedestrian detection technology does not propose a reliable and efficient pedestrian detection method to address the above problems. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and device for pedestrian detection based on video sequences, and a computer storage medium, so as to at least solve the problems of large computational complexity, poor reliability of detection results and low detection efficiency in existing pedestrian detection methods.

[0006] To solve the above technical problems, the present invention provides a pedestrian detection method based on video sequences, the pedestrian detection method comprising:

[0007] S1, extract the moving target area in the video sequence and define the extracted moving target area as a motion candidate frame;

[0008] S2, concatenates the motion candidate frames of the same target in different sequences to establish the target ID;

[0009] S3, pedestrian detection is performed on the motion candidate frames that have successfully established the target ID at a lower frame rate, and the pedestrian target position is regressed, and the obtained pedestrian target position is defined as the pedestrian frame;

[0010] S4, setting the target ID for which a pedestrian is successfully detected as an active target, and setting the target ID for which a pedestrian is not detected as a dormant target;

[0011] S5, the relative positions of the activated target and the pedestrians in the detected frame in the motion candidate frame are used to infer the positions of pedestrians in other frames, and the dormant target waits for the next detection.

[0012] Optionally, in the video sequence-based pedestrian detection method, step S1 performs operations in a YUV color space, including:

[0013] Acquire video images;

[0014] Downsample the video image to the target size and convert it to YUV color space;

[0015] Use the Sobel operator to calculate the horizontal gradient P of the Y component of the video image x and longitudinal gradient P y , and using the lateral gradient P x and longitudinal gradient P y The gradient image G is calculated, where

[0016] The Sobel operator is:

[0017]

[0018] The gradient image calculation formula is:

[0019]

[0020] The moving background frame in GUV format is generated by sliding average, where the calculation formula of the moving background frame is:

[0021] G b,t =(1-α)G b,t-1 +αG t

[0022] U b,t =(1-α)U b,t-1 +αU t

[0023] V b，t =(1-α)V b，t -1+αV t

[0024] Among them, α is the weight factor, t is the detected frame;

[0025] Obtain the difference image D of the tth frame t , difference graph D t The calculation formula is:

[0026] D t =|G t -G b，t-1 |+|U t -U b，t-1 |+|V t -V b，t-1 |

[0027] According to the threshold T n Difference graph D t Perform binarization to obtain the mask image M t , mask image M t The points with value "1" constitute the foreground area, and the points with value "0" constitute the background area. The formula for binarization is:

[0028]

[0029] Among them, m t The mask image M t The pixel value in d t is the difference graph D t The pixel value in ;

[0030] Mask image M t Perform morphological processing and connectivity analysis to obtain motion candidate boxes.

[0031] Optionally, in the pedestrian detection method based on video sequence, the threshold T n The methods for obtaining include:

[0032] When performing pedestrian detection on the nth motion candidate frame, save the difference map D of this frame t ;

[0033] When a pedestrian is detected in the nth motion candidate frame, the current binarization optimal threshold T is calculated o，n :

[0034]

[0035] Among them, R m is the motion candidate box, R p is a set of pedestrian frames;

[0036] Use the weight β to update the binary threshold to obtain the threshold T n :

[0037]

[0038] Optionally, in the video sequence-based pedestrian detection method, if the video image is in RGB format, then before downsampling the video image, step S1 further includes:

[0039] Transform the video image in RGB format. The transformation formula is:

[0040] Y=0.299R+0.587G+0.114B

[0041] U=-0.147R-0.289G+0.437B

[0042] V=0.615R-0.515G-0.100B.

[0043] Optionally, in the video sequence-based pedestrian detection method, step S2 includes:

[0044] Filter out scattered motion candidate frames in video sequences that cannot establish target IDs;

[0045] If the target ID can be established but it is blocked or missed in three consecutive frames, the video sequence is supplemented with motion candidate frames;

[0046] For targets for which target IDs can be established, the tracking candidate frames corrected by the tracking algorithm are used instead of the motion candidate frames.

[0047] Optionally, in the video sequence-based pedestrian detection method, step S3 includes:

[0048] Perform HOG feature fusion on the motion candidate box that successfully establishes the target ID, where the HOG feature includes 1 grayscale channel, 1 gradient amplitude channel and 6 gradient direction channels;

[0049] Extract deep features to predict the confidence and bounding box of pedestrian targets;

[0050] The pedestrian box is obtained using the confidence and bounding rectangle of the predicted pedestrian target.

[0051] In order to solve the above technical problems, the present invention also provides a pedestrian detection device based on video sequences, which is used to execute the pedestrian detection method based on video sequences as described in any of the above items. The pedestrian detection device includes a motion detection module, a target tracking module, a pedestrian detection module and a target management module; the motion detection module is used to extract the motion target area in the video sequence and define the extracted motion target area as a motion candidate frame; the target tracking module is used to connect the motion candidate frames of the same target in different sequences in series to establish a target ID; the pedestrian detection module is used to perform pedestrian detection on the motion candidate frames that successfully establish the target ID at a lower frame rate, and regress the pedestrian target position, and define the obtained pedestrian target position as a pedestrian frame; the target management module is used to set the target ID of the successfully detected pedestrian as an active target, and set the target ID of the undetected pedestrian as a dormant target, and use the relative position of the activated target and the pedestrian of the detected frame in the motion candidate frame to infer the pedestrian position of other frames, and the dormant target waits for the next detection.

[0052] Optionally, in the video sequence-based pedestrian detection device, the pedestrian detection device also includes a video sequence sampling module and a threshold update module; the video sequence sampling module is used to convert the video image into YUV format; the threshold update module is used to slidingly update the motion detection binarization threshold based on the maximization of the intersection between the detected pedestrian frame and the motion candidate frame.

[0053] Optionally, in the video sequence-based pedestrian detection device, the pedestrian detection module includes a network training unit, a model conversion unit, a memory optimization unit and a format selection unit; the network training unit is used to form a training set of pedestrian positive samples and difficult negative samples, and perform network training on the tensorflow platform in combination with data enhancement strategy; the model conversion unit is used to convert the tensorflow model into a C model, and compile it on the target embedded platform to obtain an executable neural network pedestrian detection library; the memory optimization unit is used to establish an intermediate layer memory structure according to the feature map memory reuse strategy to reduce the memory consumption of the pedestrian detection process; the format selection unit is used to select the aspect ratio closest to the pedestrian box size from the alternative box sizes as the network input format during online pedestrian detection.

[0054] To solve the above technical problems, the present invention further provides a computer storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for pedestrian detection based on a video sequence as described in any one of the above items is implemented.

[0055] The present invention provides a video sequence-based pedestrian detection method and device, as well as a computer storage medium. The method comprises extracting a moving target region from a video sequence and defining the extracted moving target region as a moving candidate frame; concatenating the moving candidate frames of the same target in different sequences to establish a target ID; performing pedestrian detection on the moving candidate frames that have successfully established target IDs in turn at a low frame rate, regressing the pedestrian target position, and defining the obtained pedestrian target position as a pedestrian frame; setting the target ID of a successfully detected pedestrian as an active target, and setting the target ID of an undetected pedestrian as a dormant target; and inferring the pedestrian position in other frames using the relative position of the active target and the pedestrian in the detected frame in the moving candidate frame, with the dormant target waiting for the next detection. High-quality candidate regions are screened from the spatial and temporal domains using the motion information and target tracking information in the video sequence, respectively, so that pedestrian targets can be located with minimal computational and memory costs, thereby solving the problems of existing pedestrian detection methods, such as high computational complexity, poor detection result reliability, and low detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Flowchart of the pedestrian detection method based on video sequence provided in this embodiment;

[0057] Figure 2 A schematic diagram of the structure of a pedestrian detection device based on video sequences provided in this embodiment;

[0058] Figure 3 A schematic structural diagram of a preferred pedestrian detection device based on video sequences provided in this embodiment;

[0059] Figure 4 A schematic diagram of the structure of a neural network detector in the video sequence-based pedestrian detection device provided in this embodiment;

[0060] Figure 5 A schematic diagram of the memory reuse strategy provided in this embodiment. DETAILED DESCRIPTION

[0061] The following is a further detailed description of the video sequence-based pedestrian detection method and device, and computer storage medium proposed in the present invention, in conjunction with the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and use non-precise scales, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In addition, the structures shown in the drawings are often part of the actual structure. In particular, different drawings may need to be presented with different focuses, and sometimes different scales may be used.

[0062] It should be noted that the terms "including" and "having" and any variations thereof in the specification, claims and drawings of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or are inherent to these processes, methods, products or apparatus.

[0063] This embodiment provides a pedestrian detection method based on video sequences, such as Figure 1 As shown, the pedestrian detection method includes:

[0064] S1, extract the moving target area in the video sequence and define the extracted moving target area as a motion candidate frame;

[0065] S2, concatenates the motion candidate frames of the same target in different sequences to establish the target ID;

[0066] S3, pedestrian detection is performed on the motion candidate frames that have successfully established the target ID at a lower frame rate, and the pedestrian target position is regressed, and the obtained pedestrian target position is defined as the pedestrian frame;

[0067] S4, setting the target ID for which a pedestrian is successfully detected as an active target, and setting the target ID for which a pedestrian is not detected as a dormant target;

[0068] S5, the relative positions of the activated target and the pedestrians in the detected frame in the motion candidate frame are used to infer the positions of pedestrians in other frames, and the dormant target waits for the next detection.

[0069] The basic idea of ​​the video sequence-based pedestrian detection method provided in this embodiment is to use the motion information and target tracking information in the video sequence to screen high-quality candidate regions from the spatial domain and temporal domain respectively, and customize a high-precision and low-target-size-span neural network pedestrian detector for this purpose, so as to locate pedestrian targets with extremely low computational and memory costs.

[0070] Specifically, in this embodiment, step S1 performs operations in the YUV color space, including:

[0071] Acquire video images;

[0072] Downsample the video image to the target size and convert it to YUV color space;

[0073] Use the Sobel operator to calculate the horizontal gradient P of the Y component of the video image x and longitudinal gradient P y , and using the lateral gradient P x and longitudinal gradient P y The gradient image G is calculated, where

[0074] The Sobel operator is:

[0075]

[0076] The gradient image calculation formula is:

[0077]

[0078] The moving background frame in GUV format is generated by sliding average, where the calculation formula of the moving background frame is:

[0079] G b，t =(1-α)G b，t -1+αG t

[0080] U b，t =(1-α)U b，t-1 +αU t

[0081] V b，t =(1-α)V b，t-1 +αV t

[0082] Among them, α is the weight factor, t is the detected frame;

[0083] Obtain the difference image D of the tth frame t , difference graph D t The calculation formula is:

[0084] D t =|G t -G b，t-1 |+|U t -U b，t-1 |+|V t -M b，t-1 |

[0085] According to the threshold T n Difference graph D t Perform binarization to obtain the mask image M t , mask image M t The points with value "1" constitute the foreground area, and the points with value "0" constitute the background area. The formula for binarization is:

[0086]

[0087] Among them, m t The mask image M t The pixel value in d t is the difference graph D t The pixel value in ;

[0088] Mask image M t Perform morphological processing and connectivity analysis to obtain motion candidate boxes.

[0089] Among them, the threshold T n The methods for obtaining include:

[0090] When performing pedestrian detection on the nth motion candidate frame, save the difference map D of this frame t ;

[0091] When a pedestrian is detected in the nth motion candidate frame, the current binarization optimal threshold T is calculated o，n :

[0092]

[0093] Among them, R m is the motion candidate box, R p is a set of pedestrian frames;

[0094] Use the weight β to update the binary threshold to obtain the threshold T n :

[0095]

[0096] Considering that the video image may exist in multiple formats, such as RGB format. If the video image is in RGB format, then before downsampling the video image, step S1 further includes:

[0097] Transform the video image in RGB format. The transformation formula is:

[0098] Y=0.299R+0.587G+0.114B

[0099] U=-0.147R-0.289G+0.437B

[0100] V=0.615R-0.515G-0.100B.

[0101] Furthermore, in this embodiment, step S2 includes:

[0102] Filter out scattered motion candidate frames in video sequences that cannot establish target IDs;

[0103] If the target ID can be established but it is blocked or missed in three consecutive frames, the video sequence is supplemented with motion candidate frames;

[0104] For targets for which target IDs can be established, the tracking candidate frames corrected by the tracking algorithm are used instead of the motion candidate frames.

[0105] And, in this embodiment, step S3 includes:

[0106] Perform HOG feature fusion on the motion candidate box that successfully establishes the target ID, where the HOG feature includes 1 grayscale channel, 1 gradient amplitude channel and 6 gradient direction channels;

[0107] Extract deep features to predict the confidence and bounding box of pedestrian targets;

[0108] The pedestrian box is obtained using the confidence and bounding rectangle of the predicted pedestrian target.

[0109] This embodiment also provides a pedestrian detection device based on video sequences, such as Figure 2 As shown, the pedestrian detection device includes a motion detection module, a target tracking module, a pedestrian detection module and a target management module; the motion detection module is used to extract the motion target area in the video sequence, and define the extracted motion target area as a motion candidate frame; the target tracking module is used to connect the motion candidate frames of the same target in different sequences in series to establish a target ID; the pedestrian detection module is used to perform pedestrian detection on the motion candidate frames that have successfully established the target ID in turn at a lower frame rate, and regress the pedestrian target position, and define the obtained pedestrian target position as a pedestrian frame; the target management module is used to set the target ID of the successfully detected pedestrian as an active target, and set the target ID of the undetected pedestrian as a dormant target, and use the relative position of the activated target and the detected frame pedestrian in the motion candidate frame to infer the pedestrian position in other frames, and the dormant target waits for the next detection.

[0110] The video sequence-based pedestrian detection device provided in this embodiment uses motion information and target tracking information in the video sequence to filter out high-quality candidate areas from the spatial domain and temporal domain respectively, so as to locate pedestrian targets with extremely low computational and memory costs, thereby solving the problems of existing pedestrian detection methods such as large computational complexity, poor detection result reliability, and low detection efficiency.

[0111] Preferably, in this embodiment, Figure 2 As shown, the pedestrian detection device also includes a video sequence sampling module and a threshold updating module; the video sequence sampling module is used to convert the video image into YUV format; the threshold updating module is used to slide and update the motion detection binarization threshold based on the maximization of the intersection between the detected pedestrian frame and the motion candidate frame.

[0112] In addition, in this embodiment, the pedestrian detection module includes a network training unit, a model conversion unit, a memory optimization unit and a format selection unit; the network training unit is used to form a training set of pedestrian positive samples and difficult negative samples, and combine the data enhancement strategy to perform network training on the tensorflow platform; the model conversion unit is used to convert the tensorflow model into a C model, and compile it on the target embedded platform to obtain a runnable neural network pedestrian detection library; the memory optimization unit is used to establish an intermediate layer memory structure according to the feature map memory reuse strategy to reduce the memory consumption of the pedestrian detection process; the format selection unit is used to select the aspect ratio closest to the pedestrian box size from the alternative box sizes as the network input format during online pedestrian detection.

[0113] The following describes a specific embodiment of the pedestrian detection method and device based on video sequences provided by the present invention.

[0114] The video sequence-based pedestrian detection device provided in this embodiment includes a motion detection module, a target tracking module, a pedestrian detection module, and a target management module.

[0115] In this embodiment, the motion detection module adopts a background difference method based on gradient and UV chrominance (GUV). First, after acquiring the video image, the image is first downsampled to the target size and converted to the YUV color space. In this embodiment, it is recommended that the maximum side length of the target size is 320 pixels and the aspect ratio is maintained; then, the horizontal and vertical gradients of the Y component of the image are obtained using the Sobel operator according to the method described above, and the two are calculated to obtain the gradient image G; then the sliding average is used to generate the motion background frame in the GUV format, that is, the video sequence adds the three components of GUV to the background frame according to the weight α; after a period of background learning, the tth frame is used to obtain the difference image D in the manner described above. t ; Then we need to calculate the value of the threshold T n Difference graph D t Perform binarization to obtain the mask image M t , mask image M t The points with "1" in the image constitute the foreground area, and the points with "0" constitute the background area. t Perform morphological processing such as dilation and erosion and connectivity analysis to obtain motion candidate frames.

[0116] In this embodiment, the threshold updating module will update the threshold T according to the pedestrian detection result. n When the pedestrian detection module detects pedestrians on the nth motion candidate frame, the optimal threshold calculation unit in the threshold update module will simultaneously save the differential map D of this frame. t; When a pedestrian is detected in the nth motion candidate frame, the current binarization optimal threshold T is calculated o,n ; Then use the weight β to update the binary threshold and get the threshold T n .

[0117] In this embodiment, the target tracking module uses the Simple Online and Realtime Tracking (SORT) algorithm. This algorithm tracks targets by associating position and shape information, eliminating the need for feature extraction from image content. It can maintain a tracking frame rate of over 10 fps at a low computational cost. The target tracking algorithm can be used to integrate moving frames. The integration strategy includes the following:

[0118] 1) Filter out scattered motion frames in the video sequence that cannot establish tracking IDs to reduce unnecessary candidate frame detection;

[0119] 2) Supplement candidate frames for video sequences where the tracking target is blocked or missed within three consecutive frames for which a tracking ID can be established;

[0120] 3) For the tracking target for which a tracking ID can be established, the tracking candidate frame corrected by the tracking algorithm is used to replace the original motion candidate frame.

[0121] In this embodiment, the frame rate of the video sequence used by the pedestrian detection module is 10 to 15 fps, thereby ensuring pedestrian detection at a lower frame rate and thus ensuring the accuracy of pedestrian detection. A pedestrian detector is designed in the pedestrian detection module of this embodiment, such as Figure 4 As shown in the figure, the pedestrian detector is a neural network detector. During training, the input image of the neural network needs to be cut and converted into a 64x64x3 YUV format to eliminate the YUV to RGB format conversion process during forward reasoning. The network base layer uses a HOG feature fusion module to fuse the 8-channel HOG features with the neural network features to enhance the network's shallow feature expression capabilities. The HOG features include 1 grayscale channel, 1 gradient amplitude channel, and 6 gradient direction channels. The backbone network adopts an improved EfficientNet lightweight network structure, which makes extensive use of Figure 4The MBConv module shown in the figure is used to extract deep features. However, the pedestrian detector in this embodiment focuses on small target detection and does not require a large receptive field. Therefore, all 5x5 convolution kernels are replaced with 3x3, and only the first 6 stages are retained except for the base layer. The network detection head uses a multi-head attention structure to predict the confidence and bounding rectangle of pedestrian targets. Since the detector only needs to detect pedestrian targets with a minimum length and width of 8 and a maximum of 64, only one detection branch is retained. However, the confidence prediction branch uses a multi-head prediction module to focus on different types of pedestrians to improve the accuracy of confidence prediction.

[0122] When the above-mentioned pedestrian detector is deployed on an embedded platform, in order to minimize computational waste, the pedestrian detector in this embodiment adopts an automatic size selection strategy to dynamically adjust the optimal input size of the neural network for each detection. In this embodiment, the customized convolutional neural network detector performs a maximum of 16 times downsampling, so the input size must be a multiple of 16. Then, an alternative combination of input sizes is established, and the members of the combination are represented as [width, height]. The recommended but not limited combinations are {[80,48], [64,64], [48,80]}. Each time a pedestrian is detected, the one with the closest aspect ratio to the candidate box is selected as the input size of the current detector. The candidate area is scaled and embedded into the input size for pedestrian detection.

[0123] When a neural network performs forward reasoning operations, weight parameters and feature maps usually occupy most of the memory. For example, existing neural network forward reasoning frameworks, such as NCNN, do not optimize the feature map memory of the neural network's intermediate layers. This embodiment adopts a feature map memory reuse strategy to reduce the memory consumption of the pedestrian detector. Figure 5 As shown, the feature map memory space is a pre-allocated block of memory. A neural network consists of interconnected intermediate layers. The output of each layer is represented by a top structure, which contains the starting address, memory length, and address number in the feature map memory space. The input is represented by a bottom structure, which contains only the address number of the input feature map. The network allocates addresses starting from the first layer. First, it releases memory, traversing the allocated top structure and releasing any memory not needed by subsequent layers. Then, it allocates memory, searching the released memory for a continuous address segment with a length greater than the current top memory length. If so, it occupies this segment. Otherwise, it allocates additional memory from unused space until the last layer is allocated. This feature map memory reuse strategy is executed only once during network initialization, using the maximum available size, and does not consume computing power for online real-time detection.

[0124] It should be noted that the above-mentioned pedestrian detection method based on video sequences is not limited to convolutional neural network detectors. As long as the background picture of the video sequence is static, the candidate area acquisition strategy provided in this embodiment can be used in conjunction to improve detection efficiency. For example, replacing the pedestrian detector with a traditional pedestrian detection method based on texture features can achieve a higher detection frame rate and lower memory consumption.

[0125] This embodiment further provides a computer storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for pedestrian detection based on a video sequence described in this embodiment is implemented.

[0126] In summary, the video sequence-based pedestrian detection method and device, and computer storage medium provided in this embodiment include extracting the moving target area in the video sequence and defining the extracted moving target area as a moving candidate frame; connecting the moving candidate frames of the same target in different sequences in series to establish a target ID; performing pedestrian detection on the moving candidate frames that have successfully established the target ID in turn at a lower frame rate, and regressing the pedestrian target position, and defining the obtained pedestrian target position as a pedestrian frame; setting the target ID of the successfully detected pedestrian as an active target, and setting the target ID of the undetected pedestrian as a dormant target; using the relative position of the active target and the pedestrian in the detected frame in the moving candidate frame to infer the pedestrian position in other frames, and the dormant target waits for the next detection. By using the motion information and target tracking information in the video sequence, high-quality candidate areas are screened from the spatial domain and the temporal domain respectively, so that the pedestrian target can be located with extremely low computational and memory costs, solving the problems of large computational complexity, poor reliability of detection results, and low detection efficiency of existing pedestrian detection methods.

[0127] The above description is only a description of the preferred embodiments of the present invention and does not limit the scope of the present invention. Any changes and modifications made by ordinary technicians in the field of the present invention based on the above disclosure shall fall within the scope of protection of the claims.

Claims

1. A pedestrian detection method based on video sequences, characterized in that: The pedestrian detection method comprises: S1, extract the moving target area in the video sequence and define the extracted moving target area as a motion candidate frame; S2, concatenates the motion candidate frames of the same target in different sequences to establish the target ID; S3, pedestrian detection is performed on the motion candidate frames that have successfully established the target ID at a lower frame rate, and the pedestrian target position is regressed, and the obtained pedestrian target position is defined as the pedestrian frame; S4, setting the target ID for which a pedestrian is successfully detected as an active target, and setting the target ID for which a pedestrian is not detected as a dormant target; S5, using the relative positions of the activated target and the pedestrian in the detected frame in the motion candidate frame to infer the positions of pedestrians in other frames, and the dormant target waits for the next detection; Step S1 performs operations in the YUV color space, including: acquiring a video image; downsampling the video image to a target size and converting it into the YUV color space; using the Sobel operator to obtain the horizontal gradient P of the Y component of the video image; x and longitudinal gradient P y , and using the lateral gradient P x and longitudinal gradient P y Calculate the gradient image G; generate the moving background frame in GUV format by sliding average; obtain the difference image D of the tth frame t , difference graph D t The calculation formula is: D t =|G t -G b,t-1 |+|U t -U b,t-1 |+|V t -V b,t-1 |; According to the threshold T n Difference graph D t Perform binarization to obtain the mask image M t , mask image M t The points with "1" in the image constitute the foreground area, and the points with "0" constitute the background area; t Perform morphological processing and connectivity analysis to obtain motion candidate frames; Threshold T n The methods for obtaining include: When performing pedestrian detection on the nth motion candidate frame, save the difference map D of this frame t ; When a pedestrian is detected in the nth motion candidate frame, the current binarization optimal threshold T is calculated o,n : Among them, R m is the motion candidate box, R p is a set of pedestrian frames; Use the weight β to update the binary threshold to obtain the threshold T n : Step S2 includes: Filter out scattered motion candidate frames in video sequences that cannot establish target IDs; If the target ID can be established but it is blocked or missed in three consecutive frames, the video sequence is supplemented with motion candidate frames; For targets for which target IDs can be established, the tracking candidate frames corrected by the tracking algorithm are used instead of the motion candidate frames.

2. The pedestrian detection method based on video sequence according to claim 1, characterized in that The Sobel operator is: The gradient image calculation formula is: The calculation formula for the motion background frame is: G b,t =(1-α)G b,t-1 +αG t U b,t =(1-α)U b,t-1 +αU t V b,t =(1-α)V b,t-1 +αV t Among them, α is the weight factor, t is the detected frame; The formula for binarization is: Among them, m t The mask image M t The pixel value in d t is the difference graph D t The pixel values ​​in .

3. The pedestrian detection method based on video sequence according to claim 1, characterized in that If the video image is in RGB format, before downsampling the video image, step S1 further includes: Transform the video image in RGB format. The transformation formula is: Y=0.299R+0.587G+0.114B U=-0.147R-0.289G+0.437B V=0.615R-0.515G-0.100B.

4. The pedestrian detection method based on video sequence according to claim 1, characterized in that Step S3 includes: Perform HOG feature fusion on the motion candidate box that successfully establishes the target ID, where the HOG feature includes 1 grayscale channel, 1 gradient amplitude channel and 6 gradient direction channels; Extract deep features to predict the confidence and bounding box of pedestrian targets; The pedestrian box is obtained using the confidence and bounding rectangle of the predicted pedestrian target.

5. A pedestrian detection device based on video sequences, used to execute the pedestrian detection method based on video sequences according to any one of claims 1 to 4, characterized in that: The pedestrian detection device includes a motion detection module, a target tracking module, a pedestrian detection module and a target management module; the motion detection module is used to extract the moving target area in the video sequence and define the extracted moving target area as a motion candidate frame; The target tracking module is used to connect the motion candidate frames of the same target in different sequences in series to establish a target ID; The pedestrian detection module is used to perform pedestrian detection on the motion candidate frames that have successfully established target IDs at a low frame rate, regress the pedestrian target positions, and define the obtained pedestrian target positions as pedestrian frames; The target management module is used to set the target ID of a successfully detected pedestrian as an active target, and set the target ID of an undetected pedestrian as a dormant target, and use the relative position of the activated target and the pedestrian in the detected frame in the motion candidate frame to infer the position of pedestrians in other frames, and the dormant target waits for the next detection.

6. The pedestrian detection device based on video sequence according to claim 5, characterized in that The pedestrian detection device also includes a video sequence sampling module and a threshold updating module; the video sequence sampling module is used to convert the video image into YUV format; the threshold updating module is used to slidingly update the motion detection binarization threshold based on the maximization of the intersection between the detected pedestrian frame and the motion candidate frame.

7. The pedestrian detection device based on video sequence according to claim 5, characterized in that The pedestrian detection module includes a network training unit, a model conversion unit, a memory optimization unit, and a format selection unit. The network training unit is used to form a training set of pedestrian positive samples and difficult negative samples, and combine data enhancement strategies to perform network training on the TensorFlow platform. The model conversion unit is used to convert the TensorFlow model into a C model and compile it on the target embedded platform to obtain an executable neural network pedestrian detection library. The memory optimization unit is used to establish an intermediate layer memory structure according to the feature map memory reuse strategy to reduce the memory consumption of the pedestrian detection process; The frame selection unit is used to select the frame with the aspect ratio closest to the pedestrian frame size from the candidate frame sizes during online pedestrian detection as the network input frame.

8. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the computer program is executed by the processor, the method for pedestrian detection based on a video sequence according to any one of claims 1 to 4 is implemented.

Images