Method, device and storage medium for detecting objects in video

Abstract

The present invention discloses a method, device, and storage medium for detecting objects in videos. The method comprises: obtaining a first image feature of a target video, wherein the first image feature is used to represent the target object to be detected in the target video; performing enhancement processing on the first image feature to obtain a second image feature; and performing convolution processing on the second image feature to obtain a detection result of the target object, wherein the detection result is used to represent the detection accuracy and orientation of the target object. The present invention solves the technical problem of low detection results for small-scale pedestrians and severely occluded pedestrians when detecting pedestrians in videos.

Description

Technical Field

[0001] The present invention relates to the field of vehicles, and in particular to a method, device and storage medium for detecting objects in a video. Background Art

[0002] Currently, the common method for detecting pedestrians in videos is a two-stage pedestrian detector. This method uses the correlation between frames to predict the position of pedestrians and integrates local and global information to assist in the detection of key frames. However, this method has low detection results for small-scale pedestrians and severely occluded pedestrians.

[0003] Currently, no effective solution has been proposed to the problem of low detection results for small-scale pedestrians and severely occluded pedestrians when detecting pedestrians in videos. Summary of the Invention

[0004] Embodiments of the present invention provide a method, device, and storage medium for detecting objects in a video, to at least solve the technical problem of low detection results for small-scale pedestrians and severely occluded pedestrians when detecting pedestrians in a video.

[0005] According to one aspect of an embodiment of the present invention, a method, apparatus, and storage medium for detecting an object in a video are provided. The method comprises: obtaining a first image feature of a target video, wherein the first image feature is used to represent a target object to be detected in the target video; performing enhancement processing on the first image feature to obtain a second image feature; and performing convolution processing on the second image feature to obtain a detection result of the target object, wherein the detection result is used to represent the detection accuracy and orientation of the target object.

[0006] Optionally, obtaining the second image feature based on performing enhancement processing on the first image feature includes: performing channel enhancement processing on the first image feature to obtain the third image feature; and performing spatial enhancement processing on the third image feature to obtain the second image feature.

[0007] Optionally, before obtaining the first image feature of the target video, the method also includes: using a residual neural network to extract the fourth image feature of the target object in each frame image to obtain a sequence feature, wherein the residual neural network is trained based on the fourth image feature; each frame image is used to represent each frame image of the target video; and channel stitching is performed on the sequence features to obtain the first image feature.

[0008] Optionally, before using the residual neural network to extract the fourth image feature of the target object in each frame image to obtain the sequence feature, the method also includes: using a fast convolutional neural network to process multiple frames of images in the target video to obtain the image features of the target object in each frame image and the orientation data of the target object in the first frame image, wherein the fast convolutional neural network is trained based on multiple frames of images in the target video, and the multiple frames of images are used to represent all frame images in the target video.

[0009] Optionally, before using the residual neural network to extract the fourth image features of the target object in each frame image to obtain the sequence features, the method also includes: using a single target tracking neural network to track the image features of the target object in each frame image and the orientation data of the target object in the target frame image to obtain multi-frame image features of the target object corresponding to the orientation, wherein the single target tracking neural network is trained based on the image features of the target object in each frame image and the orientation of the target object in the first frame image.

[0010] Optionally, based on channel enhancement processing of the first image feature, a third image feature is obtained, including: performing nonlinear processing on the first image feature to obtain a channel attention weight coefficient, wherein the channel attention weight coefficient is the degree of channel enhancement; performing linear processing on the channel attention weight coefficient and the first image feature to obtain the third image feature.

[0011] Optionally, based on spatial enhancement processing of the third image feature, the second image feature is obtained, including: performing nonlinear processing on the third image feature to obtain a spatial attention weight coefficient, wherein the spatial attention weight coefficient is the degree of spatial enhancement; performing linear processing on the spatial attention weight coefficient and the third image feature to obtain the second image feature.

[0012] According to another aspect of an embodiment of the present invention, a device for detecting an object in a video is also provided, including: an acquisition unit for acquiring a first image feature of a target video, wherein the first image feature is used to represent a target object to be detected in the target video; a first processing unit for performing enhancement processing on the first image feature to obtain a second image feature; and a second processing unit for performing convolution processing on the second image feature to obtain a detection result of the target object, wherein the detection result is used to represent the detection accuracy and orientation of the target object.

[0013] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device containing the computer-readable storage medium is controlled to execute the method for detecting objects in a video according to an embodiment of the present invention.

[0014] According to another aspect of an embodiment of the present invention, a processor is provided, which is configured to run a program, wherein the program executes the method for detecting objects in a video according to an embodiment of the present invention when the program is run.

[0015] According to another aspect of the embodiments of the present invention, a vehicle is provided. The vehicle is used to perform the method for detecting an object in a video according to the embodiment of the present invention.

[0016] In an embodiment of the present invention, a first image feature of a target video is obtained, wherein the first image feature is used to represent a target object to be detected in the target video; the first image feature is enhanced to obtain a second image feature; the second image feature is convolved to obtain a detection result of the target object, wherein the detection result is used to represent the detection accuracy and orientation of the target object. That is, the embodiment of the present invention first obtains the first image feature of the target video, then enhances the first image feature to obtain the second image feature, and finally convolves the second image feature to obtain a detection result of the target object, which solves the technical problem of low detection results for small-scale pedestrians and severely occluded pedestrians when detecting pedestrians in a video, and achieves the technical effect of high detection results for small-scale pedestrians and severely occluded pedestrians when detecting pedestrians in a video. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 is a flow chart of a method for detecting an object in a video according to an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of another method for detecting objects in a video according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of a pedestrian tracking module according to an embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of a channel splicing module according to an embodiment of the present invention;

[0022] Figure 5 2. It is a schematic diagram of a feature fusion module of an attention mechanism according to an embodiment of the present invention;

[0023] Figure 6 is a schematic diagram of a global pooling module according to an embodiment of the present invention;

[0024] Figure 72 is a schematic diagram of a maximum pooling module according to an embodiment of the present invention;

[0025] Figure 8 FIG. 4 is a schematic diagram of detecting an object in a video according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Example 1

[0029] According to an embodiment of the present invention, a method for detecting objects in a video is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than shown here.

[0030] Figure 1 FIG. 1 is a flow chart of a method for detecting a target object in a video according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0031] Step S101: Acquire a first image feature of a target video, wherein the first image feature is used to represent a target object to be detected in the target video.

[0032] In the technical solution provided in the above step S101 of the present invention, the first image feature of the target video is obtained, wherein the target object to be detected may be a pedestrian, an animal, etc., which is not specifically limited here.

[0033] Step S102: performing enhancement processing on the first image feature to obtain a second image feature.

[0034] In the technical solution provided in the above step S102 of the present invention, channel enhancement processing and space enhancement processing are performed on the first image to obtain second image features after enhancing the features of the first image.

[0035] Step S103 : performing convolution processing on the second image feature to obtain a detection result of the target object, wherein the detection result is used to indicate the detection accuracy and orientation of the target object.

[0036] In the technical solution provided in the above step S103 of the present invention, convolution processing is performed on the second image feature to obtain the detection result of the target object in the first frame image. The detection accuracy can be the detection accuracy of the target object, such as ninety percent, and the orientation can be the coordinates of the target object in the first frame, such as the point coordinates with a horizontal coordinate of sixty and a vertical coordinate of seventy.

[0037] In the above steps S101 to S103 of the present application, in an embodiment of the present invention, the first image feature of the target video is obtained, wherein the first image feature is used to represent the target object to be detected in the target video; the first image feature is enhanced to obtain the second image feature; the second image feature is convolved to obtain the detection result of the target object, wherein the detection result is used to represent the detection accuracy and orientation of the target object. That is to say, the embodiment of the present invention first obtains the first image feature of the target video, then enhances the first image feature to obtain the second image feature, and finally convolves the second image feature to obtain the detection result of the target object, which solves the technical problem of low detection results for small-scale pedestrians and severely occluded pedestrians when detecting pedestrians in the video, and achieves the technical effect of high detection results for small-scale pedestrians and severely occluded pedestrians when detecting pedestrians in the video.

[0038] The above method of this embodiment is further introduced below.

[0039] As an optional embodiment, step S102, based on enhancing the first image feature, obtains the second image feature, including: performing channel enhancement processing on the first image feature to obtain the third image feature; performing spatial enhancement processing on the third image feature to obtain the second image feature.

[0040] In this embodiment, channel enhancement processing is performed on a channel with obvious features in the first image features to obtain a third image feature, and space enhancement processing is performed on a space with obvious features in the third image features to obtain a second image feature.

[0041] For example, when the features of channels 2, 6, and 8 among the 9 channels in the first image feature are obvious, the features of channels 2, 6, and 8 are enhanced to obtain the third image feature; when the features of the first space in the third image feature are obvious, the first space is enhanced to obtain the second image feature.

[0042] As an optional embodiment, in step S101, before obtaining the first image feature of the target video, the method also includes: using a residual neural network to extract the fourth image feature of the target object in each frame image to obtain a sequence feature, wherein the residual neural network is trained based on the fourth image feature; each frame image is used to represent each frame image of the target video; and channel splicing is performed on the sequence features to obtain the first image feature.

[0043] In this embodiment, when the video has three frames, a residual neural network is used to extract the image features of the target object in the first frame image of the video to obtain a first feature, a residual neural network is used to extract the image features of the target object in the second frame image of the video to obtain a second feature, and a residual neural network is used to extract the image features of the target object in the third frame image of the video to obtain a third feature. The first feature, the second feature, and the third feature are arranged in sequence to obtain a sequence feature, and the sequence features are channel-spliced ​​to obtain a first image feature.

[0044] For another example, when the first feature has three channels, the second feature has three channels, and the third feature has three channels, the three channels of the first feature, the three channels of the second feature, and the three channels of the third feature are spliced ​​to obtain the first image feature.

[0045] As an optional embodiment, in step S101, before using a residual neural network to extract the fourth image features of the target object in each frame image to obtain sequence features, the method also includes: using a fast convolutional neural network to process multiple frames of images in the target video to obtain the fourth image features of the target object in each frame image and the orientation data of the target object in the first frame image, wherein the fast convolutional neural network is trained based on multiple frames of images in the target video, and the multiple frames of images are used to represent all frame images in the target video.

[0046] In this embodiment, each frame image in the video is processed using a residual convolutional neural network to obtain the depth features of the target object in each frame, and the depth features are marked using a candidate frame extraction network to obtain many marked frames of the depth features. The marked frames are marked on the corresponding depth features to obtain marked depth features, and the scales of the marked depth features are normalized to obtain the normalized image features of the target object in each frame image. The normalized image features of the target object in each frame image are put into the regression layer to obtain the orientation of the target object in the first frame image.

[0047] As an optional embodiment, in step S101, before using a residual neural network to extract the fourth image features of the target object in each frame image to obtain sequence features, the method also includes: using a single target tracking neural network to track and process the fourth image features of the target object in each frame image and the orientation data of the target object in the first frame image to obtain multi-frame image features of the target object corresponding to the orientation, wherein the single target tracking neural network is trained based on the image features of the target object in each frame image and the orientation of the target object in the first frame image.

[0048] In this embodiment, the image features of the target object in each frame image and the orientation of the target object in the first frame image are tracked using a single target tracking neural network. First, the coordinates of the target object from the second frame to the first frame are obtained by forward tracking, and the coordinates of the target object from the second frame to the third frame are obtained by backward tracking. Then, the multi-frame image features of the target object corresponding to the coordinates are obtained.

[0049] As an optional implementation method, step S102, based on channel enhancement processing of the first image feature, obtains the third image feature, including: performing nonlinear processing on the first image feature to obtain a channel attention weight coefficient, wherein the channel attention weight coefficient is the degree of channel enhancement; performing linear processing on the channel attention weight coefficient and the first image feature to obtain the third image feature.

[0050] In this embodiment, global average pooling is performed on the first image feature to obtain a fourth image feature; maximum pooling is performed on the fourth image feature to obtain a fifth image feature; full connection is performed twice on the fifth image feature to obtain a sixth image feature; activation is performed on the sixth image feature to obtain a channel attention weight coefficient, wherein the channel attention weight coefficient is the degree of channel enhancement; linear processing is performed on the channel attention weight coefficient and the first image feature to obtain a third image feature.

[0051] For example, the sixth image feature is processed with an activation function to obtain a channel attention weight coefficient, wherein the channel attention weight coefficient can be a matrix, and the channel attention weight coefficient is multiplied by the first image feature to obtain the third image feature.

[0052] As an optional implementation method, step S102, based on spatial enhancement processing of the third image feature, obtains the second image feature, including: performing nonlinear processing on the third image feature to obtain a spatial attention weight coefficient, wherein the spatial attention weight coefficient is the degree of spatial enhancement; performing linear processing on the spatial attention weight coefficient and the third image feature to obtain the second image feature.

[0053] In this embodiment, the third image feature is subjected to global average pooling processing to obtain the seventh image feature; the seventh image feature is subjected to maximum pooling processing to obtain the eighth image feature; the eighth image feature is subjected to convolution and activation processing to obtain a spatial attention weight coefficient, wherein the spatial attention weight coefficient is the degree of spatial enhancement; the spatial attention weight coefficient and the third image feature are linearly processed to obtain the second image feature.

[0054] For example, the eighth image feature is convolved with a 7×7 convolution kernel, and the convolved feature is processed with an activation function to obtain a spatial attention weight coefficient, where the spatial attention weight coefficient can be a matrix. The spatial attention weight coefficient is multiplied by the third image feature to obtain the second image feature.

[0055] As an optional embodiment, in step S103, convolution processing is performed on the second image feature to obtain a detection result of the target object, wherein the detection result is used to represent the detection accuracy and orientation of the target object.

[0056] In this embodiment, convolution processing is performed on the second image feature to obtain a detection result of the target object in the first frame image.

[0057] This embodiment processes multiple frames of images in a target video using a fast convolutional neural network to obtain image features of a target object in each frame and the orientation of a target object in the first image, wherein the fast convolutional neural network is trained based on multiple frames of images in the video, and tracks the image features of the target object in each frame and the orientation of the target object in the first frame using a single target tracking neural network to obtain multi-frame image features of the target object corresponding to the orientation, wherein the single target tracking neural network is trained based on the image features of the target object in each frame and the orientation of the target object in the first frame, and extracts features from the image features of the target object in each frame using a residual neural network to obtain sequence features, wherein the residual neural network is trained based on the image features of the target object; channel splicing is performed on the sequence features to obtain the first image features. feature, perform global average pooling on the first image feature to obtain the fourth image feature; perform maximum pooling on the fourth image feature to obtain the fifth image feature; perform two full connection processes on the fifth image feature to obtain the sixth image feature; perform activation processing on the sixth image feature to obtain the channel attention weight coefficient, wherein the channel attention weight coefficient is the degree of channel enhancement; perform linear processing on the channel attention weight coefficient and the first image feature to obtain the third image feature, perform global average pooling on the third image feature to obtain the seventh image feature; perform maximum pooling on the seventh image feature to obtain the eighth image feature; perform convolution and activation processing on the eighth image feature to obtain the spatial attention weight coefficient, wherein the spatial attention weight coefficient is the degree of space enhancement; perform linear processing on the spatial attention weight coefficient and the third image feature to obtain the second image feature. The second image features are convolved to obtain the detection results of the target object in the first frame image, which solves the technical problem of low detection results for small-scale pedestrians and severely occluded pedestrians when detecting pedestrians in videos, and achieves the technical effect of high detection results for small-scale pedestrians and severely occluded pedestrians when detecting pedestrians in videos.

[0058] Example 2

[0059] The technical solutions of the embodiments of the present invention are described below with reference to preferred implementation methods.

[0060] In video pedestrian detection, the appearance of pedestrians changes over time, leading to a series of challenges such as object deformation and scale transformation. During object motion, significant changes in position coordinates can lead to challenging issues such as occlusion and motion blur. Furthermore, because video datasets exhibit temporal continuity, video detection methods can leverage temporal information to strengthen the association between features of the same object in adjacent frames. When leveraging temporal information to enhance pedestrian detection, video pedestrian detection methods must consider two key aspects of performance: detection accuracy and speed.

[0061] Existing video pedestrian detection methods mainly rely on a two-stage pedestrian detector, which uses the correlation between frames to predict the position of pedestrians and integrates local and global information to assist in the detection of key frames.

[0062] Therefore, in order to overcome the above problems, in a related technology, a video target detection system and method based on multi-level spatiotemporal feature fusion is proposed, wherein the video target detection system includes a deformable feature alignment module and a multi-level spatiotemporal feature fusion module. The deformable feature alignment module predicts the motion offset of the support frame relative to the reference frame based on the reference frame and the support frame, and aligns the features of the support frame with the features of the reference frame through the motion offset; the multi-level spatiotemporal feature fusion module uses the features of the aligned support frame to enhance the features of the reference frame, and finally obtains the enhanced candidate frame features of the reference frame, and inputs them into the classification and regression network to obtain the detection results of the reference frame. It can well cope with the situations of low-quality video frames such as blur, occlusion, out-of-focus and rare postures, improve the stability and temporal consistency of video target detection, and can be widely used in fields such as autonomous driving and intelligent video surveillance.

[0063] In another related technology, a video target detection method and system based on multi-scale feature fusion is proposed. The method includes first applying a feature pyramid network to the backbone network residual network 101 used by the fast convolutional neural network to construct a deep neural network; then inputting the video target detection dataset into the backbone network, multi-scale feature fusion structure extraction candidate box network and detection head in the deep neural network for training and verification to obtain a target detection model; finally, inputting the target video image to be detected into the target detection model for prediction to obtain the target prediction category and specific location. The embodiment of the present invention simultaneously integrates high-level semantic features from the upper layer and detailed features from the lower layer, thereby more fully obtaining target feature information and effectively improving the detection accuracy of video targets.

[0064] In another related technology, a multi-scale pedestrian detection method based on feature fusion is proposed, which belongs to the field of pedestrian detection technology in computer vision. It solves the problem that the feature fusion and multi-level detection methods used in the existing technology for pedestrian detection will result in low small target detection accuracy, or long detection time and high resource requirements, thus failing to achieve real-time performance. The present invention includes: preprocessing the acquired pedestrian detection data set; constructing a multi-scale pedestrian detection convolutional neural network based on feature fusion, the multi-scale pedestrian detection convolutional neural network including a shared convolutional neural network for feature fusion extraction and a scale sub-network for detecting fused features; inputting the preprocessed pedestrian detection data set into the multi-scale pedestrian detection convolutional neural network for training to obtain a trained multi-scale pedestrian detection convolutional neural network; inputting the pedestrian image to be detected into the trained multi-scale pedestrian detection convolutional neural network to obtain the final detection result.

[0065] However, an embodiment of the present invention proposes a method for detecting pedestrians in videos based on multi-frame feature fusion. First, a deep residual network is used to extract image features, and preliminary detection results are obtained through a region proposal network. Then, each detection result of the current frame is used to use a convolutional network on adjacent frames to calculate the relevant results of the detection object channel by channel and track it to obtain a short-term tracking sequence. Finally, feature fusion is performed on this short-term tracking sequence, and the position of the original detection result is corrected and the confidence is updated to obtain the final result. The purpose of this patent is to provide a method for detecting pedestrians in videos based on multi-frame feature fusion, which uses pedestrian detection and pedestrian tracking to obtain richer pedestrian features, enhances pedestrian features of poor quality, increases the robustness of detection, and improves the detection results for small-scale pedestrians and severely occluded pedestrians.

[0066] The schematic diagram of the target object detection method in the video mainly consists of three modules, such as Figure 2 As shown, it mainly includes a single-frame pedestrian detection module 21, a pedestrian tracking module 22, and a multi-frame feature fusion module 23.

[0067] Among them, the pedestrian detection module 21 includes a deep residual network 50, a candidate box extraction network, a region of interest pooling, and a regression head.

[0068] The single-frame pedestrian detection module 21 inputs a continuous multi-frame image (from the t-τth frame to the t+τth frame), and outputs the image features of the pedestrian in each frame and the preliminary coordinates of the pedestrian target in the tth frame image.

[0069] The single-frame pedestrian detection module first passes the image through the deep residual network 50 to obtain the feature map x l ∈R Hl×Wl×Dl , where W l ,H l and Dl The width, height, and number of channels of the output feature map of the first layer are respectively calculated. The fast convolutional neural network uses selective search to generate proposal windows. After obtaining the convolutional feature map, the fast convolutional neural network uses a region proposal network to select candidate boxes of interest. This makes the fast convolutional neural network method much less time-consuming than the fast convolutional neural network. When training the region proposal network, each anchor is assigned a binary label to distinguish between positive and negative samples. If the intersection-over-union ratio (IoU) between the proposed box and the ground truth is greater than 0.7, the proposed box is classified as a positive sample. If the IoU ratio (IoU) between the proposed box and the ground truth is less than 0.3, the proposed box is classified as a negative sample. The positive-to-negative sample ratio of the proposed box is set to 1:3. The total number of positive and negative samples is fixed; negative samples are used to supplement the number of positive samples. The pooled features are passed through a regression network and a classification network, respectively, to obtain more accurate bounding box locations and scores. The two-stage object detection framework consists of two stages: region proposal and classification. The region proposal stage finds all possible object locations and then classifies each proposed box. A large number of suggestion boxes will be generated in the region proposal stage, which will affect the detection speed. The detection is not real-time and end-to-end training is impossible. The region of interest pooling layer accelerates training and testing and improves the accuracy of detection.

[0070] Among them, the pedestrian tracking module is as follows Figure 3 As shown, the pedestrian tracking module 22 includes a feature template of the tth frame, search areas of other frames, an adjustment layer, a depth correlation layer, a regression head, a classification head, a regression box, and a confidence level.

[0071] The input of pedestrian tracking module 22 is Figure 2 The network (a) outputs the image features and preliminary coordinates of the pedestrian target. The coordinates of the pedestrian target from the t-th frame to the t-th frame are obtained by forward tracking and the coordinates of the pedestrian target from the t-th frame to the t+τ-th frame are obtained by backward tracking. The output is the multi-frame image features corresponding to the coordinates.

[0072] The basic idea of ​​the pedestrian tracking algorithm is to give an initial template image z and use a similarity function f(z,x) to search for the image area x that is most similar to z in adjacent continuous t frames as the location result of pedestrian tracking.

[0073] Pedestrian detection and tracking use the same backbone network, the residual neural network 50, to extract image features, allowing them to be performed in parallel. During the pedestrian tracking process, features from the network's last three layers (layers 3-5) are selected and fused using linear weighting. The fused features are then classified and regressed to obtain the coordinates B tracked at each layer, as well as the degree of similarity S between the coordinate region features and the template image features at each layer.

[0074] The network template features and search area features of the pedestrian tracking module are respectively reduced through the adjustment layer, and the template features are then used as the convolution kernel to obtain the response map of the search area. The above operations are performed in both the classification network and the regression network to finally obtain the confidence and regression box.

[0075] The input of the multi-frame feature fusion module 23 is Figure 2 The multi-frame image features output by the network (b) in , and the output is the result of classification and regression using the fused features.

[0076] Pedestrian detection in videos has more temporal information than pedestrian detection in images. Because the features of the same pedestrian in adjacent frames are similar and continuous, obtaining feature maps of the same pedestrian in multiple frames can obtain more feature information of the pedestrian. The feature information of adjacent frames can supplement the features of the pedestrian in a single frame, and the fused features are more stable. This method selects the following fusion methods to improve pedestrian detection performance.

[0077] Channel splicing module such as Figure 4 As shown, the sequence generated by the tracking result is feature extracted, and the obtained feature channels are connected. The formula of the channel splicing module is:

[0078]

[0079] In the above formula, t is the t-th frame detected, τ is the tracking interval of the tracking sequence, the tracking sequence length is 2τ+1 frames, concat(·) is the channel splicing operation, is the number of channels of the image features of the t-th frame after channel splicing and dimensionality reduction, where the blue circle represents the three-channel features of the pedestrian in the t-τ-th frame image, the white circle represents the three-channel features of the pedestrian in the t-th frame image, the green circle represents the three-channel features of the pedestrian in the t+τ-th frame image, and the gray circle represents the three-channel features of the pedestrian after dimensionality reduction.

[0080] The feature fusion module of the attention mechanism is as follows Figure 5 As shown in the figure, the channel attention module and the spatial attention module are used to enhance the expressiveness of features. The channel attention module feeds the features of global average pooling and maximum pooling into a two-layer neural network. The average pooling operation reduces the difference between features and is sensitive to background features. The maximum pooling operation extracts values ​​with large feature differences and is sensitive to the texture features of objects in the image. The two features are superimposed and the channel attention weight coefficient M is obtained using the Sigmoid activation function. c, and multiply it with the input feature F to get the new feature F′. The spatial attention module can splice a channel dimension feature. The spliced ​​feature is average pooled and max pooled, and then activated by a 7×7 convolution kernel Sigmoid function to get the spatial attention weight coefficient M. s , and multiply it by the input feature F′ to get the new feature F″. The feature obtained by the attention mechanism is

[0081]

[0082] In the above formula, F is the feature extracted from the t-th frame, M is the weight corresponding to each channel of the feature, N is the total number of channels, and F′ is the feature after weighting using the attention mechanism.

[0083] Among them, the global pooling module is as follows Figure 6 As shown, the features are averaged channel by channel, that is, feature fusion is performed using average pooling. The formula for average pooling is:

[0084]

[0085] In the above formula, F t For the feature map extracted by the full convolutional network for the t-th frame, the image in the interval [t-τ, t+τ] is average pooled, F′ t is the obtained fusion feature map.

[0086] Among them, the maximum pooling module is as follows Figure 7 As shown, the maximum value of each feature channel is taken, and the maximum pooling is used for feature fusion. The formula of maximum pooling is:

[0087] F′ t =max(F i ), i∈[t-τ, t+τ]

[0088] In the above formula, F t is the feature map extracted by the fully convolutional network, and the image in the interval [t-τ, t+τ] is taken for maximum pooling. F′ is the obtained fusion feature map, and max represents the maximum value of the feature pixels in the selected area.

[0089] Example 3

[0090] According to an embodiment of the present invention, a device for detecting objects in a video is also provided. It should be noted that the device for detecting objects in a video can be used to execute the method for detecting objects in a video in Example 1.

[0091] Figure 8 FIG is a schematic diagram of a device for detecting an object in a video according to an embodiment of the present invention. Figure 8As shown, the apparatus 800 for detecting an object in a video may include: a first acquiring unit 801 , a first processing unit 802 , and a second processing unit 803 .

[0092] The first acquisition unit 801 is configured to acquire a first image feature of a target video, wherein the first image feature is used to represent a target object to be detected in the target video.

[0093] The first processing unit 802 is configured to perform enhancement processing on the first image feature to obtain a second image feature.

[0094] The second processing unit 803 is configured to perform convolution processing on the second image feature to obtain a detection result of the target object, wherein the detection result is used to indicate the detection accuracy and orientation of the target object.

[0095] Optionally, the second processing unit 802 may include: a first processing module that performs channel enhancement processing on the first image feature to obtain a third image feature.

[0096] Optionally, the second processing unit 802 may include: a second processing module, performing spatial enhancement processing on the third image feature to obtain the second image feature.

[0097] The apparatus further includes: a third processing unit configured to, before obtaining the first image feature of the target video, extract the fourth image feature of the target object from each frame of the image using a residual neural network to obtain a sequence feature, wherein the residual neural network is trained based on the fourth image feature; and each frame of the image is used to represent each frame of the target video. A splicing unit configured to perform channel splicing on the sequence features to obtain the first image feature.

[0098] The device also includes: a fourth processing unit, which is used to extract the fourth image features of the target object in each frame image by using a residual neural network to obtain sequence features. The method also includes: using a fast convolutional neural network to process multiple frames of images in the target video to obtain the fourth image features of the target object in each frame image and the orientation data of the target object in the first frame image, wherein the fast convolutional neural network is trained based on multiple frames of images in the target video, and the multiple frames of images are used to represent all frame images in the target video.

[0099] Among them, the device also includes: a fifth processing unit, which is used to extract the fourth image features of the target object of each frame image by using a residual neural network to obtain sequence features. The method also includes: using a single target tracking neural network to track the fourth image features of the target object of each frame image and the orientation data of the target object of the first frame image to obtain multi-frame image features of the target object corresponding to the orientation, wherein the single target tracking neural network is trained based on the image features of the target object of each frame image and the orientation of the target object of the first frame image.

[0100] Optionally, the second processing unit 802 may include: a first processing submodule, performing nonlinear processing on the first image feature to obtain a channel attention weight coefficient, wherein the channel attention weight coefficient is the degree of channel enhancement.

[0101] Optionally, the second processing unit 802 may include: a second processing submodule, performing linear processing on the channel attention weight coefficient and the first image feature to obtain a third image feature.

[0102] Optionally, the second processing unit 802 may include: a third processing submodule, performing nonlinear processing on the third image feature to obtain a spatial attention weight coefficient, wherein the spatial attention weight coefficient is the degree of spatial enhancement.

[0103] Optionally, the second processing unit 802 may include: a fourth processing submodule, performing linear processing on the spatial attention weight coefficient and the third image feature to obtain the second image feature.

[0104] In this embodiment, an acquisition unit is used to acquire a first image feature of a target video, wherein the first image feature is used to represent a target object to be detected in the target video; a first processing unit is used to perform enhancement processing on the first image feature to obtain a second image feature; and a second processing unit is used to perform convolution processing on the second image feature to obtain a detection result of the target object, wherein the detection result is used to represent the detection accuracy and orientation of the target object. This solves the technical problem of low detection results for small-scale pedestrians and severely occluded pedestrians when detecting pedestrians in a video, and achieves the technical effect of high detection results for small-scale pedestrians and severely occluded pedestrians when detecting pedestrians in a video.

[0105] Example 4

[0106] According to an embodiment of the present invention, a computer-readable storage medium is further provided. The storage medium includes a stored program, wherein the program executes the method for detecting an object in a video in embodiment 1.

[0107] Example 5

[0108] According to an embodiment of the present invention, a processor is further provided, which is used to run a program, wherein the method for detecting objects in a video in embodiment 1 is executed when the program is run.

[0109] Example 6

[0110] According to an embodiment of the present invention, a vehicle is further provided, which is used to execute the method for detecting an object in a video as described in any one of claims 1 to 7.

[0111] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0112] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0114] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0115] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0117] The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for detecting objects in a video, characterized in that: include: Acquire a first image feature of a target video, wherein the first image feature is used to represent a target object to be detected in the target video; performing enhancement processing on the first image feature to obtain a second image feature; Performing convolution processing on the second image feature to obtain a detection result of the target object, wherein the detection result is used to indicate the detection accuracy and orientation of the target object; The method further includes: using a residual convolutional neural network to process each frame image in the target video to obtain the depth features of the target object of each frame image; using a candidate frame extraction network to mark the depth features to obtain multiple marked frames of the depth features; marking the multiple marked frames on the corresponding depth features to obtain marked depth features; normalizing the scale of the marked depth features to obtain normalized image features of the target object of each frame image; and putting the image features into a regression layer to obtain the orientation of the target object of the first frame image.

2. The method according to claim 1, characterized in that Obtaining a second image feature based on enhancing the first image feature includes: performing channel enhancement processing on the first image feature to obtain a third image feature; Perform spatial enhancement processing on the third image feature to obtain the second image feature.

3. The method according to claim 1, characterized in that Before obtaining the first image feature of the target video, the method further includes: Using a residual neural network to extract the fourth image feature of the target object in each frame image to obtain a sequence feature, wherein the residual neural network is trained based on the fourth image feature; each frame image is used to represent each frame image of the target video; Channel splicing is performed on the sequence features to obtain the first image features.

4. The method according to claim 3, characterized in that Before extracting the fourth image feature of the target object in each frame of image using the residual neural network to obtain the sequence feature, the method further includes: A fast convolutional neural network is used to process multiple frames of images in the target video to obtain fourth image features of the target object in each frame image and orientation data of the target object in the first frame image, wherein the fast convolutional neural network is trained based on multiple frames of images in the target video, and the multiple frames of images are used to represent all frame images in the target video.

5. The method according to claim 3, characterized in that Before extracting the fourth image feature of the target object in each frame of image using the residual neural network to obtain the sequence feature, the method further includes: The fourth image features of the target object in each frame image and the orientation data of the target object in the first frame image are tracked and processed using a single target tracking neural network to obtain multi-frame image features of the target object corresponding to the orientation, wherein the single target tracking neural network is trained based on the image features of the target object in each frame image and the orientation of the target object in the first frame image.

6. The method according to claim 2, characterized in that The third image feature is obtained based on performing channel enhancement processing on the first image feature, including: Performing nonlinear processing on the first image feature to obtain a channel attention weight coefficient, wherein the channel attention weight coefficient is a degree of enhancement of the channel; Linearly process the channel attention weight coefficient and the first image feature to obtain the third image feature.

7. The method according to claim 2, characterized in that The second image feature is obtained based on performing spatial enhancement processing on the third image feature, including: Performing nonlinear processing on the third image feature to obtain a spatial attention weight coefficient, wherein the spatial attention weight coefficient is a degree of enhancement of the space; Linearly process the spatial attention weight coefficient and the third image feature to obtain the second image feature.

8. A device for detecting objects in a video, characterized in that: include: an acquiring unit, configured to acquire a first image feature of a target video, wherein the first image feature is used to represent a target object to be detected in the target video; a first processing unit, configured to perform enhancement processing on the first image feature to obtain a second image feature; a second processing unit, configured to perform convolution processing on the second image feature to obtain a detection result of the target object, wherein the detection result is used to indicate a detection accuracy and orientation of the target object; The device is further used to: use a residual convolutional neural network to process each frame image in the target video to obtain the depth features of the target object of each frame image; use a candidate frame extraction network to mark the depth features to obtain multiple marked frames of the depth features; mark the multiple marked frames on the corresponding depth features to obtain marked depth features; normalize the scale of the marked depth features to obtain normalized image features of the target object of each frame image; put the image features into the regression layer to obtain the orientation of the target object of the first frame image.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A vehicle, characterized in that: The vehicle is used to execute the method for detecting an object in a video as described in any one of claims 1 to 7.

Images