Target detection method, apparatus, device, and medium

Abstract

This invention relates to a target detection method, apparatus, device, and medium. By acquiring a target video frame and its corresponding associated video frames in a video sequence, and based on the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frames, an offset feature map is determined to indicate the positional differences between corresponding sampling points in the key and auxiliary feature maps. Then, feature fusion processing is performed based on the offset feature map, key feature map, and auxiliary feature map to obtain a fused feature map, which is used to determine the target detection result of the target video frame. When the target to be detected in an image is severely occluded, the solution provided by this invention can supplement some lost information about the occluded target in the target video frame based on the continuous motion of the target video frame and associated video frames, thereby enabling the localization and classification of the target in the target video frame.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a target detection method, apparatus, device, and medium. Background Technology

[0002] Object detection, as an important research area in machine learning, has significant application value in fields such as video tracking, autonomous driving, and human-computer interaction. The development and widespread adoption of deep learning technology have greatly promoted the application and research of object detection, improving the speed and efficiency of the object detection process, and enhancing the accuracy and reliability of the results.

[0003] In related technologies, when using deep learning to achieve object detection, the main approach is to use deep learning models to process images in order to locate and classify the objects to be detected in the images.

[0004] However, when the target to be detected in an image is severely occluded, deep learning models cannot effectively locate and classify the target. Therefore, there is an urgent need for a target detection method to locate and classify the target in an image when it is severely occluded. Summary of the Invention

[0005] This invention provides a target detection method, apparatus, device, and medium to address the shortcomings of related technologies.

[0006] According to a first aspect of the present invention, a target detection method is provided, the method comprising:

[0007] The target video frame to be detected and the associated video frame corresponding to the target video frame in the video sequence are acquired. The motion of the target to be detected in the associated video frame and the target video frame is continuous.

[0008] Based on the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame, an offset feature map is determined. The offset feature map is used to indicate the positional difference between the corresponding sampling points in the key feature map and the auxiliary feature map.

[0009] Feature fusion processing is performed based on offset feature map, key feature map and auxiliary feature map to obtain fused feature map;

[0010] Based on the fused feature map, the target detection result of the target video frame is determined.

[0011] In some embodiments, the offset feature map is determined based on the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame, including:

[0012] The key feature map and the auxiliary feature map are concatenated to obtain the concatenated feature map.

[0013] The offset feature map is obtained by performing convolution processing on the spliced ​​feature map.

[0014] In some embodiments, feature fusion processing is performed based on the offset feature map, key feature map, and auxiliary feature map to obtain a fused feature map, including:

[0015] The first feature map is obtained by performing deformable convolution based on the auxiliary feature map and the offset feature map.

[0016] The first feature map and the key feature map are fused to obtain a fused feature map.

[0017] In some embodiments, deformable convolution processing is performed based on auxiliary feature maps and offset feature maps to obtain a first feature map, including:

[0018] The second feature map is obtained by interpolation based on the auxiliary feature map and the offset feature map.

[0019] The first feature map is obtained by performing convolution processing based on the second feature map.

[0020] In some embodiments, feature fusion processing is performed on the first feature map and the key feature map to obtain a fused feature map, including:

[0021] Based on preset weighting coefficients, feature fusion processing is performed on the first feature map and the key feature map to obtain a fused feature map.

[0022] In some embodiments, the target detection result is used to indicate the location and category of the target to be detected in the target video frame.

[0023] In some embodiments, the associated video frame includes the preceding video frame of the target video frame in the video sequence and / or the following video frame of the target video frame in the video sequence.

[0024] According to a second aspect of the present invention, a target detection apparatus is provided, the apparatus comprising:

[0025] The acquisition module is used to acquire the target video frame of the target to be detected and the associated video frame corresponding to the target video frame in the video sequence. The motion of the target to be detected in the associated video frame and the target video frame is continuous.

[0026] The determination module is used to determine the offset feature map based on the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame. The offset feature map is used to indicate the positional difference between corresponding sampling points in the key feature map and the auxiliary feature map.

[0027] The fusion processing module is used to perform feature fusion processing based on the offset feature map, key feature map and auxiliary feature map to obtain the fused feature map;

[0028] The determination module is also used to determine the target detection results of the target video frame based on the fused feature map.

[0029] In some embodiments, the determining module, when determining the offset feature map based on the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame, is configured to:

[0030] The key feature map and the auxiliary feature map are concatenated to obtain the concatenated feature map.

[0031] The offset feature map is obtained by performing convolution processing on the spliced ​​feature map.

[0032] In some embodiments, the fusion processing module, when performing feature fusion processing based on the offset feature map, key feature map, and auxiliary feature map to obtain a fused feature map, is used to:

[0033] The first feature map is obtained by performing deformable convolution based on the auxiliary feature map and the offset feature map.

[0034] The first feature map and the key feature map are fused to obtain a fused feature map.

[0035] In some embodiments, the fusion processing module, when performing deformable convolution processing based on the auxiliary feature map and the offset feature map to obtain the first feature map, is used to:

[0036] The second feature map is obtained by interpolation based on the auxiliary feature map and the offset feature map.

[0037] The first feature map is obtained by performing convolution processing based on the second feature map.

[0038] In some embodiments, the fusion processing module, when performing feature fusion processing on the first feature map and the key feature map to obtain a fused feature map, is used to:

[0039] Based on preset weighting coefficients, feature fusion processing is performed on the first feature map and the key feature map to obtain a fused feature map.

[0040] In some embodiments, the target detection result is used to indicate the location and category of the target to be detected in the target video frame.

[0041] In some embodiments, the associated video frame includes the preceding video frame of the target video frame in the video sequence and / or the following video frame of the target video frame in the video sequence.

[0042] According to a third aspect of the present invention, a computing device is provided, the computing device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it performs the operations performed by the target detection method provided in the first aspect and any embodiment of the first aspect.

[0043] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, it performs the operations performed by the target detection method provided in the first aspect and any embodiment of the first aspect.

[0044] According to a fifth aspect of the present invention, a computer program product is provided, the computer program product including a computer program, which, when executed by a processor, performs the operations performed by the target detection method provided in the first aspect and any embodiment of the first aspect.

[0045] This invention acquires a target video frame and its corresponding associated video frames within a video sequence. Based on the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frames, it determines an offset feature map to indicate the positional differences between corresponding sampling points in the key and auxiliary feature maps. Then, it performs feature fusion processing based on the offset feature map, key feature map, and auxiliary feature map to obtain a fused feature map. Based on this fused feature map, the target detection result of the target video frame can be determined. When the target to be detected in an image is severely occluded, the solution provided by this invention can supplement some lost information about the occluded target in the target video frame based on the continuous motion of the target video frame and associated video frames, thereby enabling the localization and classification of the target target in the target video frame.

[0046] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0048] Figure 1 This is a flowchart illustrating a target detection method according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram illustrating the principle of a conventional convolution sampling method and a deformable convolution sampling method according to an embodiment of the present invention;

[0050] Figure 3This is a flowchart illustrating a deformable convolution process according to an embodiment of the present invention;

[0051] Figure 4 This is a flowchart illustrating a process for obtaining a fused feature map according to an embodiment of the present invention;

[0052] Figure 5 This is a flowchart illustrating a possible implementation of a target detection method according to an embodiment of the present invention;

[0053] Figure 6 This is a block diagram of a target detection device according to an embodiment of the present invention;

[0054] Figure 7 This is a schematic diagram of the structure of a computing device according to an embodiment of the present invention. Detailed Implementation

[0055] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0056] When a target object included in a target video frame of a video sequence is occluded, it is difficult to detect the target video frame based solely on the information provided by that single frame. Therefore, this invention provides a target detection method that can be used to detect the target video frame when the target object is occluded, by combining supplementary information provided by associated videos within the video sequence. Since the movement of the target object in the target video frame and associated video frames is continuous, the information between the two continuously moving video frames can complement each other, thereby enhancing the feature representation of the target object and enabling the target detection process within the target video frame.

[0057] The target to be detected can be any type of object, such as a vehicle, animal, plant, person, object, etc. The present invention does not limit the specific type of the target to be detected.

[0058] The above-mentioned target detection method can be executed by a computing device, which can be a terminal device, such as a desktop computer, portable computer, laptop computer, tablet computer, smartphone, etc., or a server, such as a single server, multiple servers, server cluster, cloud computing platform, etc. The present invention does not limit the type of computing device.

[0059] The above is merely an illustrative description of the application scenarios of the present invention and does not constitute a limitation on the application scenarios of the present invention. In more possible implementations, the target detection method provided by the present invention can also be applied to other possible scenarios involving the target detection process. The present invention does not limit the specific application scenarios.

[0060] After introducing the application scenarios of the present invention, the target detection method provided by the present invention will be described below.

[0061] See Figure 1 , Figure 1 This is a flowchart illustrating a target detection method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes:

[0062] Step 101: Obtain the target video frame and the associated video frame in the video sequence, including the target to be detected. The motion of the target to be detected in the associated video frame and the target video frame is continuous.

[0063] The video sequence can be a video captured based on the continuous motion process of the target to be detected, and the target video frame and the associated video frame can be adjacent video frames in the video sequence, so that the motion of the target to be detected in the associated video frame and the target video frame has continuity.

[0064] Optionally, there can be one or more associated video frames. For example, if there is one associated video frame, it can be the preceding video frame of the target video frame in the video sequence, or it can be the following video frame of the target video frame in the video sequence. As another example, if there are two associated video frames, they can include both the preceding and following video frames of the target video frame in the video sequence. In many possible implementations, the number of associated video frames can be greater; this invention does not limit the specific number of associated video frames.

[0065] The target to be detected can be of any type, and the specific type of the target to be detected can be determined according to the actual application scenario. For example, in the object detection scenario, the target to be detected can be an apple, a table, a TV, etc., while in the autonomous driving scenario, the target to be detected can be a vehicle, a pedestrian, a road sign, etc. The present invention does not limit the specific type of the target to be detected.

[0066] Step 102: Based on the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame, determine the offset feature map. The offset feature map is used to indicate the positional difference between the corresponding sampling points in the key feature map and the auxiliary feature map.

[0067] Optionally, after obtaining the target video frame and the associated video frame through step 101, feature extraction can be performed on the target video frame and the associated video frame respectively to obtain the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame.

[0068] It should be noted that multiple sampling points are set in the associated feature map, auxiliary feature map, and offset feature map. The position and number of sampling points can be determined based on the size of the convolution kernel. This invention does not limit the specific number and position of sampling points. For example, if the size of the convolution kernel is 9, that is, a 3×3 convolution kernel, then the convolution kernel includes 9 sampling points. During the convolution process, the feature value of each sampling point can be determined based on the pixel value of the corresponding pixel in the feature map.

[0069] In this invention, for any sampling point in the offset feature map, the feature value of the sampling point can be used to indicate the positional difference between the corresponding sampling points in the key feature map and the auxiliary feature map. The positional difference can also be the offset of the sampling point. For example, the positional difference can be the difference between the position coordinates of the two. Optionally, the positional difference can also be represented by other forms of data, which is not limited in this invention.

[0070] Step 103: Perform feature fusion processing based on the offset feature map, key feature map and auxiliary feature map to obtain the fused feature map.

[0071] Step 104: Based on the fused feature map, determine the target detection result of the target video frame.

[0072] This invention acquires a target video frame and its corresponding associated video frames within a video sequence. Based on the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frames, it determines an offset feature map to indicate the positional differences between corresponding sampling points in the key and auxiliary feature maps. Then, it performs feature fusion processing based on the offset feature map, key feature map, and auxiliary feature map to obtain a fused feature map. Based on this fused feature map, the target detection result of the target video frame can be determined. When the target to be detected in an image is severely occluded, the solution provided by this invention can supplement some lost information about the occluded target in the target video frame based on the continuous motion of the target video frame and associated video frames, thereby enabling the localization and classification of the target target in the target video frame.

[0073] After introducing the basic implementation process of the target detection method provided by the present invention, the following describes various optional embodiments of the target detection method.

[0074] It should be noted that after obtaining the target video frame and the associated video frame through step 101, feature extraction can be performed on the target video frame and the associated video frame respectively to obtain the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame.

[0075] In some embodiments, when performing feature extraction on the target video frame and the associated video frame respectively to obtain the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame, a feature extraction model can be used to perform feature extraction on the target video frame and the associated video frame respectively to obtain the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame. That is, the target video frame and the associated video frame can be input into the feature extraction model respectively to obtain the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame.

[0076] The feature extraction model can be any type of neural network model. For example, the feature extraction model can be a Visual Geometry Group Network (VGG) model or a Residual Neural Network (ResNet) model. Optionally, the feature extraction model can also be other types of neural network models. This invention does not limit the specific type of feature extraction model.

[0077] Taking the VGG model as an example, the VGG model as a feature extraction model can include convolutional layers, pooling layers, and fully connected layers. For example, the VGG16 model can include 13 convolutional layers, 5 pooling layers, and 3 fully connected layers (the total number of convolutional and fully connected layers is 16, hence the name VGG16 model).

[0078] Taking the process of obtaining key feature maps of a target video frame using the VGG16 model as an example, we can first obtain the first convolutional feature based on the target video frame through two convolutional layers of the VGG16 model, and then obtain the first pooling feature based on the first convolutional feature through one pooling layer of the VGG16 model; then obtain the second convolutional feature based on the first pooling feature through three convolutional layers of the VGG16 model, and then obtain the second pooling feature based on the second convolutional feature through one pooling layer of the VGG16 model; finally, obtain the third convolutional feature based on the second pooling feature through three convolutional layers of the VGG16 model. The process involves using one pooling layer in the VGG16 model to obtain the third pooling feature based on the third convolutional feature. Then, using three convolutional layers in the VGG16 model, a fourth convolutional feature is obtained based on the third pooling feature. This process is repeated three times, resulting in a fifth convolutional feature. Finally, three fully connected layers are used to obtain the key feature map based on the fifth pooling feature. The process of obtaining auxiliary feature maps for associated video frames using the VGG16 model is similar and will not be elaborated here.

[0079] After obtaining the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame through the above process, the offset feature map can be determined in step 102 based on the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame.

[0080] In some embodiments, when determining the offset feature map based on the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame in step 102, the following steps can be taken:

[0081] Step 1021: Concatenate the key feature map and the auxiliary feature map to obtain the concatenated feature map.

[0082] It should be noted that the above concatenation process can be horizontal concatenation. Horizontal concatenation of the key feature map and the auxiliary feature map means combining them into a single concatenated feature map with more channels. For example, if the key feature map has 'a' channels and the auxiliary feature map has 'b' channels, then horizontal concatenation of the key and auxiliary feature maps will result in a feature map with (a+b) channels; that is, the concatenated feature map has (a+b) channels.

[0083] Step 1022: Perform convolution processing based on the spliced ​​feature map to obtain the offset feature map.

[0084] It should be noted that the spliced ​​feature map is obtained by splicing the key feature map and the auxiliary feature map. By performing convolution processing on the spliced ​​feature map, the result can indicate the positional difference of the corresponding sampling points between the key feature map and the auxiliary feature map, that is, the offset feature map can be obtained.

[0085] By acquiring the offset feature map, a deformable convolution process can be implemented based on the offset feature map to achieve the acquisition of the fused feature map.

[0086] In some embodiments, step 103, when performing feature fusion processing based on the offset feature map, key feature map, and auxiliary feature map to obtain the fused feature map, can be achieved through the following steps:

[0087] Step 1031: Perform deformable convolution processing based on the auxiliary feature map and the offset feature map to obtain the first feature map.

[0088] It should be noted that this invention proposes a scheme for target detection based on the integration of multiple video frames. To ensure the smooth progress of the target detection process, feature alignment of multiple video frames is required. Considering that deformable convolution processing can provide the ability to process spatial geometric deformation, feature alignment of multiple video frames with continuous motion of the target to be detected can be achieved, so as to perform feature fusion based on the result obtained after feature alignment.

[0089] To facilitate understanding, the principle of deformable convolution processing will be explained first.

[0090] Unlike conventional convolutional sampling, deformable convolutional sampling considers the offset of each sampling point in addition to the conventional method. See [link to documentation]. Figure 2 , Figure 2 This is a schematic diagram illustrating the principle of a conventional convolutional sampling method and a deformable convolutional sampling method according to embodiments of the present invention, as shown below. Figure 2 As shown in the schematic diagram of the conventional convolution sampling method, the black sampling points are the sampling points of a common 3×3 convolution kernel. In the schematic diagram of the deformable convolution sampling method, the white sampling points are the sampling points of a deformable convolution kernel, and the black sampling points represent the sampling points of a common 3×3 convolution kernel in the conventional convolution sampling method. The arrows indicate the offset of the deformable convolution sampling points relative to the conventional convolution sampling points. With the offset of each deformable convolution sampling point relative to the conventional convolution sampling points, the offset feature map is obtained, and thus the convolution processing can be implemented based on the offset feature map.

[0091] In one possible implementation, interpolation can be performed based on the auxiliary feature map and the offset feature map to obtain a second feature map, and then convolution can be performed based on the second feature map to obtain a first feature map.

[0092] Optionally, interpolation can be performed based on the auxiliary feature map, the offset feature map, and pre-set offset information to obtain the second feature map. The offset information can indicate the pixel corresponding to each sampling point. The offset information can be determined according to the size of the convolution kernel in the convolution process. For example, when the convolution kernel size is 9, that is, when the convolution kernel is a 3×3 convolution kernel, the offset information can be represented as the matrix shown in the following formula (1):

[0093]

[0094] Where, p k This is the offset information.

[0095] The positions of each sampling point in the auxiliary feature map are known. For any sampling point, based on its position and offset information, it is possible to determine which pixels it corresponds to. Therefore, the feature value of the sampling point can be determined based on the feature values ​​of these pixels. Combined with the offset feature map, the interpolation process can be implemented. It should be noted that since the feature values ​​in the offset feature map may be decimals, a bilinear interpolation method can be used to obtain the first feature map.

[0096] For example, the deformable convolution process described in the above steps can be achieved using the following formula (2):

[0097]

[0098] Where K is the number of sampling points in the convolution kernel, wk is the weight of the k-th sampling point obtained through training, y(p) represents the first feature map, x(p) represents the auxiliary feature map, and p k The offset information is pre-set, and the feature value of the kth sampling point in the offset feature map is used.

[0099] Through the above deformable convolution process, the first feature map can be obtained based on the auxiliary feature map. The features of the first feature map are aligned with those of the key feature map, thus achieving feature alignment between the auxiliary feature map and the key feature map.

[0100] The above only describes the deformable convolution process of the key feature map and one auxiliary feature map. When there are multiple associated video frames, there are also multiple auxiliary feature maps. In this case, the deformable convolution process of the key feature map and the remaining auxiliary feature maps is the same as the above process, and will not be repeated here.

[0101] See Figure 3 , Figure 3 This is a flowchart illustrating a deformable convolution process according to an embodiment of the present invention, such as... Figure 3As shown, taking a 3×3 convolution kernel as an example, the shape of the deformed convolution kernel and the position of each pixel in the convolution kernel can be determined based on the offset feature map, thereby enabling the determination of the feature value of each sampling point, thus realizing the deformable convolution processing process.

[0102] Step 1032: Perform feature fusion processing on the first feature map and the key feature map to obtain the fused feature map.

[0103] In one possible implementation, the first feature map and the key feature map can be fused using the Adaptively Spatial Feature Fusion (ASFF) method to obtain a fused feature map.

[0104] In other words, based on preset weighting coefficients, feature fusion processing can be performed on the first feature map and the key feature map to obtain a fused feature map.

[0105] Taking the associated video frames, including the target video frame, as an example, the video frames preceding and following the target video frame in the video sequence, there are two auxiliary feature maps. In this case, the feature fusion processing of the first feature map and the key feature map can be achieved through ASFF using the following formula (3):

[0106] F=α·F0+β·F1+γ·F2 (3)

[0107] Where F is the fused feature map, F0 is the key feature map corresponding to the target video frame, F1 is the auxiliary feature map corresponding to the previous video frame, F2 is the auxiliary feature map corresponding to the next video frame, and , , and are pre-set weighting coefficients.

[0108] The process of obtaining the fused feature map based on the target video frame and associated video frames described in steps 101 to 103 above can be found in [reference needed]. Figure 4 , Figure 4 This is a flowchart illustrating a process for obtaining a fused feature map according to an embodiment of the present invention, such as... Figure 4As shown, after extracting features from the target video frame (also known as the reference frame) to obtain a key feature map (denoted as Feature Map0), and extracting features from the associated video frame (also known as the supporting frame) to obtain an auxiliary feature map (denoted as Feature Map1), the key feature map and the auxiliary feature map can be horizontally concatenated (Concat). Based on the concatenated feature map, an offset feature map (denoted as Offset Map) is obtained. Then, deformable convolution (Def Conv) processing is performed on the offset feature map and the auxiliary feature map to perform adaptive spatial feature fusion (ASFF) on the key feature map and the feature map obtained by deformable convolution processing to obtain a fused feature map (denoted as Feature Map).

[0109] After obtaining the fused feature map through the above process, the target detection result of the target video frame can be determined based on the fused feature map in step 104.

[0110] In one possible implementation, a classifier can be used to determine the target detection result of the target video frame based on the fused feature map.

[0111] The target detection results can be used to indicate the location and category of the target in the target video frame. For example, the target detection results can include bounding boxes and classification labels; the bounding boxes can indicate the location of the target in the target video frame, and the classification labels can indicate the category of the target in the target video frame.

[0112] Optionally, when determining the target detection result of the target video frame based on the fused feature map, feature enhancement processing can also be performed on the fused feature map, so as to determine the target detection result of the target video frame based on the feature map obtained after feature enhancement processing.

[0113] In a more specific implementation, improvements can be made to the YOLO V5 object detection network to obtain an object detection method that can implement the present invention.

[0114] The YOLO V5 object detection network can be divided into three parts: a backbone network, a neck network, and a prediction network. The backbone is used for feature extraction, the neck is used for feature enhancement processing of the features extracted by the backbone, and the prediction network is used to determine the object detection result. Therefore, the backbone can be used as a feature extraction network, and the prediction network can be used as a classifier. By adding the network structure corresponding to the feature alignment and feature fusion processes provided in the above embodiments to the YOLO V5 object detection network, an object detection network that can implement the object detection method provided in this invention can be obtained.

[0115] The process of implementing the target detection method provided in this invention based on the YOLO V5 target detection network can be found in [link to documentation]. Figure 5 , Figure 5 This is a flowchart illustrating a possible implementation of a target detection method according to an embodiment of the present invention, such as... Figure 5 As shown, based on the backbone, neck, and prediction networks of the YOLO V5 object detection network, the backbone network can obtain key feature maps based on the target video frame, auxiliary feature maps based on the previous video frame, and auxiliary feature maps based on the next video frame. Then, feature alignment and feature fusion are performed on the key feature map and these two auxiliary feature maps to obtain a fused feature map. The neck network then enhances the fused feature map. Finally, the prediction network performs object classification and bounding box regression based on the feature map obtained from the feature enhancement to obtain the object detection result for the target video frame.

[0116] To verify the effectiveness of the target detection method provided by this invention, tests were conducted on the DukeMTMC public dataset to verify the effectiveness of the target detection method provided by this invention.

[0117] The DukeMTMC dataset is a large-scale, high-definition video dataset used for pedestrian re-identification and tracking tasks. The videos in the DukeMTMC dataset were simultaneously captured by fixed cameras in eight scenes, with a frame rate of 25fps and a resolution of 1280×720. During testing, 40-second video clips were extracted from each of the eight scenes. The test results are shown in Table 1 below.

[0118] Table 1

[0119]

[0120] Wherein, the miss rate (MR) represents the proportion of positive samples that are classified as negative samples, and the precision (Pre) represents the proportion of samples that are actually positive among all samples predicted as positive samples. Optionally, the miss rate can be calculated using the following formula (4), and the precision can be calculated using the following formula (5):

[0121]

[0122]

[0123] Where MR is the false negative rate, Pre is the accuracy, TP is the positive sample predicted as positive, FP is the negative sample predicted as positive, and FN is the positive sample predicted as negative.

[0124] As can be seen from the data shown in Table 1 above, the target detection method provided by this invention has higher accuracy and higher recall (recall = 1 - false negative rate) compared to the YOLOv5 target detection network.

[0125] Embodiments of the present invention also propose a target detection device, see [link to embodiment]. Figure 6 , Figure 6 This is a block diagram of a target detection device according to an embodiment of the present invention, such as... Figure 6 As shown, the device includes:

[0126] The acquisition module 601 is used to acquire a target video frame including the target to be detected and the associated video frame corresponding to the target video frame in the video sequence, wherein the motion of the target to be detected in the associated video frame and the target video frame is continuous;

[0127] The determination module 602 is used to determine the offset feature map based on the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame. The offset feature map is used to indicate the positional difference between corresponding sampling points in the key feature map and the auxiliary feature map.

[0128] The fusion processing module 603 is used to perform feature fusion processing based on the offset feature map, key feature map and auxiliary feature map to obtain a fused feature map;

[0129] The determination module 602 is also used to determine the target detection result of the target video frame based on the fused feature map.

[0130] In some embodiments, the determining module 602, when determining the offset feature map based on the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame, is used to:

[0131] The key feature map and the auxiliary feature map are concatenated to obtain the concatenated feature map.

[0132] The offset feature map is obtained by performing convolution processing on the spliced ​​feature map.

[0133] In some embodiments, the fusion processing module 603, when performing feature fusion processing based on the offset feature map, key feature map, and auxiliary feature map to obtain a fused feature map, is used to:

[0134] The first feature map is obtained by performing deformable convolution based on the auxiliary feature map and the offset feature map.

[0135] The first feature map and the key feature map are fused to obtain a fused feature map.

[0136] In some embodiments, the fusion processing module 603, when performing deformable convolution processing based on the auxiliary feature map and the offset feature map to obtain the first feature map, is used for:

[0137] The second feature map is obtained by interpolation based on the auxiliary feature map and the offset feature map.

[0138] The first feature map is obtained by performing convolution processing based on the second feature map.

[0139] In some embodiments, the fusion processing module 603, when performing feature fusion processing on the first feature map and the key feature map to obtain a fused feature map, is used to:

[0140] Based on preset weighting coefficients, feature fusion processing is performed on the first feature map and the key feature map to obtain a fused feature map.

[0141] In some embodiments, the target detection result is used to indicate the location and category of the target to be detected in the target video frame.

[0142] In some embodiments, the associated video frame includes the preceding video frame of the target video frame in the video sequence and / or the following video frame of the target video frame in the video sequence.

[0143] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0144] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the units can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0145] The present invention also provides a computing device, see [link to relevant documentation]. Figure 7 , Figure 7 This is a schematic diagram of the structure of a computing device according to an embodiment of the present invention. Figure 7 As shown, the computing device includes a processor 701, a memory 702, and a network interface 703. The memory 702 stores computer program code that can run on the processor 701. The processor 701 executes the computer program code to implement the target detection method provided in any embodiment of the present invention. The network interface 703 is used to implement input / output functions. In more possible implementations, the computing device may also include other hardware, which is not limited by the present invention.

[0146] This invention also provides a computer-readable storage medium, which can take many forms, such as RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (e.g., hard disk drives), solid-state drives, any type of storage disk (e.g., optical discs, DVDs), or similar storage media, or combinations thereof. Specifically, the computer-readable medium can also be paper or other suitable media capable of printing programs. A computer program is stored on the computer-readable storage medium, and when executed by a processor, the computer program implements the target detection method provided in any embodiment of this invention.

[0147] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the target detection method provided in any embodiment of the present invention.

[0148] In this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.

[0149] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0150] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A target detection method characterized by, The method comprises: obtaining a target video frame comprising a target to be detected and an associated video frame corresponding to the target video frame in a video sequence, motion of the target to be detected in the associated video frame and the target video frame being continuous; determining an offset feature map based on a key feature map corresponding to the target video frame and an auxiliary feature map corresponding to the associated video frame, the offset feature map being used to indicate a position difference between corresponding sampling points in the key feature map and the auxiliary feature map; performing feature fusion processing based on the offset feature map, the key feature map and the auxiliary feature map to obtain a fusion feature map; determining a target detection result of the target video frame based on the fusion feature map; wherein the feature fusion processing based on the offset feature map, the key feature map and the auxiliary feature map to obtain the fusion feature map comprises: performing deformable convolution processing based on the auxiliary feature map and the offset feature map to obtain a first feature map; performing feature fusion processing on the first feature map and the key feature map to obtain the fusion feature map.

2. The method of claim 1, wherein, The determination of the offset feature map based on the key feature map corresponding to the target video frame and the auxiliary feature map corresponding to the associated video frame comprises: splicing the key feature map and the auxiliary feature map to obtain a spliced feature map; performing convolution processing based on the spliced feature map to obtain the offset feature map.

3. The method of claim 1, wherein, The deformable convolution processing based on the auxiliary feature map and the offset feature map to obtain the first feature map comprises: performing interpolation processing based on the auxiliary feature map and the offset feature map to obtain a second feature map; performing convolution processing based on the second feature map to obtain the first feature map.

4. The method of claim 1, wherein, The feature fusion processing on the first feature map and the key feature map to obtain the fusion feature map comprises: performing feature fusion processing on the first feature map and the key feature map based on a preset weighting coefficient to obtain the fusion feature map.

5. The method of claim 1, wherein, The target detection result is used to indicate a position and a category of the target to be detected in the target video frame.

6. The method of claim 1, wherein, The associated video frame comprises a previous video frame of the target video frame in the video sequence and / or a next video frame of the target video frame in the video sequence.

7. A target detection apparatus characterized by comprising: The apparatus comprises: an acquisition module configured to obtain a target video frame comprising a target to be detected and an associated video frame corresponding to the target video frame in a video sequence, motion of the target to be detected in the associated video frame and the target video frame being continuous; a determination module configured to determine an offset feature map based on a key feature map corresponding to the target video frame and an auxiliary feature map corresponding to the associated video frame, the offset feature map being used to indicate a position difference between corresponding sampling points in the key feature map and the auxiliary feature map; a fusion processing module configured to perform feature fusion processing based on the offset feature map, the key feature map and the auxiliary feature map to obtain a fusion feature map; the determination module is further configured to determine a target detection result of the target video frame based on the fusion feature map; In the fusion processing module, when performing feature fusion processing based on the offset feature map, the key feature map and the auxiliary feature map to obtain a fusion feature map, the fusion processing module is configured to: perform deformable convolution processing based on the auxiliary feature map and the offset feature map to obtain a first feature map; perform feature fusion processing on the first feature map and the key feature map to obtain the fusion feature map.

8. A computing device, comprising: The computing device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the operations performed by the target detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program, and the program is executed by the processor to implement the operations performed by the target detection method according to any one of claims 1 to 6.

Images