Target detection method and training method of target detection model

Abstract

The invention provides a target detection method and a training method of a target detection model, which can be applied to the technical field of neural networks. The target detection method comprises the following steps: performing depth feature extraction on a to-be-detected image to obtain a first intermediate image feature; the convolution kernel of the variable convolution layer is split, N variable convolution kernels are obtained, and N is larger than or equal to 1; based on the respective sampling modulation scalar and sampling position offset of the N variable convolution kernels, respectively adjusting the positions of the N variable convolution kernels to obtain the N variable convolution kernels after position adjustment; performing multi-dimensional feature extraction on the first intermediate image feature by using the N variable convolution kernels after position adjustment to obtain a second intermediate image feature; performing feature fusion on the first intermediate image feature and the second intermediate image feature to obtain a first image feature; and performing target detection on the first image feature to obtain a target detection result.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and more specifically to a target detection method and a training method for a target detection model. Background Art

[0002] The object detection task aims to accurately locate and classify objects in an image and is widely used in many fields such as image recognition, video analysis, and autonomous driving.

[0003] The positioning accuracy of existing target detection algorithms in low-resolution images is often low, and due to the interference of background and other related factors, the model has poor generalization ability when extracting features, making it difficult to perform adaptive feature extraction. Summary of the invention

[0004] In view of the above problems, the present invention provides a target detection method and a training method for a target detection model.

[0005] According to a first aspect of the present invention, a target detection method is provided, comprising: performing deep feature extraction on an image to be detected to obtain a first intermediate image feature; splitting a convolution kernel of a variable convolution layer to obtain N variable convolution kernels, where N ≥ 1; adjusting the positions of the N variable convolution kernels based on the sampling modulation scalar and sampling position offset of each of the N variable convolution kernels to obtain N variable convolution kernels after position adjustment; performing multi-dimensional feature extraction on the first intermediate image feature using the N variable convolution kernels after position adjustment to obtain a second intermediate image feature; performing feature fusion on the first intermediate image feature and the second intermediate image feature to obtain a first image feature; performing target detection on the first image feature to obtain a target detection result.

[0006] According to an embodiment of the present invention, deep feature extraction is performed on the image to be detected to obtain a first intermediate image feature, including: performing convolution processing on the image to be detected to obtain an initial image feature; performing format conversion on the initial image feature to obtain a one-dimensional image feature; and performing spatial feature extraction on the one-dimensional image feature based on a spatial attention mechanism to obtain a first intermediate image feature.

[0007] According to an embodiment of the present invention, the format of the initial image features is converted to obtain one-dimensional image features, including: globally pooling the initial image features to obtain initial one-dimensional image features; weight prediction of the initial image features to obtain feature weights; and compressing the initial one-dimensional image features based on the feature weights to obtain one-dimensional image features.

[0008] According to an embodiment of the present invention, performing target detection on the first image feature to obtain a target detection result includes: upsampling the first image feature to obtain a first upsampled image feature; performing target detection on the first upsampled image feature to obtain a target detection result.

[0009] According to an embodiment of the present invention, the above method also includes: performing deep feature extraction on the first image feature to obtain a third intermediate image feature; using the N variable convolution kernels after position adjustment, performing multi-dimensional feature extraction on the third intermediate image feature to obtain a fourth intermediate image feature; performing feature fusion on the third intermediate image feature and the fourth intermediate image feature to obtain a second image feature; wherein, performing target detection on the first image feature to obtain a target detection result includes: performing target detection on the first image feature and the second image feature to obtain a target detection result.

[0010] According to an embodiment of the present invention, target detection is performed on a first image feature and a second image feature to obtain a target detection result, including: performing feature fusion and upsampling processing on the first image feature and the second image feature to obtain a second upsampled image feature; performing feature fusion and downsampling processing on the second image feature and the second upsampled image feature to obtain a first downsampled image feature; performing target detection on the second upsampled image feature to obtain a first detection result; performing target detection on the first downsampled image feature to obtain a second detection result; and determining the target detection result based on the first detection result and the second detection result.

[0011] According to an embodiment of the present invention, the above method also includes: performing deep feature extraction on the second image feature to obtain a fifth intermediate image feature; using the N variable convolution kernels after position adjustment, performing multi-dimensional feature extraction on the fifth intermediate image feature to obtain a sixth intermediate image feature; performing feature fusion on the fifth intermediate image feature and the sixth intermediate image feature to obtain a seventh intermediate image feature; performing multi-scale feature extraction on the seventh intermediate image feature to obtain a multi-scale image feature; wherein, performing target detection on the first image feature to obtain a target detection result includes: performing target detection on the first image feature, the second image feature and the multi-scale image feature to obtain a target detection result.

[0012] According to an embodiment of the present invention, target detection is performed on the first image feature, the second image feature and the multi-scale image feature to obtain a target detection result, including: performing feature fusion and up-sampling processing on the first image feature, the second image feature and the multi-scale image feature to obtain a third up-sampled image feature; performing feature fusion and up-sampling processing on the second image feature and the multi-scale image feature to obtain a fourth up-sampled image feature; performing feature fusion and down-sampling processing on the third up-sampled image feature and the fourth up-sampled image feature to obtain a second down-sampled image feature; performing feature fusion and down-sampling processing on the second down-sampled image feature and the multi-scale image feature to obtain a third down-sampled image feature; performing target detection on the third up-sampled image feature, the second down-sampled image feature and the third down-sampled image feature to obtain a target detection result.

[0013] According to an embodiment of the present invention, target detection is performed on the third up-sampled image features, the second down-sampled image features and the third down-sampled image features to obtain a target detection result, including: performing target detection on the third up-sampled image features to obtain a third detection result; performing target detection on the second down-sampled image features to obtain a fourth detection result; performing target detection on the third down-sampled image features to obtain a fifth detection result; and determining the target detection result based on the third detection result, the fourth detection result and the fifth detection result.

[0014] The second aspect of the present invention provides a training method for a target detection model, wherein the target detection model includes a deep feature extraction module and a target detection module, wherein the deep feature extraction module includes a deep feature extraction submodule, wherein the deep feature extraction submodule includes a deep feature extraction unit, and wherein the deep feature extraction unit includes a deformable convolution subunit and a fusion subunit; the method includes: using the deep feature extraction module to perform deep feature extraction on a sample image to obtain a first intermediate sample image feature; using the deformable convolution subunit to split the convolution kernel of a variable convolution layer to obtain N variable convolution kernels, where N ≥ 1; using the deformable convolution subunit to perform sampling modulation based on the N variable convolution kernels. The positions of the N variable convolution kernels are adjusted according to the scalar quantity and the sampling position offset to obtain the N variable convolution kernels after the positions are adjusted; the deformable convolution subunit is used to perform multi-dimensional feature extraction on the first intermediate sample image features using the N variable convolution kernels after the positions are adjusted to obtain the second intermediate sample image features; the fusion subunit is used to perform feature fusion on the first intermediate sample image features and the second intermediate sample image features to obtain the first sample image features; the target detection module is used to perform target detection on the first sample image features to obtain the predicted detection results; according to the predicted detection results, the parameters of the target detection model are adjusted to obtain the trained target detection model.

[0015] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0016] The fourth aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0017] The fifth aspect of the present invention also provides a computer program product, including a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0018] According to an embodiment of the present invention, deep feature extraction is performed on the image to be detected to obtain the first intermediate image feature; the convolution kernel of the variable convolution layer is split to obtain N variable convolution kernels; based on the sampling modulation scalar and sampling position offset of each of the N variable convolution kernels, the positions of the N variable convolution kernels are adjusted respectively, so as to use the N variable convolution kernels after position adjustment to perform multi-dimensional feature extraction on the first intermediate image feature to obtain the second intermediate image feature; target detection is performed based on the first intermediate image feature and the second intermediate image feature to obtain the target detection result. Since the variable convolution layer is introduced to extract features of the image to be detected, the problem of poor generalization ability in the related technology when performing feature extraction is avoided, so that the position of the convolution kernel of the variable convolution layer can be adaptively adjusted to cope with the challenge of target posture changes. In addition, deep feature extraction is performed on the image to be detected, which avoids the interference of related factors such as background, highlights the key feature areas in the image to be detected, especially under complex background and low pixel conditions, and significantly improves the stability of the detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings.

[0020] Figure 1 A diagram showing an application scenario of a target detection method according to an embodiment of the present invention is shown.

[0021] Figure 2 A flow chart of a target detection method according to an embodiment of the present invention is shown.

[0022] Figure 3 A schematic diagram of a depth extraction submodule according to an embodiment of the present invention is shown.

[0023] Figure 4 A schematic diagram of a depth extraction unit according to an embodiment of the present invention is shown.

[0024] Figure 5 A schematic diagram of a format conversion subunit according to an embodiment of the present invention is shown.

[0025] Figure 6 A schematic diagram of a deep feature extraction subunit according to an embodiment of the present invention is shown.

[0026] Figure 7 A schematic diagram of a deformable convolution subunit according to an embodiment of the present invention is shown.

[0027] Figure 8 A schematic diagram of a target detection model according to an embodiment of the present invention is shown.

[0028] Fig. 9 A schematic diagram showing a target detection result according to an embodiment of the present invention is shown.

[0029] Fig.10 A flow chart of a method for training a target detection model according to an embodiment of the present invention is shown.

[0030] Fig.11 A structural block diagram of a target detection device according to an embodiment of the present invention is shown.

[0031] Fig.12 A block diagram of an electronic device suitable for implementing a target detection method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0032] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0033] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0034] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0035] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0036] The positioning accuracy of existing target detection algorithms in low-resolution images is often low, resulting in missed or false detection of small objects. The similarity between the target and the background increases the impact of background interference, making traditional feature extraction-based detection methods susceptible to noise, thereby affecting the accuracy and stability of detection. In addition, due to the changeable posture and motion patterns, such as rotation, tilt, occlusion, etc., the model has great uncertainty when extracting object features, further affecting the model's adaptability and accuracy to dynamic targets.

[0037] Based on this, an embodiment of the present invention provides a target detection method, including: performing deep feature extraction on an image to be detected to obtain a first intermediate image feature; splitting the convolution kernel of a variable convolution layer to obtain N variable convolution kernels, where N≥1; adjusting the positions of the N variable convolution kernels based on the sampling modulation scalar and sampling position offset of each of the N variable convolution kernels to obtain N variable convolution kernels after position adjustment; performing multi-dimensional feature extraction on the first intermediate image feature using the N variable convolution kernels after position adjustment to obtain a second intermediate image feature; performing feature fusion on the first intermediate image feature and the second intermediate image feature to obtain a first image feature; performing target detection on the first image feature to obtain a target detection result.

[0038] Figure 1 A diagram showing an application scenario of a target detection method according to an embodiment of the present invention is shown.

[0039] like Figure 1 As shown, the application scenario according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0040] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0041] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0042] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0043] It should be noted that the target detection method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the target detection device provided in the embodiment of the present invention can generally be set in the server 105. The target detection method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the target detection device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0044] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0045] The following will be based on Figure 1 The scene described by Figure 2~Figure 9 The target detection method of the embodiment of the invention is described in detail.

[0046] Figure 2 A flow chart of a target detection method according to an embodiment of the present invention is shown.

[0047] like Figure 2As shown, the target detection method of this embodiment includes operations S210 to S260.

[0048] In operation S210, depth feature extraction is performed on the image to be detected to obtain a first intermediate image feature.

[0049] According to an embodiment of the present invention, the image to be detected may be an image under any weather conditions, lighting environment or complex terrain, and the image contains a target to be identified.

[0050] According to an embodiment of the present invention, deep feature extraction is performed on the image to be detected, and features of key areas in the image to be detected are extracted to obtain first intermediate image features.

[0051] In operation S220, the convolution kernel of the variable convolution layer is split to obtain N variable convolution kernels, where N≥1.

[0052] According to an embodiment of the present invention, a variable convolution layer introduces a learnable offset on the basis of a standard convolution layer, so that the convolution kernel of the variable convolution layer can be dynamically adjusted according to the local features of the input data during sampling.

[0053] According to an embodiment of the present invention, the convolution kernel of the variable convolution layer is split to obtain N variable convolution kernels, and the N variable convolution kernels perform separate and parallel feature extraction processing on different parts of the input first intermediate image features, thereby enhancing the ability of the target detection model to adapt to and learn the spatial information of the image to be detected, so as to achieve adaptive extraction of features of the image to be detected when the image to be detected presents significant appearance changes due to its perspective, scale or deformation.

[0054] In operation S230, based on the sampling modulation scalar and the sampling position offset of each of the N variable convolution kernels, the positions of the N variable convolution kernels are adjusted respectively to obtain N variable convolution kernels after position adjustment.

[0055] According to an embodiment of the present invention, the sampling position offset of the variable convolution kernel can be obtained by learning the first intermediate image features by the variable convolution layer, and is used to adjust the sampling position of the variable convolution kernel. Specifically, the sampling position offset will be added to the predefined sampling position to obtain a new sampling position.

[0056] According to an embodiment of the present invention, the sampling modulation scalar of the variable convolution kernel can be obtained by learning the features of the first intermediate image through the variable convolution layer, and is used to control the importance of the sampling points of each variable convolution kernel. Specifically, the sampling modulation scalar is multiplied by the characteristic value obtained at the new sampling position, thereby modulating the importance of the new sampling points of each variable convolution kernel.

[0057] In operation S240, multi-dimensional feature extraction is performed on the first intermediate image features using the N variable convolution kernels after position adjustment to obtain second intermediate image features.

[0058] According to an embodiment of the present invention, multi-dimensional feature extraction of the first intermediate image feature may be performed as shown in formula (1).

[0059] (1)

[0060] in, is the second intermediate image feature, G represents the total number of variable convolution kernels (ie, G = N), where for the g-th variable convolution kernel, represents the space-independent projection weight of the variable convolution kernel, which is learned through the variable convolution layer and is used to share weights between sampling points of different variable convolution kernels. C′=C / G represents the dimension of the variable convolution kernel. is the sampling modulation scalar of the kth sampling point in the gth variable convolution kernel, represents the input features of the variable convolution kernel, is the sampling position offset of the kth grid corresponding to the variable convolution kernel, Represents the center point of the input feature of the variable convolution kernel, Represents a preset fixed offset relative to the center point, and K represents the size of the variable convolution kernel.

[0061] According to an embodiment of the present invention, the spatially independent weights across groups This ensures that the target detection model can generalize well under different spatial arrangements, while the sampling modulation scalar and sampling position offset It provides flexibility for the target detection model, enabling it to more accurately capture the characteristics of each target, thereby improving the accuracy and robustness of target detection.

[0062] According to an embodiment of the present invention, by introducing spatially independent projection weights, a splitting mechanism of variable convolution kernels and standardized sampling modulation scalars, it is possible to process target detection tasks more complexly, which is particularly suitable for targets with complex shapes and diverse postures, and enhances the ability to adaptively capture target details and spatial deformations, providing a more flexible way to handle diverse postures and spatial changes.

[0063] According to an embodiment of the present invention, by introducing a variable convolution layer, the position of the convolution kernel of the variable convolution layer can be adaptively adjusted to cope with the challenge of target posture changes, thereby significantly improving the accuracy of dynamic target detection, especially target detection under diversified motion modes.

[0064] In operation S250, feature fusion is performed on the first intermediate image feature and the second intermediate image feature to obtain a first image feature.

[0065] According to an embodiment of the present invention, by performing feature fusion on the first intermediate image feature and the second intermediate image feature, the obtained first image feature can adapt to dynamic changes of the image to be detected while highlighting the features of the key area in the image to be detected.

[0066] In operation S260, target detection is performed on the first image feature to obtain a target detection result.

[0067] According to an embodiment of the present invention, performing target detection on the first image feature may be to extract high-level semantic features and spatial detail information so as to accurately identify the target object to be detected.

[0068] According to an embodiment of the present invention, the target detection result may be a bounding box and category information of the target object in the image to be detected.

[0069] According to an embodiment of the present invention, deep feature extraction is performed on the image to be detected to obtain the first intermediate image feature; the convolution kernel of the variable convolution layer is split to obtain N variable convolution kernels; based on the sampling modulation scalar and sampling position offset of each of the N variable convolution kernels, the positions of the N variable convolution kernels are adjusted respectively, so as to use the N variable convolution kernels after position adjustment to perform multi-dimensional feature extraction on the first intermediate image feature to obtain the second intermediate image feature; target detection is performed based on the first intermediate image feature and the second intermediate image feature to obtain the target detection result. Since the variable convolution layer is introduced to extract features of the image to be detected, the problem of poor generalization ability in the related technology when performing feature extraction is avoided, so that the position of the convolution kernel of the variable convolution layer can be adaptively adjusted to cope with the challenge of target posture changes. In addition, deep feature extraction is performed on the image to be detected, which avoids the interference of related factors such as background, highlights the key feature areas in the image to be detected, especially under complex background and low pixel conditions, and significantly improves the stability of the detection effect.

[0070] According to an embodiment of the present invention, deep feature extraction is performed on the image to be detected to obtain a first intermediate image feature, including: performing convolution processing on the image to be detected to obtain an initial image feature; performing format conversion on the initial image feature to obtain a one-dimensional image feature; and performing spatial feature extraction on the one-dimensional image feature based on a spatial attention mechanism to obtain a first intermediate image feature.

[0071] According to an embodiment of the present invention, convolution processing is performed on the image to be detected to extract basic features of the image to be detected, that is, initial image features.

[0072] According to an embodiment of the present invention, the format of the initial image features is converted, and the obtained one-dimensional image features can enhance the feature extraction capability of the target detection model for the image.

[0073] According to an embodiment of the present invention, based on the spatial attention mechanism, one-dimensional image features are processed to obtain a spatial attention weight matrix corresponding to the one-dimensional image features. The spatial attention weight matrix can highlight the features of key areas in the one-dimensional image features. The spatial attention weight matrix is ​​fused with the one-dimensional image features to obtain the first intermediate image features that can alleviate the information sparsity problem caused by the low-pixel image to be detected.

[0074] According to an embodiment of the present invention, by combining the format conversion of image features and feature extraction based on the spatial attention mechanism, the target detection model can automatically highlight the key feature areas in the image to be detected, especially under complex background and low-pixel conditions, thereby significantly improving the stability of the detection effect.

[0075] According to an embodiment of the present invention, the format of the initial image features is converted to obtain one-dimensional image features, including: globally pooling the initial image features to obtain initial one-dimensional image features; weight prediction of the initial image features to obtain feature weights; and compressing the initial one-dimensional image features based on the feature weights to obtain one-dimensional image features.

[0076] According to an embodiment of the present invention, initial image features are flattened into one-dimensional features based on a global pooling layer to obtain initial one-dimensional image features, thereby increasing the feature extraction capability of the target detection model for the image to be detected.

[0077] According to an embodiment of the present invention, the initial image features are weighted predicted through a fully connected layer to obtain feature weights, which may be the weights of each feature channel in the initial image features, so as to adaptively allocate the weights of the initial image features and improve the feature weights of the key channels of the initial image features.

[0078] According to an embodiment of the present invention, the initial one-dimensional image features are compressed based on the feature weights so that the size of the compressed one-dimensional image features is consistent with the initial image features, thereby enhancing the ability of the model to be detected to focus on the target area (which may be a key channel of the initial image features).

[0079] Figure 3 A schematic diagram of a depth extraction submodule according to an embodiment of the present invention is shown; Figure 4 A schematic diagram of a depth extraction unit according to an embodiment of the present invention is shown.

[0080] According to an embodiment of the present invention, the operation of obtaining the first image feature from the initial image feature may be performed as follows: Figure 3The deep feature extraction submodule (SPD2f) of the object detection model shown is implemented.

[0081] Specifically, after the convolution processing, the initial image features are split by the splitting layer (Split) to obtain n sub-initial image features. The n sub-initial image features are respectively input into n deep feature extraction units (SPBottleneck) for deep feature extraction. Among them, the n deep feature extraction units perform the deep feature extraction task in parallel, and the n extracted image features are fused by the fusion layer (Concat) and output after convolution processing.

[0082] It is used to split the initial image features, while ensuring a high feature extraction capability, reducing the computational complexity, ensuring the richness of the gradient flow, and avoiding a significant increase in the scale of the target detection model. In addition, the fusion is processed through a 1×1 convolutional layer and the output ensures the consistency of the first image feature dimension, thereby improving the utilization efficiency of the feature information of the first image feature and reducing the computational overhead.

[0083] Among them, Figure 4 The deep feature extraction unit shown includes a format conversion subunit (SE, Squeeze-and-Excitation), a deep feature extraction subunit (PA, Pixel Attention), a convolution subunit (Conv), a deformable convolution subunit (DCN_Yolo) and a fusion subunit.

[0084] Figure 5 A schematic diagram of a format conversion subunit according to an embodiment of the present invention is shown; Figure 6 A schematic diagram of a deep feature extraction subunit according to an embodiment of the present invention is shown; Figure 7 A schematic diagram of a deformable convolution subunit according to an embodiment of the present invention is shown.

[0085] The SE subunit is used to perform format conversion on the sub-initial image features to obtain one-dimensional image features; the PA subunit is used to process the one-dimensional image features to obtain the first intermediate image features; the Conv subunit is used to perform convolution on the first intermediate image features; the DCN_Yolo subunit is used to perform multi-dimensional feature extraction on the convolved first intermediate image features to obtain the second intermediate image features; the fusion subunit is used to perform feature fusion on the first intermediate image features and the second intermediate image features to obtain the first image features.

[0086] Among them, SE subunits such as Figure 5As shown in the figure, the global pooling layer (Global Pooling) performs global pooling on the initial image features to obtain the initial one-dimensional image features; the fully connected layer (FC) compresses the initial one-dimensional image features based on the feature weights to obtain the one-dimensional image features, which are then processed by the activation function layer (Sigmoid) and output.

[0087] PA subunits such as Figure 6 As shown, a two-dimensional convolutional layer (Conv2d) is used to extract spatial features from one-dimensional image features to obtain initial first intermediate image features. The initial first intermediate image features are processed by an activation function layer (Sigmoid) and then fused with the one-dimensional image. The first intermediate image features obtained by feature fusion are output.

[0088] DCN_Yolo subunits such as Figure 7 As shown in the figure, after the first intermediate image feature is convolved by a two-dimensional convolutional layer (Conv2d), it is input into a variable convolutional layer (DCNv3) to obtain the second intermediate image feature, which is processed by a normalization layer (BatchNorm2d) and an activation function layer (SiLU) and then output.

[0089] According to an embodiment of the present invention, performing target detection on the first image feature to obtain a target detection result includes: upsampling the first image feature to obtain a first upsampled image feature; performing target detection on the first upsampled image feature to obtain a target detection result.

[0090] According to an embodiment of the present invention, the first image feature is upsampled to restore spatial detail information of the first image feature to obtain a first upsampled image feature.

[0091] According to an embodiment of the present invention, a decoupling head structure is used to separate the classification task and the positioning task of the target object, and the category information and the bounding box of the target object are outputted respectively as the target detection result.

[0092] According to an embodiment of the present invention, the above method also includes: performing deep feature extraction on the first image feature to obtain a third intermediate image feature; using the N variable convolution kernels after position adjustment, performing multi-dimensional feature extraction on the third intermediate image feature to obtain a fourth intermediate image feature; performing feature fusion on the third intermediate image feature and the fourth intermediate image feature to obtain a second image feature; wherein, performing target detection on the first image feature to obtain a target detection result includes: performing target detection on the first image feature and the second image feature to obtain a target detection result.

[0093] According to an embodiment of the present invention, further feature extraction is performed on the first image feature to extract high-level feature information to obtain a second image feature, wherein the input and output feature sizes of the first image feature and the second image feature are consistent to ensure that information is transmitted without distortion.

[0094] According to an embodiment of the present invention, target detection is performed on the first image feature and the second image feature to achieve multi-scale feature fusion, so that low-level features can better guide the detection of high-level targets, thereby enabling the obtained target detection result to retain more detail information.

[0095] According to an embodiment of the present invention, target detection is performed on a first image feature and a second image feature to obtain a target detection result, including: performing feature fusion and upsampling processing on the first image feature and the second image feature to obtain a second upsampled image feature; performing feature fusion and downsampling processing on the second image feature and the second upsampled image feature to obtain a first downsampled image feature; performing target detection on the second upsampled image feature to obtain a first detection result; performing target detection on the first downsampled image feature to obtain a second detection result; and determining the target detection result based on the first detection result and the second detection result.

[0096] According to an embodiment of the present invention, the first image feature and the second image feature are subjected to feature fusion and upsampling processing, which may be to upsample the second image feature, fuse the upsampled feature of the second image feature with the first image feature, and upsample the fused feature to obtain a second upsampled image feature.

[0097] According to an embodiment of the present invention, the second image feature and the second upsampled image feature are feature fused and downsampled. The second image feature and the second upsampled image feature are feature fused, and the fused feature is downsampled to obtain the first downsampled image feature.

[0098] According to an embodiment of the present invention, cross-layer fusion of image features is performed through downsampling and upsampling, wherein downsampling is used to extract high-level semantic features, and upsampling is used to restore spatial detail information.

[0099] According to an embodiment of the present invention, target detection is performed on the second up-sampled image features using the first decoupling head structure to obtain a first detection result.

[0100] According to an embodiment of the present invention, a second decoupling head structure is used to perform target detection on the first down-sampled image features to obtain a second detection result.

[0101] According to the embodiment of the present invention, since the first detection result and the second detection result are obtained by detecting image features of different scales, the detection effects are different for images to be detected of different sizes or different definitions.

[0102] According to an embodiment of the present invention, the decoupling head structure may be provided with multiple parallel detection channels to perform target detection on the same image feature at the same time. For example, the first decoupling head structure performs target detection on the second upsampled image feature, and uses the multiple parallel detection channels of the first decoupling head structure to output multiple initial detection results. The confidence of each initial detection result in the multiple initial detection results is determined by calculating the loss value between the multiple initial detection results to select the first detection result and the confidence corresponding to the first detection result.

[0103] Based on the confidences corresponding to the first detection result and the second detection result, the one with the highest confidence is used as the target detection result, so as to more accurately locate the target object and classify different types of target objects.

[0104] According to an embodiment of the present invention, the above method also includes: performing deep feature extraction on the second image feature to obtain a fifth intermediate image feature; using the N variable convolution kernels after position adjustment, performing multi-dimensional feature extraction on the fifth intermediate image feature to obtain a sixth intermediate image feature; performing feature fusion on the fifth intermediate image feature and the sixth intermediate image feature to obtain a seventh intermediate image feature; performing multi-scale feature extraction on the seventh intermediate image feature to obtain a multi-scale image feature; wherein, performing target detection on the first image feature to obtain a target detection result includes: performing target detection on the first image feature, the second image feature and the multi-scale image feature to obtain a target detection result.

[0105] According to an embodiment of the present invention, multi-scale feature extraction is performed on the seventh intermediate image feature, and multi-scale image features are extracted through multi-scale pooling operations, which can effectively improve the small target detection performance of the target detection model at long distances and in complex backgrounds, while enhancing the adaptability of the target detection model to image features of multiple scales.

[0106] According to an embodiment of the present invention, the second image feature is further feature extracted to extract high-level feature information to obtain a third image feature, wherein the input and output feature sizes of the second image feature and the third image feature remain consistent to ensure that information is transmitted without distortion.

[0107] According to an embodiment of the present invention, target detection is performed on the first image features, the second image features and the multi-scale image features to achieve multi-scale feature fusion, so that low-level features can better guide the detection of high-level targets, thereby enabling the obtained target detection results to retain more detail information.

[0108] According to an embodiment of the present invention, target detection is performed on the first image feature, the second image feature and the multi-scale image feature to obtain a target detection result, including: performing feature fusion and up-sampling processing on the first image feature, the second image feature and the multi-scale image feature to obtain a third up-sampled image feature; performing feature fusion and up-sampling processing on the second image feature and the multi-scale image feature to obtain a fourth up-sampled image feature; performing feature fusion and down-sampling processing on the third up-sampled image feature and the fourth up-sampled image feature to obtain a second down-sampled image feature; performing feature fusion and down-sampling processing on the second down-sampled image feature and the multi-scale image feature to obtain a third down-sampled image feature; performing target detection on the third up-sampled image feature, the second down-sampled image feature and the third down-sampled image feature to obtain a target detection result.

[0109] According to an embodiment of the present invention, the second image feature and the multi-scale image feature are feature fused and up-sampled. The second image feature and the multi-scale image feature are feature fused to obtain a first fused image feature, and the first fused image feature is up-sampled to obtain a fourth up-sampled image feature.

[0110] According to an embodiment of the present invention, the first image feature, the second image feature and the multi-scale image feature are feature fused and upsampled. The fourth upsampled image feature and the first image feature are feature fused to obtain a second fused image feature, and the second fused image feature is upsampled to obtain a third upsampled image feature.

[0111] According to an embodiment of the present invention, feature fusion and down-sampling processing are performed on the third up-sampled image feature and the fourth up-sampled image feature, and the third up-sampled image feature and the fourth up-sampled image feature can be feature fused to obtain a third fused image feature, and the third fused image feature is down-sampled to obtain a second down-sampled image feature.

[0112] According to an embodiment of the present invention, the second down-sampled image features and the multi-scale image features are feature fused and down-sampled, and the second down-sampled image features and the multi-scale image features are feature fused to obtain a fourth fused image feature, and the fourth fused image feature is down-sampled to obtain a third down-sampled image feature.

[0113] According to an embodiment of the present invention, target detection is performed on the third up-sampled image feature, the second down-sampled image feature and the third down-sampled image feature respectively, and the target detection result is determined according to the respective detection results.

[0114] According to an embodiment of the present invention, target detection is performed on the third up-sampled image features, the second down-sampled image features and the third down-sampled image features to obtain a target detection result, including: performing target detection on the third up-sampled image features to obtain a third detection result; performing target detection on the second down-sampled image features to obtain a fourth detection result; performing target detection on the third down-sampled image features to obtain a fifth detection result; and determining the target detection result based on the third detection result, the fourth detection result and the fifth detection result.

[0115] According to an embodiment of the present invention, target detection is performed on the third up-sampled image feature using the first decoupling head structure to obtain a third detection result.

[0116] According to an embodiment of the present invention, target detection is performed on the second down-sampled image features using the second decoupling head structure to obtain a fourth detection result.

[0117] According to an embodiment of the present invention, target detection is performed on the third down-sampled image features using the third decoupling head structure to obtain a fifth detection result.

[0118] According to an embodiment of the present invention, based on the confidence levels corresponding to the third detection result, the fourth detection result and the fifth detection result, the one with the highest confidence level is used as the target detection result to more accurately locate the target object and classify different types of target objects.

[0119] Figure 8 A schematic diagram of a target detection model according to an embodiment of the present invention is shown; Fig. 9 A schematic diagram showing a target detection result according to an embodiment of the present invention is shown.

[0120] According to an embodiment of the present invention, using Figure 8 The target detection model executes the above target detection method.

[0121] The image to be detected P1 is input into the deep feature extraction module (Backbone module), and the convolution submodule (Conv submodule) is used to perform convolution processing on the image to be detected to obtain the initial image features; the initial image features are input into the deep feature extraction submodule (SPD2f submodule) for deep feature extraction. Specifically, the initial image features are subjected to preliminary feature extraction by the SPD2f (1) submodule, and then processed by the SPD2f (2) submodule to output the first image features; the first intermediate image features are input into the SPD2f (3) submodule, and the second image features are output after processing; the second image features are input into the SPD2f (4) submodule, and the multi-scale feature extraction submodule (SPPF submodule) is output after processing to obtain multi-scale image features.

[0122] Subsequently, the sampling submodule (Neck submodule) and the detection submodule (Head submodule) in the detection module are used to perform target detection on the first image feature, the second image feature and the multi-scale image feature to obtain the target detection result. Specifically, the multi-scale image feature and the second image feature are fused in the fusion unit (Concat (1) unit) and then input into the sampling unit (C2f (1) unit) to obtain the fourth up-sampled image feature; the fourth up-sampled image feature and the first image feature are fused in the Concat (2) unit and then input into the C2f (2) unit to obtain the third up-sampled image feature; the third up-sampled image feature and the fourth up-sampled image feature are fused in the Concat (3) unit and then input into the C2f (3) unit to obtain the second down-sampled image feature; the second down-sampled image feature and the multi-scale image feature are fused in the Concat (4) unit and then input into the C2f (4) unit to obtain the third down-sampled image feature.

[0123] The third up-sampled image features, the second down-sampled image features and the third down-sampled image features are respectively input into the detection submodule, and the first decoupling head structure (Detect (1)) is used to perform target detection on the third up-sampled image features to obtain a third detection result; the second decoupling head structure (Detect (2)) is used to perform target detection on the second down-sampled image features to obtain a fourth detection result; and the third decoupling head structure (Detect (3)) is used to perform target detection on the third down-sampled image features to obtain a fifth detection result.

[0124] Finally, based on the confidences corresponding to the third detection result, the fourth detection result, and the fifth detection result, the one with the highest confidence is taken as the target detection result, and the bounding box and category information of the target object of the target detection result are added to the image to be detected, and the following is obtained: Fig. 9 The target image P2 shown contains the target detection result. Specifically, the target detection result may include a bounding box, category information and a confidence value. For the detected target object (such as the bear shown in the figure), on the target image, the detected bear is wrapped by a bounding box, and the bounding box attachment is marked with category information "bear" and the corresponding confidence values.

[0125] According to the embodiment of the present invention, the performance comparison between the target detection model of the present invention and the related technology is shown in Table 1. The target detection method is performed using the target detection model of the present invention. Under the premise of ensuring high detection accuracy, a high processing speed is maintained. In terms of the mean average precision (mAP) index, the target detection model of the present invention surpasses the existing model, especially in complex background and small target detection tasks, showing obvious advantages. In terms of frame rate (Frames Per Second, FPS) performance, the target detection model of the present invention can process video streams at a high frame rate to ensure that the requirements for efficient detection can be met in real-time monitoring tasks. And in terms of the number of floating point operations per second (FLOPs) index and the model parameter amount (Params) index, the target detection model of the present invention also has excellent performance compared with the related technology. It can be seen that the target detection model of the present invention has a strong ability in terms of robustness and real-time performance, can adapt to the detection requirements under different environmental conditions, and ensure stability in practical applications.

[0126]

[0127] The second aspect of the present invention provides a training method for a target detection model, wherein the target detection model includes a deep feature extraction module and a target detection module, the deep feature extraction module includes a deep feature extraction sub-module, the deep feature extraction sub-module includes a deep feature extraction unit, and the deep feature extraction unit includes a deformable convolution sub-unit and a fusion sub-unit.

[0128] Fig.10 A flow chart of a method for training a target detection model according to an embodiment of the present invention is shown.

[0129] like Fig.10 As shown, the training method of the target detection model of this embodiment includes operations S1010 to S1070.

[0130] In operation S1010, a depth feature extraction module is used to extract depth features from a sample image to obtain first intermediate sample image features.

[0131] In operation S1020, the convolution kernel of the variable convolution layer is split using a deformable convolution subunit to obtain N variable convolution kernels, where N≥1.

[0132] In operation S1030, the positions of the N variable convolution kernels are respectively adjusted based on the sampling modulation scalar and the sampling position offset of each of the N variable convolution kernels using the deformable convolution subunit to obtain N variable convolution kernels after position adjustment.

[0133] In operation S1040, a deformable convolution subunit is used to perform multi-dimensional feature extraction on the first intermediate sample image features using the N variable convolution kernels after position adjustment to obtain second intermediate sample image features.

[0134] In operation S1050, a fusion subunit is used to perform feature fusion on the first intermediate sample image feature and the second intermediate sample image feature to obtain a first sample image feature.

[0135] In operation S1060, a target detection module is used to perform target detection on the first sample image feature to obtain a predicted detection result.

[0136] In operation S1070, the parameters of the target detection model are adjusted according to the predicted detection result to obtain a trained target detection model.

[0137] According to an embodiment of the present invention, the sample image may be an unlabeled image in a sample image set, wherein the sample image set may be an image containing a target object to be detected in a variety of scenes covering various weather conditions, lighting environments (such as day and night), and complex terrain. The sample image set uses high-performance monitoring equipment for long-term image acquisition and combines data from multiple sources to ensure the diversity and representativeness of the data set.

[0138] The labeled images in the sample image set can be accurately annotated using the LabelImg tool, and the bounding box and category information are generated in the YOLO format. The images in the sample image set are randomly divided into unlabeled images and labeled images in a ratio of 8:2 to ensure the rationality of the training.

[0139] According to an embodiment of the present invention, the predicted detection result includes a predicted bounding box and predicted category information of the target object to be detected. The predicted bounding box and the true bounding box are calculated to determine a bounding box loss value, the predicted category information and the true category information are calculated to determine a category information loss value, and the parameters of the target detection model are adjusted based on the bounding box loss value and the category information loss value to obtain a trained target detection model.

[0140] According to the embodiments of the present invention, the trained target detection model can maintain good detection effects under various complex environmental conditions. In particular, in low light, complex background and long-distance target detection, the trained target detection model shows obvious advantages, which not only improves the accuracy and real-time performance of target detection, but also significantly enhances the robustness of the model in complex environments, providing strong technical support for actual monitoring tasks.

[0141] Based on the above target detection method, the present invention also provides a target detection device. Fig.11The device is described in detail.

[0142] Fig.11 A structural block diagram of a target detection device according to an embodiment of the present invention is shown.

[0143] like Fig.11 As shown, the target detection device 1100 of this embodiment includes a first extraction module 1110 , a splitting module 1120 , an adjustment module 1130 , a second extraction module 1140 , a first fusion module 1150 and a detection module 1160 .

[0144] The first extraction module 1110 is used to extract the depth feature of the image to be detected to obtain the first intermediate image feature. In one embodiment, the first extraction module 1110 can be used to perform the operation S210 described above, which will not be described in detail here.

[0145] The splitting module 1120 is used to split the convolution kernel of the variable convolution layer to obtain N variable convolution kernels, where N≥1. In one embodiment, the splitting module 1120 can be used to perform the operation S220 described above, which will not be described in detail here.

[0146] The adjustment module 1130 is used to adjust the positions of the N variable convolution kernels based on the sampling modulation scalar and sampling position offset of each of the N variable convolution kernels to obtain the N variable convolution kernels after position adjustment. In one embodiment, the adjustment module 1130 can be used to perform the operation S230 described above, which will not be repeated here.

[0147] The second extraction module 1140 is used to perform multi-dimensional feature extraction on the first intermediate image features using the N variable convolution kernels after position adjustment to obtain second intermediate image features. In one embodiment, the second extraction module 1140 can be used to perform the operation S240 described above, which will not be described in detail here.

[0148] The first fusion module 1150 is used to fuse the first intermediate image feature and the second intermediate image feature to obtain the first image feature. In one embodiment, the first fusion module 1150 can be used to perform the operation S250 described above, which will not be described in detail here.

[0149] The detection module 1160 is used to perform target detection on the first image feature to obtain a target detection result. In one embodiment, the detection module 1160 can be used to perform the operation S260 described above, which will not be described in detail here.

[0150] According to an embodiment of the present invention, deep feature extraction is performed on the image to be detected to obtain the first intermediate image feature; the convolution kernel of the variable convolution layer is split to obtain N variable convolution kernels; based on the sampling modulation scalar and sampling position offset of each of the N variable convolution kernels, the positions of the N variable convolution kernels are adjusted respectively, so as to use the N variable convolution kernels after position adjustment to perform multi-dimensional feature extraction on the first intermediate image feature to obtain the second intermediate image feature; target detection is performed based on the first intermediate image feature and the second intermediate image feature to obtain the target detection result. Since the variable convolution layer is introduced to extract features of the image to be detected, the problem of poor generalization ability in the related technology when performing feature extraction is avoided, so that the position of the convolution kernel of the variable convolution layer can be adaptively adjusted to cope with the challenge of target posture changes. In addition, deep feature extraction is performed on the image to be detected, which avoids the interference of related factors such as background, highlights the key feature areas in the image to be detected, especially under complex background and low pixel conditions, and significantly improves the stability of the detection effect.

[0151] According to an embodiment of the present invention, any multiple modules of the first extraction module 1110, the splitting module 1120, the adjustment module 1130, the second extraction module 1140, the first fusion module 1150 and the detection module 1160 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the first extraction module 1110, the splitting module 1120, the adjustment module 1130, the second extraction module 1140, the first fusion module 1150 and the detection module 1160 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a suitable combination of any of them. Alternatively, at least one of the first extraction module 1110, the splitting module 1120, the adjustment module 1130, the second extraction module 1140, the first fusion module 1150 and the detection module 1160 can be at least partially implemented as a computer program module, which can perform corresponding functions when executed.

[0152] According to an embodiment of the present invention, the first extraction module 1110 includes a first processing submodule, a conversion submodule and a first extraction submodule.

[0153] The first processing submodule is used to perform convolution processing on the image to be detected to obtain initial image features.

[0154] The conversion submodule is used to convert the format of the initial image features to obtain one-dimensional image features.

[0155] The first extraction submodule is used to perform spatial feature extraction on the one-dimensional image feature based on the spatial attention mechanism to obtain the first intermediate image feature.

[0156] According to an embodiment of the present invention, the first processing submodule includes a pooling unit, a prediction unit and a compression unit.

[0157] The pooling unit is used to globally pool the initial image features to obtain initial one-dimensional image features.

[0158] The prediction unit is used to perform weight prediction on the initial image features to obtain feature weights.

[0159] The compression unit is used to compress the initial one-dimensional image features based on the feature weights to obtain the one-dimensional image features.

[0160] According to an embodiment of the present invention, the detection module 1160 includes a first upsampling submodule and a first detection submodule.

[0161] The first upsampling submodule is used to upsample the first image feature to obtain a first upsampled image feature.

[0162] The first detection submodule is used to perform target detection on the first up-sampled image features to obtain a target detection result.

[0163] According to an embodiment of the present invention, the object detection device 1100 further includes a third extraction module, a fourth extraction module and a second fusion module.

[0164] The third extraction module is used to perform deep feature extraction on the first image feature to obtain a third intermediate image feature.

[0165] The fourth extraction module is used to use the N variable convolution kernels after position adjustment to perform multi-dimensional feature extraction on the third intermediate image features to obtain fourth intermediate image features.

[0166] The second fusion module is used to perform feature fusion on the third intermediate image feature and the fourth intermediate image feature to obtain the second image feature.

[0167] The detection module 1160 includes a second detection submodule.

[0168] The second detection submodule is used to perform target detection on the first image feature and the second image feature to obtain a target detection result.

[0169] According to an embodiment of the present invention, the second detection submodule includes a first up-sampling unit, a first down-sampling unit, a first detection unit, a second detection unit and a first determination unit.

[0170] The first up-sampling unit is used to perform feature fusion and up-sampling processing on the first image feature and the second image feature to obtain a second up-sampled image feature.

[0171] The first down-sampling unit is used to perform feature fusion and down-sampling processing on the second image feature and the second up-sampled image feature to obtain a first down-sampled image feature.

[0172] The first detection unit is used to perform feature fusion and down-sampling processing on the second image feature and the second up-sampled image feature to obtain a first down-sampled image feature.

[0173] The second detection unit is used to perform target detection on the first down-sampled image features to obtain a second detection result.

[0174] The first determining unit is used to determine a target detection result according to the first detection result and the second detection result.

[0175] According to an embodiment of the present invention, the object detection device 1100 further includes a fifth extraction module, a sixth extraction module, a third fusion module and a seventh extraction module.

[0176] The fifth extraction module is used to perform deep feature extraction on the second image feature to obtain a fifth intermediate image feature.

[0177] The sixth extraction module is used to perform multi-dimensional feature extraction on the fifth intermediate image feature by using the N variable convolution kernels after position adjustment to obtain the sixth intermediate image feature.

[0178] The third fusion module is used to perform feature fusion on the fifth intermediate image feature and the sixth intermediate image feature to obtain a seventh intermediate image feature.

[0179] The seventh extraction module is used to perform multi-scale feature extraction on the seventh intermediate image feature to obtain multi-scale image features.

[0180] The detection module 1160 includes a third detection submodule.

[0181] The third detection submodule is used to perform target detection on the first image feature, the second image feature and the multi-scale image feature to obtain a target detection result.

[0182] According to an embodiment of the present invention, the third detection submodule includes a second upsampling unit, a third upsampling unit, a second downsampling unit, a third downsampling unit and a third detection unit.

[0183] The second up-sampling unit is used to perform feature fusion and up-sampling processing on the first image feature, the second image feature and the multi-scale image feature to obtain a third up-sampled image feature.

[0184] The third up-sampling unit is used to perform feature fusion and up-sampling processing on the second image feature and the multi-scale image feature to obtain a fourth up-sampled image feature.

[0185] The second down-sampling unit is used to perform feature fusion and down-sampling processing on the third up-sampled image feature and the fourth up-sampled image feature to obtain a second down-sampled image feature.

[0186] The third down-sampling unit is used to perform feature fusion and down-sampling processing on the second down-sampled image features and the multi-scale image features to obtain third down-sampled image features.

[0187] The third detection unit is used to perform target detection on the third up-sampled image features, the second down-sampled image features and the third down-sampled image features to obtain a target detection result.

[0188] According to an embodiment of the present invention, the third detection unit includes a third detection subunit, a fourth detection subunit, a fifth detection subunit and a second determination subunit.

[0189] The third detection subunit is used to perform target detection on the third up-sampled image features to obtain a third detection result.

[0190] The fourth detection subunit is used to perform target detection on the second down-sampled image features to obtain a fourth detection result.

[0191] The fifth detection subunit is used to perform target detection on the third down-sampled image features to obtain a fifth detection result.

[0192] The second determining subunit is used to determine the target detection result according to the third detection result, the fourth detection result and the fifth detection result.

[0193] Fig.12 A block diagram of an electronic device suitable for implementing a target detection method according to an embodiment of the present invention is shown.

[0194] like Fig.12As shown, the electronic device according to an embodiment of the present invention includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage part 1208 to a random access memory (RAM) 1203. The processor 1201 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1201 may also include an onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0195] In RAM 1203, various programs and data required for the operation of the electronic device are stored. The processor 1201, ROM 1202, and RAM 1203 are connected to each other via a bus 1204. The processor 1201 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 1202 and / or RAM 1203. It should be noted that the program may also be stored in one or more memories other than ROM 1202 and RAM 1203. The processor 1201 may also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0196] According to an embodiment of the present invention, the electronic device may further include an input / output (I / O) interface 1205, which is also connected to the bus 1204. The electronic device may further include one or more of the following components connected to the input / output (I / O) interface 1205: an input portion 1206 including a keyboard, a mouse, etc.; an output portion 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1208 including a hard disk, etc.; and a communication portion 1209 including a network interface card such as a LAN card, a modem, etc. The communication portion 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the input / output (I / O) interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed, so that a computer program read therefrom is installed into the storage portion 1208 as needed.

[0197] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0198] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 1202 and / or RAM 1203 described above and / or one or more memories other than ROM 1202 and RAM 1203.

[0199] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the target detection method provided by the embodiment of the present invention.

[0200] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 1201. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0201] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1209, and / or installed from the removable medium 1211. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0202] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1209, and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0203] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0204] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0205] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0206] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A target detection method, characterized in that: The method comprises: Performing deep feature extraction on the image to be detected to obtain a first intermediate image feature; Split the convolution kernel of the variable convolution layer to obtain N variable convolution kernels, N ≥ 1; Based on the sampling modulation scalar and the sampling position offset of each of the N variable convolution kernels, the positions of the N variable convolution kernels are adjusted respectively to obtain N variable convolution kernels after position adjustment; Using the N variable convolution kernels after the position adjustment, perform multi-dimensional feature extraction on the first intermediate image features to obtain second intermediate image features; Performing feature fusion on the first intermediate image feature and the second intermediate image feature to obtain a first image feature; Perform target detection on the first image feature to obtain a target detection result.

2. The method according to claim 1, characterized in that: The step of performing deep feature extraction on the image to be detected to obtain a first intermediate image feature comprises: Performing convolution processing on the image to be detected to obtain initial image features; Performing format conversion on the initial image features to obtain one-dimensional image features; Based on the spatial attention mechanism, spatial feature extraction is performed on the one-dimensional image feature to obtain the first intermediate image feature.

3. The method according to claim 2, characterized in that The step of converting the format of the initial image features to obtain one-dimensional image features includes: Performing global pooling on the initial image features to obtain initial one-dimensional image features; Performing weight prediction on the initial image features to obtain feature weights; The initial one-dimensional image features are compressed based on the feature weights to obtain the one-dimensional image features.

4. The method according to claim 1, characterized in that: The performing target detection on the first image feature to obtain a target detection result includes: Upsampling the first image feature to obtain a first upsampled image feature; Performing target detection on the first up-sampled image features to obtain the target detection result.

5. The method according to claim 1, characterized in that: The method further comprises: Performing deep feature extraction on the first image feature to obtain a third intermediate image feature; Using the N variable convolution kernels after the position adjustment, perform multi-dimensional feature extraction on the third intermediate image feature to obtain a fourth intermediate image feature; Performing feature fusion on the third intermediate image feature and the fourth intermediate image feature to obtain a second image feature; The performing target detection on the first image feature to obtain a target detection result includes: Performing target detection on the first image feature and the second image feature to obtain the target detection result.

6. The method according to claim 5, characterized in that The performing target detection on the first image feature and the second image feature to obtain the target detection result includes: Performing feature fusion and upsampling processing on the first image feature and the second image feature to obtain a second upsampled image feature; Performing feature fusion and downsampling processing on the second image feature and the second upsampled image feature to obtain a first downsampled image feature; Performing target detection on the second up-sampled image feature to obtain a first detection result; Performing target detection on the first down-sampled image features to obtain a second detection result; The target detection result is determined according to the first detection result and the second detection result.

7. The method according to claim 6, characterized in that The method further comprises: Performing deep feature extraction on the second image feature to obtain a fifth intermediate image feature; Using the N variable convolution kernels after the position adjustment, perform multi-dimensional feature extraction on the fifth intermediate image feature to obtain a sixth intermediate image feature; performing feature fusion on the fifth intermediate image feature and the sixth intermediate image feature to obtain a seventh intermediate image feature; Performing multi-scale feature extraction on the seventh intermediate image feature to obtain a multi-scale image feature; The performing target detection on the first image feature to obtain a target detection result includes: Performing target detection on the first image feature, the second image feature, and the multi-scale image feature to obtain the target detection result.

8. The method according to claim 7, characterized in that The performing target detection on the first image feature, the second image feature and the multi-scale image feature to obtain the target detection result includes: Performing feature fusion and up-sampling processing on the first image feature, the second image feature, and the multi-scale image feature to obtain a third up-sampled image feature; Performing feature fusion and up-sampling processing on the second image feature and the multi-scale image feature to obtain a fourth up-sampled image feature; Performing feature fusion and downsampling processing on the third up-sampled image feature and the fourth up-sampled image feature to obtain a second down-sampled image feature; Performing feature fusion and downsampling processing on the second down-sampled image features and the multi-scale image features to obtain third down-sampled image features; Performing object detection on the third up-sampled image feature, the second down-sampled image feature, and the third down-sampled image feature to obtain the object detection result.

9. The method according to claim 8, characterized in that The performing target detection on the third up-sampled image feature, the second down-sampled image feature and the third down-sampled image feature to obtain the target detection result includes: Performing target detection on the third up-sampled image feature to obtain a third detection result; Performing target detection on the second down-sampled image features to obtain a fourth detection result; Performing target detection on the third down-sampled image features to obtain a fifth detection result; The target detection result is determined according to the third detection result, the fourth detection result and the fifth detection result.

10. A method for training a target detection model, characterized in that: The target detection model includes a deep feature extraction module and a target detection module, the deep feature extraction module includes a deep feature extraction submodule, the deep feature extraction submodule includes a deep feature extraction unit, and the deep feature extraction unit includes a deformable convolution subunit and a fusion subunit; The method comprises: Using the depth feature extraction module, extract the depth feature of the sample image to obtain the first intermediate sample image feature; Using the deformable convolution subunit, the convolution kernel of the variable convolution layer is split to obtain N variable convolution kernels, where N≥1; Using the deformable convolution subunit, based on the sampling modulation scalar and the sampling position offset of each of the N variable convolution kernels, the positions of the N variable convolution kernels are respectively adjusted to obtain N variable convolution kernels after position adjustment; Using the deformable convolution subunit and the N variable convolution kernels after the position adjustment, perform multi-dimensional feature extraction on the first intermediate sample image features to obtain second intermediate sample image features; Using the fusion subunit, performing feature fusion on the first intermediate sample image feature and the second intermediate sample image feature to obtain a first sample image feature; Using the target detection module, perform target detection on the first sample image feature to obtain a predicted detection result; According to the predicted detection results, the parameters of the target detection model are adjusted to obtain a trained target detection model.

Images