Target detection method, system and device and computer readable storage medium

Abstract

The invention discloses a target detection method, system and device and a computer readable storage medium, and the method comprises the steps: carrying out the feature extraction of a collected image through a shallow feature extraction stage of a RepVGG backbone network, and obtaining a shallow feature corresponding to the shallow feature extraction stage; inputting the shallow features into a deep feature extraction stage of the RepVGG backbone network, and performing convolution operation through different convolution layers of a multi-scale attention mechanism module in the deep feature extraction stage to obtain deep features corresponding to the deep feature extraction stage; performing feature fusion on the shallow-layer features and the deep-layer features to obtain fusion features; and performing target detection based on the fusion features. While the target detection precision is improved, the lightweight design of the target detection algorithm is realized so as to improve the target detection speed.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a target detection method, system, device and computer-readable storage medium. Background Art

[0002] Object detection is of great significance in the field of computer vision. It can help computers understand data such as images and videos and realize automated object recognition, classification, and tracking tasks.

[0003] In practical applications, the FCOS algorithm is deployed on mobile devices and used for target detection in areas such as autonomous driving and face recognition.

[0004] The backbone network of the FCOS algorithm uses ResNet50, which has the characteristics of large network parameters and high memory usage. However, the resources and performance of mobile devices are limited. If the FCOS algorithm is directly deployed on the mobile device, the FCOS algorithm occupies more resources of the mobile device, resulting in a decrease in the target detection speed of the mobile device. Therefore, it is necessary to deploy a lightweight target detection algorithm in the mobile device. However, the current lightweight target detection algorithm has the problem of low detection accuracy and cannot take into account both target detection accuracy and detection speed. Summary of the invention

[0005] The embodiments of the present application provide a target detection method, system, device and computer-readable storage medium, aiming to improve the target detection accuracy while achieving a lightweight design of the target detection algorithm to increase the target detection speed.

[0006] The present application provides a target detection method, the target detection method comprising:

[0007] The collected image is subjected to feature extraction in the shallow feature extraction stage of the RepVGG backbone network to obtain shallow features corresponding to the shallow feature extraction stage;

[0008] The shallow features are input into the deep feature extraction stage of the RepVGG backbone network, and convolution operations are performed on different convolutional layers of the multi-scale attention mechanism module of the deep feature extraction stage to obtain the deep features corresponding to the deep feature extraction stage;

[0009] Fusing the shallow features with the deep features to obtain fused features;

[0010] Target detection is performed based on the fused features.

[0011] Optionally, the step of inputting the shallow features into the deep feature extraction stage of the RepVGG backbone network, and performing convolution operations on different convolutional layers of the multi-scale attention mechanism module of the deep feature extraction stage to obtain the deep features corresponding to the deep feature extraction stage includes:

[0012] Inputting the shallow features into the residual block of the deep feature extraction stage, the residual block includes a first convolution branch, a second convolution branch and a residual branch, the shallow features are fused through the convolution layer and the normalization layer of the first convolution branch to obtain a fused first convolution branch, the shallow features are fused through the convolution layer and the normalization layer of the second convolution branch to obtain a fused second convolution branch, the shallow features are fused through different convolution layers of the multi-scale attention mechanism module to obtain a fused residual branch;

[0013] The convolution kernels in the fused first convolution branch, the fused second convolution branch, and the fused residual branch are respectively converted into target convolution kernels of the same size;

[0014] The target convolution kernels in all branches are merged to obtain the deep features corresponding to the deep feature extraction stage.

[0015] Optionally, the shallow features are fused through different convolutional layers of the multi-scale attention mechanism module to obtain a fused residual branch, which includes:

[0016] Convolution operation is performed on the shallow features through the first convolution layer of the multi-scale attention mechanism module to obtain the first deep features;

[0017] Using the second convolutional layer of the multi-scale attention mechanism module to perform a convolution operation on the first deep feature to obtain a second deep feature;

[0018] The second deep feature and the shallow feature are fused to obtain a fused residual branch.

[0019] Optionally, the shallow feature extraction stage includes a first shallow feature extraction stage and a second shallow feature extraction stage, and the step of extracting features of the acquired image through the shallow feature extraction stage of the RepVGG backbone network to obtain shallow features corresponding to the shallow feature extraction stage includes:

[0020] The collected image is sequentially subjected to the first shallow feature extraction stage and the second shallow feature extraction stage for feature extraction, to obtain shallow features corresponding to the shallow feature extraction stage;

[0021] The step of inputting the shallow features into the deep feature extraction stage of the RepVGG backbone network comprises:

[0022] The shallow features corresponding to the second shallow feature extraction stage are input into the deep feature extraction stage of the RepVGG backbone network.

[0023] Optionally, the first shallow feature extraction stage includes a first feature extraction stage and a second feature extraction stage, the second shallow feature extraction stage includes a third feature extraction stage and a fourth feature extraction stage, and the step of subjecting the acquired image to feature extraction in the first shallow feature extraction stage and the second shallow feature extraction stage in sequence to obtain shallow features corresponding to the shallow feature extraction stage includes:

[0024] Extract features from the acquired image in the first feature extraction stage and the second feature extraction stage in sequence to obtain a first feature;

[0025] Extract the first feature through the third feature extraction stage to obtain a shallow feature corresponding to the third feature extraction stage;

[0026] Extracting the shallow features corresponding to the third feature extraction stage through the fourth feature extraction stage to obtain the shallow features corresponding to the fourth feature extraction stage;

[0027] The first feature, the shallow features corresponding to the third feature extraction stage, and the shallow features corresponding to the fourth feature extraction stage are used as the shallow features corresponding to the shallow feature extraction stage.

[0028] Optionally, the step of inputting the shallow features and the deep features into a feature fusion network for feature fusion to obtain fused features comprises:

[0029] The shallow features corresponding to the third feature extraction stage, the shallow features corresponding to the fourth feature extraction stage, and the deep features are input into the feature fusion network for feature fusion to obtain the fused features.

[0030] Optionally, the step of inputting the shallow features and the deep features into a feature fusion network for feature fusion to obtain fused features comprises:

[0031] The shallow features and the deep features are input into a feature fusion network, and the deep features are upsampled and fused with the shallow features to obtain the fused features.

[0032] In addition, to achieve the above object, the present invention also provides a target detection system, comprising:

[0033] The shallow feature extraction module is used to extract features from the collected images through the shallow feature extraction stage of the RepVGG backbone network to obtain shallow features corresponding to the shallow feature extraction stage;

[0034] A deep feature extraction module is used to input the shallow features into the deep feature extraction stage of the RepVGG backbone network, and sequentially pass through different convolutional layers of the multi-scale attention mechanism module of the deep feature extraction stage to perform convolution operations to obtain the deep features corresponding to the deep feature extraction stage;

[0035] A fusion module, used for fusing the shallow features with the deep features to obtain fused features;

[0036] The target detection module is used to perform target detection based on the fusion features.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides a mobile device comprising: a memory, a processor, and a target detection program stored in the memory and executable on the processor, wherein the target detection program implements the steps of the above-mentioned target detection method when executed by the processor.

[0038] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a target detection program, and when the target detection program is executed by a processor, the steps of the above-mentioned target detection method are implemented.

[0039] A technical solution of a target detection method, system, device and computer-readable storage medium provided in the embodiments of the present application adopts a lightweight RepVGG backbone network to replace the backbone network of the FCOS anchor-free target detection algorithm, reduces the number of model parameters and the amount of calculation to achieve real-time target detection, and adds an attention mechanism in the deep feature extraction stage to improve the detection accuracy. While maintaining the detection accuracy, the model can run efficiently on mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of the first embodiment of the target detection method of the present invention;

[0041] Figure 2 It is a schematic diagram of the structure of the target detection network of the present invention;

[0042] Figure 3 It is a structural diagram of the RepVGG-ECA module corresponding to the deep feature extraction stage of the prior art;

[0043] Figure 4 It is a schematic diagram of the structure of the RepVGG-MSECA module corresponding to the deep feature extraction stage of the present invention;

[0044] Figure 5 It is a structural diagram of the MSECANet model of the present invention;

[0045] Figure 6 It is a functional module diagram of the target detection system of the present invention;

[0046] Figure 7 It is a structural schematic diagram of the mobile device of the present invention.

[0047] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. The above-mentioned drawings are only an embodiment diagram, not the entire invention. DETAILED DESCRIPTION

[0048] At present, the FCOS algorithm is deployed on mobile devices and used for target detection in the fields of autonomous driving, face recognition, etc. The backbone network of the FCOS algorithm uses ResNet50, which has the characteristics of large network parameters and high memory usage. The resources and performance of mobile devices are limited. If the FCOS algorithm is directly deployed on the mobile device, the FCOS algorithm occupies more resources of the mobile device, resulting in a decrease in the target detection speed of the mobile device. In addition, the current lightweight target detection algorithm has the problem of low detection accuracy, making it impossible to take into account both target detection accuracy and detection speed at the same time.

[0049] In response to the above problems, the present application proposes a new target detection method, and the technical solution adopted includes: extracting features from the captured image through the shallow feature extraction stage of the RepVGG backbone network to obtain shallow features corresponding to the shallow feature extraction stage; inputting the shallow features into the deep feature extraction stage of the RepVGG backbone network, and sequentially passing through different convolutional layers of the multi-scale attention mechanism module of the deep feature extraction stage for convolution operations to obtain deep features corresponding to the deep feature extraction stage; fusing the shallow features and the deep features to obtain fused features; and performing target detection based on the fused features.

[0050] Specifically, the present application adopts a lightweight backbone network RepVGG to replace the original backbone network, and uses the structural reparameterization technology to convert the multi-branch structure of the training phase into a single-path structure of the reasoning phase, thereby improving the reasoning speed of the model.

[0051] Secondly, the lightweight attention mechanism was improved, and a multi-scale attention mechanism was proposed. A multi-scale attention mechanism module was added in the deep feature extraction stage to automatically adjust the weights between channels to highlight key features, solve the problem of insufficient feature information extraction from lightweight networks, and improve the detection accuracy of lightweight target detection algorithms.

[0052] Since this application has made lightweight improvements to the backbone network of the FCOS anchor-free target detection algorithm to reduce the number of model parameters and computational complexity to achieve real-time target detection, and added an attention mechanism to improve detection accuracy, the model can run efficiently on mobile devices while maintaining detection accuracy.

[0053] In order to better understand the above technical solution, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0054] like Figure 1 As shown, in the first embodiment of the present application, the target detection method of the present application is applied to a mobile device, and the mobile device can be smart glasses such as VR glasses, MR glasses, etc., unmanned vehicles, drones and other devices, and a target detection program is stored in the mobile device to implement the steps of the target detection method of the present application. The present application takes the mobile device as smart glasses as an example. The lightweight target detection network structure of the present application is as follows Figure 2 As shown, the entire network is divided into three parts, namely the backbone network, the neck network and the prediction head network. Among them, the backbone network is at the forefront of the entire network and plays a role in feature extraction. The performance of the backbone network directly affects the performance of the entire network. The target detection method of this application includes the following steps:

[0055] Step S110, extracting features of the collected image through the shallow feature extraction stage of the RepVGG backbone network to obtain shallow features corresponding to the shallow feature extraction stage.

[0056] Optionally, when wearing smart glasses, the image acquisition device on the smart glasses acquires images, and performs feature extraction on the acquired images through the shallow feature extraction stage of the RepVGG backbone network to obtain shallow features corresponding to the shallow feature extraction stage.

[0057] Optionally, the present application adopts the RepVGG-A0 lightweight network in the RepVGG series as the backbone network of the FCOS target detection algorithm. The RepVGG backbone network is divided into a training phase and an inference phase. The training phase and the inference phase use different network architectures. The training phase focuses more on accuracy, while the inference phase focuses more on speed. In the training phase, the present application adds a Conv1x1 convolution branch to the RepVGG backbone network, which forms an asymmetric convolution structure with the original Conv3x3 convolution branch to enhance the feature extraction capability; at the same time, the residual branch is introduced to make the model easier to converge and solve the gradient disappearance problem. In the inference phase, all network layers are converted to Conv3*3 through the Op fusion strategy, which reduces the number of parameters and calculations, improves the detection speed and facilitates deployment. RepVGG-A0 has a simple structure, is easy to design and implement, and has high efficiency in terms of model size and running speed. Its model size is only 5.5MB, and it can run quickly in resource-limited scenarios such as smart glasses.

[0058] Optionally, the RepVGG backbone network of the present application includes a shallow feature extraction stage. The shallow feature extraction stage is used to extract shallow features, and the deep feature stage is used to extract deep features. Among them, the shallow feature receptive field is small and pays more attention to detail information; the overlapping area of ​​the receptive field corresponding to each pixel point is small, ensuring that the network can capture more details, and the shallow features have higher resolution; the spatial information is rich.

[0059] In the shallow feature extraction stage, the RepVGG backbone network adopts a multi-branch structure in the training stage and a single-branch structure in the reasoning stage. This application uses the structural reparameterization technology to convert the multi-branch structure in the training stage into a single-branch structure composed of 3×3 convolution units and ReLU in the reasoning stage to improve the reasoning speed of the model. The multi-branch structure includes a first convolution branch, a second convolution branch, and an identity residual branch. Among them, the first convolution branch is a Conv3x3 convolution branch, and the second convolution branch is a Conv1x1 convolution branch.

[0060] Optionally, the collected image is input into the multi-branch structure of the shallow feature extraction stage, and the collected image is subjected to feature extraction through the shallow feature extraction stage of the RepVGG backbone network by means of the structure reparameterization technology, and the shallow features corresponding to the shallow feature extraction stage include:

[0061] First, the image will be fused through the first convolution branch, the second convolution branch and the identity residual branch. Specifically, the image is fused through the convolution layer and the normalization layer of the first convolution branch to obtain the fused first convolution branch; the image is fused through the convolution layer and the normalization layer of the second convolution branch to obtain the fused second convolution branch. Specifically, the Conv3x3 and BN layers in the first convolution branch in the residual block are first fused to obtain the fused first convolution branch. At the same time, the Conv1x1 and BN layers in the second convolution branch in the residual block are fused to obtain the fused second convolution branch.

[0062] Secondly, the fused first convolution branch, the fused second convolution branch and the identity residual branch are respectively converted into target convolution kernels of the same size, and the target convolution kernel is Conv3*3. Specifically, the fused convolution layer is converted to Conv3*3, that is, the convolutions of different convolution kernels are converted into convolutions with a convolution kernel of size 3*3. For the Conv1*1 branch, the entire conversion process is to replace the 1*1 convolution kernel with a 3*3 convolution kernel, that is, to move the value in the 1*1 convolution kernel to the center point of the 3*3 convolution kernel; for the identity residual branch, since the identity residual branch does not change the value of the input feature map, a 3*3 convolution kernel can be set, and the weight values ​​at all positions are set to 1, so after it is multiplied by the input feature map, the original value is maintained.

[0063] Finally, the target convolution kernels in all branches are merged to obtain the shallow features corresponding to the shallow feature extraction stage. Specifically, the weights and biases of all branches are superimposed to obtain a fused Conv3*3 network layer, that is, the Conv3*3 in the inference stage. Then the shallow features corresponding to the shallow feature extraction stage are obtained.

[0064] Step S120, input the shallow features into the deep feature extraction stage of the RepVGG backbone network, and sequentially pass through different convolutional layers of the multi-scale attention mechanism module of the deep feature extraction stage for convolution operations to obtain the deep features corresponding to the deep feature extraction stage.

[0065] Optionally, the RepVGG backbone network of the present application includes a deep feature extraction stage. The deep feature has a large receptive field and pays more attention to global information; the receptive field corresponding to each pixel increases, and the overlapping area between receptive fields also increases. The information represented by each pixel is the information of a region; the semantic information is rich.

[0066] In the deep feature extraction stage, the training and inference stages of the RepVGG backbone network are as follows Figure 4, a multi-branch structure is adopted in the training stage, and a single-branch structure is adopted in the reasoning stage. This application uses the structural re-parameterization technology to convert the multi-branch structure in the training stage into a single-branch structure composed of 3×3 convolution units and ReLU in the reasoning stage, thereby improving the reasoning speed of the model. The multi-branch structure includes a first convolution branch, a second convolution branch, and a residual branch. Among them, the first convolution branch is a Conv3x3 convolution branch, and the second convolution branch is a Conv1x1 convolution branch. Different from the shallow feature extraction stage, this application embeds a multi-scale attention mechanism module in the residual branch, and the attention mechanism module adopts the MSECANet network. This application improves the lightweight attention mechanism ECANet, proposes a multi-scale attention mechanism MSECANet, and adds it to the deep feature extraction stage, automatically adjusts the weights between channels to highlight key features, and solves the problem of insufficient feature information extraction of lightweight networks.

[0067] Optionally, the method of determining the deep features corresponding to the deep feature extraction stage is similar to the method of determining the shallow features, except that there is a difference in the processing process of the multi-scale attention mechanism module.

[0068] Optionally, the shallow features are input into the deep feature extraction stage of the RepVGG backbone network, and are sequentially convolutionally processed through different convolutional layers of the multi-scale attention mechanism module of the deep feature extraction stage to obtain the deep features corresponding to the deep feature extraction stage, including:

[0069] First, the shallow features are input into the residual block of the deep feature extraction stage. The residual block includes a first convolution branch, a second convolution branch and a residual branch. The shallow features are fused through the convolution layer and the normalization layer of the first convolution branch to obtain the fused first convolution branch. The shallow features are fused through the convolution layer and the normalization layer of the second convolution branch to obtain the fused second convolution branch. The shallow features are fused through different convolution layers of the multi-scale attention mechanism module to obtain the fused residual branch. Specifically, the Conv3x3 and BN layers in the first convolution branch in the residual block are first fused to obtain the fused first convolution branch. At the same time, the Conv1x1 and BN layers in the second convolution branch in the residual block are fused to obtain the fused second convolution branch. The shallow features are passed through the multi-scale attention mechanism module of the residual branch and fused with different convolution layers of the multi-scale attention mechanism module to obtain the fused residual branch.

[0070] Secondly, the convolution kernels in the fused first convolution branch, the fused second convolution branch, and the fused residual branch are respectively converted into target convolution kernels of the same size, and the target convolution kernel is Conv3*3. Specifically, the fused convolution layer is converted to Conv3*3, that is, the convolutions of different convolution kernels are converted into convolutions with a convolution kernel of size 3*3. For the Conv1*1 branch, the entire conversion process is to replace the 1*1 convolution kernel with a 3*3 convolution kernel, that is, to move the value in the 1*1 convolution kernel to the center point of the 3*3 convolution kernel; for the residual branch, since the residual branch does not change the value of the input feature map, a 3*3 convolution kernel can be set, and the weight values ​​at all 9 positions are set to 1, so after it is multiplied by the input feature map, the original value is maintained.

[0071] Finally, the target convolution kernels in all branches are merged to obtain the deep features corresponding to the deep feature extraction stage. Specifically, the Conv3*3 in the residual branch is merged, that is, the weights and biases of all branches are superimposed to obtain a fused Conv3*3 network layer, that is, the Conv3*3 in the inference stage. Then the deep features corresponding to the deep feature extraction stage are obtained.

[0072] Optionally, the shallow features are fused through different convolutional layers of the multi-scale attention mechanism module to obtain a fused residual branch, which includes: convolving the shallow features through the first convolutional layer of the multi-scale attention mechanism module to obtain a first deep feature; convolving the first deep feature with the second convolutional layer of the multi-scale attention mechanism module to obtain a second deep feature; and fusing the second deep feature with the shallow features to obtain a fused residual branch.

[0073] Among them, the multi-scale attention mechanism module embedded in the deep feature extraction stage is as follows Figure 5As shown. The network structure of the multi-scale attention mechanism module includes two layers of adaptive one-dimensional convolution layers. The convolution kernel size of the first convolution layer is k, and the convolution kernel size of the second convolution layer is 2k+1. The convolution coverage between channels is increased by two layers of one-dimensional convolution. After two layers of one-dimensional convolution operations, the final 1×1×C feature map is generated after fusion with shallow features. Multi-layer one-dimensional convolution can break the limitations of single-layer convolution channel interaction, improve the cross-channel interaction coverage, strengthen the overall fusion between feature map channels, optimize the attention weight distribution, and facilitate the extraction of important features. The multi-scale attention mechanism module of the present application adopts a combination of global average pooling and global maximum pooling to reduce the dimension of the input feature map, and more completely retains high-level features. On the basis of the original one-dimensional convolution layer, another one-dimensional convolution layer is added to improve the degree of fusion between peripheral channels. Among them, the first deep feature is the feature of the shallow feature after the convolution operation of the first convolution layer, and the second deep feature is the feature of the first deep feature after the convolution operation of the second convolution layer.

[0074] Step S130, fusing the shallow features and the deep features to obtain fused features.

[0075] Step S140: performing target detection based on the fusion features.

[0076] Optionally, after obtaining the shallow features and the deep features, the shallow features and the deep features are input into a feature fusion network, and the deep features are upsampled and fused with the shallow features to obtain the fused features. The purpose of feature fusion is to merge the features extracted from the image into a feature that has better discriminative power than the input features. This application performs target detection based on the fused features, which is more conducive to the judgment of target detection.

[0077] Optionally, refer to Figure 2 After the shallow features and deep features are fused, in addition to obtaining a fused feature map of size 80x80x256, feature maps of other sizes can also be obtained, including feature maps of sizes 10x10x256, 20x20x256, and 40x40x256.

[0078] Optionally, a prediction head network is used to predict feature maps of different sizes to determine the target. The processing of the prediction head network is not the focus of this solution and will not be described here.

[0079] According to the above technical solution, this embodiment adopts a lightweight RepVGG backbone network to replace the backbone network of the FCOS anchor-free target detection algorithm, reduces the number of model parameters and calculations to achieve real-time target detection, and adds an attention mechanism in the deep feature extraction stage to improve detection accuracy. While maintaining the detection accuracy, the model can run efficiently on mobile devices.

[0080] Second embodiment

[0081] Based on the first embodiment, in a second embodiment of the present application, the target detection method of the present application includes the following steps:

[0082] Step S111, subjecting the collected image to the first shallow feature extraction stage and the second shallow feature extraction stage in sequence for feature extraction, to obtain shallow features corresponding to the shallow feature extraction stages.

[0083] Optionally, the shallow feature extraction stage of the present application includes a first shallow feature extraction stage and a second shallow feature extraction stage, and the resolution of the first shallow feature extraction stage is higher than that of the second shallow feature extraction stage. If feature fusion is performed in the first shallow feature extraction stage, there will be a problem of large amount of calculation. Therefore, the present application only performs feature fusion in the second shallow feature extraction stage.

[0084] Optionally, in order to extract as much different feature information as possible for analysis, the first shallow feature extraction stage and the second shallow feature extraction stage can be further subdivided into multiple stages according to actual conditions. Optionally, the present application takes the first shallow feature extraction stage including the first feature extraction stage and the second feature extraction stage, and the second shallow feature extraction stage including the third feature extraction stage and the fourth feature extraction stage as an example. The backbone network formed by each shallow feature extraction stage and deep feature extraction stage is as follows: Figure 2 As shown. Among them, the first feature extraction stage, the second feature extraction stage, the third feature extraction stage and the fourth feature extraction stage correspond to Figure 2 C1, C2, C3 and C4 in the figure are all shallow feature extraction stages. C5 is the deep feature extraction stage.

[0085] Optionally, the image captured by the smart glasses is subjected to feature extraction in the first feature extraction stage and the second feature extraction stage in sequence to obtain a first feature; the first feature is subjected to feature extraction in the third feature extraction stage to obtain a shallow feature corresponding to the third feature extraction stage; the shallow feature corresponding to the third feature extraction stage is subjected to feature extraction in the fourth feature extraction stage to obtain a shallow feature corresponding to the fourth feature extraction stage; the first feature, the shallow features corresponding to the third feature extraction stage, and the shallow features corresponding to the fourth feature extraction stage are used as shallow features corresponding to the shallow feature extraction stage. Among them, the first feature is the feature output by the C2 stage, the shallow feature corresponding to the third feature extraction stage is the feature output by the C3 stage, and the shallow feature corresponding to the fourth feature extraction stage is the feature output by the C4 stage. Different features in the image captured by the smart glasses are extracted through feature extraction at different stages.

[0086] Step S121, input the shallow features corresponding to the second shallow feature extraction stage into the deep feature extraction stage of the RepVGG backbone network, and perform convolution operations on different convolutional layers of the multi-scale attention mechanism module of the deep feature extraction stage to obtain the deep features corresponding to the deep feature extraction stage.

[0087] Optionally, the present application adds a multi-scale attention mechanism module in the last stage, i.e., the deep feature extraction stage. Specifically, the shallow features corresponding to the fourth feature extraction stage in the second shallow feature extraction stage can be input into the deep feature extraction stage of the RepVGG backbone network, and convolution operations are performed through different convolutional layers of the multi-scale attention mechanism module of the deep feature extraction stage to obtain the deep features corresponding to the deep feature extraction stage.

[0088] Step S130, fusing the shallow features and the deep features to obtain fused features.

[0089] Optionally, the first feature, the shallow features corresponding to the third feature extraction stage and the shallow features corresponding to the fourth feature extraction stage can be used as the shallow features corresponding to the shallow feature extraction stage, and the shallow features and the deep features are fused to obtain fused features.

[0090] Optionally, since the resolution of the first feature extraction stage and the second feature extraction stage is relatively high, there is a problem of large amount of calculation if the first feature is used for feature fusion. Therefore, the present application may only input the shallow features corresponding to the third feature extraction stage, the shallow features and deep features corresponding to the fourth feature extraction stage into the feature fusion network for feature fusion, obtain the fused features, reduce the amount of calculation, and improve the target detection speed.

[0091] Step S140: performing target detection based on the fusion features.

[0092] According to the above technical solution, this embodiment adopts a lightweight RepVGG backbone network to replace the backbone network of the FCOS anchor-free target detection algorithm, reduces the model parameters and calculation amount to achieve real-time target detection, and adds an attention mechanism in the deep feature extraction stage to improve the detection accuracy. Under the premise of maintaining the detection accuracy, the model can run efficiently on mobile devices. In addition, this application takes into account that the resolution of the first feature extraction stage and the second feature extraction stage is relatively high. If the first feature is used for feature fusion, there is a problem of large amount of calculation. This application can only input the shallow features corresponding to the third feature extraction stage, the shallow features corresponding to the fourth feature extraction stage, and the deep features into the feature fusion network for feature fusion, and obtain the fused features, which reduces the amount of calculation and improves the target detection speed.

[0093] The embodiments of the present invention provide embodiments of the target detection method. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different from that shown here.

[0094] like Figure 6 As shown, the present application provides a target detection system, comprising:

[0095] The shallow feature extraction module 10 is used to extract features from the collected image through the shallow feature extraction stage of the RepVGG backbone network to obtain shallow features corresponding to the shallow feature extraction stage.

[0096] Optionally, the shallow feature extraction module 10 is used to perform feature extraction on the collected image in the first shallow feature extraction stage and the second shallow feature extraction stage in sequence to obtain shallow features corresponding to the shallow feature extraction stage.

[0097] Optionally, the shallow feature extraction module 10 is used to perform feature extraction on the collected image in the first feature extraction stage and the second feature extraction stage in sequence to obtain a first feature; perform feature extraction on the first feature in the third feature extraction stage to obtain a shallow feature corresponding to the third feature extraction stage; perform feature extraction on the shallow feature corresponding to the third feature extraction stage in the fourth feature extraction stage to obtain a shallow feature corresponding to the fourth feature extraction stage; and use the first feature, the shallow features corresponding to the third feature extraction stage, and the shallow features corresponding to the fourth feature extraction stage as the shallow features corresponding to the shallow feature extraction stage.

[0098] The deep feature extraction module 20 is used to input the shallow features into the deep feature extraction stage of the RepVGG backbone network, and sequentially pass through different convolutional layers of the multi-scale attention mechanism module of the deep feature extraction stage for convolution operations to obtain the deep features corresponding to the deep feature extraction stage.

[0099] Optionally, the deep feature extraction module 20 is used to input the shallow features into the residual block of the deep feature extraction stage, the residual block includes a first convolution branch, a second convolution branch and a residual branch, the shallow features are fused through the convolution layer and the normalization layer of the first convolution branch respectively to obtain a fused first convolution branch, the shallow features are fused through the convolution layer and the normalization layer of the second convolution branch respectively to obtain a fused second convolution branch, the shallow features are fused through different convolution layers of the multi-scale attention mechanism module to obtain a fused residual branch; the convolution kernels in the fused first convolution branch, the fused second convolution branch and the fused residual branch are respectively converted into target convolution kernels of the same size; the target convolution kernels in all branches are merged to obtain the deep features corresponding to the deep feature extraction stage.

[0100] Optionally, the deep feature extraction module 20 is used to perform a convolution operation on the shallow feature through the first convolution layer of the multi-scale attention mechanism module to obtain a first deep feature; use the second convolution layer of the multi-scale attention mechanism module to perform a convolution operation on the first deep feature to obtain a second deep feature; and fuse the second deep feature with the shallow feature to obtain a fused residual branch.

[0101] The fusion module 30 is used to fuse the shallow features and the deep features to obtain fused features.

[0102] Optionally, the fusion module 30 is used to input the shallow features corresponding to the third feature extraction stage, the shallow features corresponding to the fourth feature extraction stage and the deep features into the feature fusion network for feature fusion to obtain the fused features.

[0103] Optionally, the fusion module 30 is used to input the shallow features and the deep features into a feature fusion network, and the deep features are upsampled and then fused with the shallow features to obtain the fused features.

[0104] The target detection module 40 is used to perform target detection based on the fused features.

[0105] The specific implementation of the target detection system of the present invention is basically the same as the above-mentioned embodiments of the target detection method, and will not be repeated here.

[0106] like Figure 7 As shown, Figure 7 Schematic diagram of the structure of the hardware operating environment of a mobile device. The mobile device may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen, an input unit such as a keyboard, and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory, such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0107] Those skilled in the art will understand that Figure 7 The structure of the mobile device shown in the figure does not constitute a limitation on the mobile device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0108] like Figure 7 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a target detection program. The operating system is a program that manages and controls the hardware and software resources of the mobile device, the target detection program, and the operation of other software or programs.

[0109] exist Figure 7 In the mobile device shown, the user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used for the background server and communicates data with the background server; the processor 1001 can be used to call the target detection program stored in the memory 1005.

[0110] In this embodiment, the mobile device includes: a memory 1005, a processor 1001, and a target detection program stored in the memory and executable on the processor, wherein:

[0111] When the processor 1001 calls the target detection program stored in the memory 1005, the following operations are performed:

[0112] The collected image is subjected to feature extraction in the shallow feature extraction stage of the RepVGG backbone network to obtain shallow features corresponding to the shallow feature extraction stage;

[0113] The shallow features are input into the deep feature extraction stage of the RepVGG backbone network, and convolution operations are performed on different convolutional layers of the multi-scale attention mechanism module of the deep feature extraction stage to obtain the deep features corresponding to the deep feature extraction stage;

[0114] Fusing the shallow features with the deep features to obtain fused features;

[0115] Target detection is performed based on the fused features.

[0116] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a target detection program. When the target detection program is executed by a processor, it implements the various steps of the target detection method described above and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0117] Since the storage medium provided in the embodiments of the present application is the storage medium used to implement the method of the embodiments of the present application, based on the method introduced in the embodiments of the present application, the person skilled in the art can understand the specific structure and deformation of the storage medium, so it is not repeated here. All storage media used in the methods of the embodiments of the present application belong to the scope of protection of this application.

[0118] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0119] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, TV, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0121] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A target detection method, characterized in that: The target detection method comprises: The collected image is subjected to feature extraction in the shallow feature extraction stage of the RepVGG backbone network to obtain shallow features corresponding to the shallow feature extraction stage; The shallow features are input into the deep feature extraction stage of the RepVGG backbone network, and convolution operations are performed on different convolutional layers of the multi-scale attention mechanism module of the deep feature extraction stage to obtain the deep features corresponding to the deep feature extraction stage; Fusing the shallow features with the deep features to obtain fused features; Target detection is performed based on the fused features.

2. The target detection method according to claim 1, characterized in that: The steps of inputting the shallow features into the deep feature extraction stage of the RepVGG backbone network, and performing convolution operations on different convolutional layers of the multi-scale attention mechanism module of the deep feature extraction stage to obtain the deep features corresponding to the deep feature extraction stage include: Inputting the shallow features into the residual block of the deep feature extraction stage, the residual block includes a first convolution branch, a second convolution branch and a residual branch, the shallow features are fused through the convolution layer and the normalization layer of the first convolution branch to obtain a fused first convolution branch, the shallow features are fused through the convolution layer and the normalization layer of the second convolution branch to obtain a fused second convolution branch, the shallow features are fused through different convolution layers of the multi-scale attention mechanism module to obtain a fused residual branch; The convolution kernels in the fused first convolution branch, the fused second convolution branch, and the fused residual branch are respectively converted into target convolution kernels of the same size; The target convolution kernels in all branches are merged to obtain the deep features corresponding to the deep feature extraction stage.

3. The target detection method according to claim 2, characterized in that: The shallow features are fused through different convolutional layers of the multi-scale attention mechanism module to obtain the fused residual branch, which includes: Convolution operation is performed on the shallow features through the first convolution layer of the multi-scale attention mechanism module to obtain the first deep features; Using the second convolutional layer of the multi-scale attention mechanism module to perform a convolution operation on the first deep feature to obtain a second deep feature; The second deep feature and the shallow feature are fused to obtain a fused residual branch.

4. The target detection method according to claim 1, characterized in that: The shallow feature extraction stage includes a first shallow feature extraction stage and a second shallow feature extraction stage. The step of extracting features of the acquired image through the shallow feature extraction stage of the RepVGG backbone network to obtain shallow features corresponding to the shallow feature extraction stage includes: The collected image is sequentially subjected to the first shallow feature extraction stage and the second shallow feature extraction stage for feature extraction, to obtain shallow features corresponding to the shallow feature extraction stage; The step of inputting the shallow features into the deep feature extraction stage of the RepVGG backbone network comprises: The shallow features corresponding to the second shallow feature extraction stage are input into the deep feature extraction stage of the RepVGG backbone network.

5. The target detection method according to claim 4, characterized in that: The first shallow feature extraction stage includes a first feature extraction stage and a second feature extraction stage, the second shallow feature extraction stage includes a third feature extraction stage and a fourth feature extraction stage, and the step of extracting features from the acquired image through the first shallow feature extraction stage and the second shallow feature extraction stage in sequence to obtain shallow features corresponding to the shallow feature extraction stage includes: Extract features from the acquired image in the first feature extraction stage and the second feature extraction stage in sequence to obtain a first feature; Extract the first feature through the third feature extraction stage to obtain a shallow feature corresponding to the third feature extraction stage; Extracting the shallow features corresponding to the third feature extraction stage through the fourth feature extraction stage to obtain the shallow features corresponding to the fourth feature extraction stage; The first feature, the shallow features corresponding to the third feature extraction stage, and the shallow features corresponding to the fourth feature extraction stage are used as the shallow features corresponding to the shallow feature extraction stage.

6. The target detection method according to claim 5, characterized in that: The step of inputting the shallow features and the deep features into a feature fusion network for feature fusion to obtain fused features comprises: The shallow features corresponding to the third feature extraction stage, the shallow features corresponding to the fourth feature extraction stage, and the deep features are input into the feature fusion network for feature fusion to obtain the fused features.

7. The target detection method according to claim 1, characterized in that: The step of inputting the shallow features and the deep features into a feature fusion network for feature fusion to obtain fused features comprises: The shallow features and the deep features are input into a feature fusion network, and the deep features are upsampled and fused with the shallow features to obtain the fused features.

8. A target detection system, characterized in that: The target detection system comprises: The shallow feature extraction module is used to extract features from the collected images through the shallow feature extraction stage of the RepVGG backbone network to obtain shallow features corresponding to the shallow feature extraction stage; A deep feature extraction module is used to input the shallow features into the deep feature extraction stage of the RepVGG backbone network, and sequentially pass through different convolutional layers of the multi-scale attention mechanism module of the deep feature extraction stage to perform convolution operations to obtain the deep features corresponding to the deep feature extraction stage; A fusion module, used for fusing the shallow features with the deep features to obtain fused features; The target detection module is used to perform target detection based on the fusion features.

9. A mobile device, characterized in that: The mobile device comprises: a memory, a processor, and a target detection program stored in the memory and executable on the processor, wherein the target detection program implements the steps of the target detection method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a target detection program, and when the target detection program is executed by a processor, the steps of the target detection method described in any one of claims 1 to 7 are implemented.

Images