Model training method, target detection method, device and readable storage medium

Abstract

The application provides a model training method, a target detection method, a target detection device and a computer readable storage medium. The model training method comprises the following steps: fusing a first stage processing layer into one channel-by-channel convolution layer, wherein the model parameters of the channel-by-channel convolution layer are obtained by superimposing a first model parameter, a second model parameter and a third model parameter; fusing a second stage processing layer into one convolution layer, wherein the model parameters of the convolution layer are obtained by superimposing a fourth model parameter and a fifth model parameter; and replacing each processing layer in a to-be-trained model with the processing layer composed of the fused channel-by-channel convolution layer and the convolution layer. In this way, when the target detection device is trained, a multi-path structure is used to pursue high performance and high precision of the model; when inference is performed, the network is reconstructed based on the structure reparameterization idea, and is converted into a single-path structure, so that the high performance advantage of the multi-branch structure network is retained, and the model parameter quantity, the memory resources required for running the model and the computing resources are reduced.

Description

Technical Field

[0001] This application relates to the field of lightweight model technology, and in particular to a model training method, an object detection method, an object detection device, and a computer-readable storage medium. Background Technology

[0002] In recent years, with the rapid development of deep learning, object detection has become one of the most important research directions in computer vision, leading to the proposal of various high-performance object detection network architectures. Simultaneously, the concept of model reparameterization has received widespread attention, with networks such as RepVGG, ACNet, and MobileOne focusing on reparameterized networks. The concept of reparameterization offers significant advantages in model compression, lightweighting, and improving inference speed on mobile platforms.

[0003] In various practical applications, such as mobile robots and security monitoring, commonly used large-scale parameter detection models are limited by the computing resources, storage resources, and running speed of mobile platforms, resulting in low feasibility, deployment difficulties, and limited implementation. A common problem is that high-accuracy models require excessive computing and storage resources, leading to slow inference speeds. While much research has focused on addressing model size issues, bottlenecks remain in model compression, insufficient consideration of required computing and storage resources, and difficulties in practical deployment on mobile devices. Summary of the Invention

[0004] This application provides a model training method, an object detection method, an object detection device, and a computer-readable storage medium.

[0005] This application provides a model training method, the model training method comprising:

[0006] Obtain the model parameters of each processing layer after the model to be trained is trained, wherein the processing layer includes a first-stage processing layer and a second-stage processing layer.

[0007] Based on the model parameters, obtain the first model parameters of the first channel-wise convolutional layer branch, the second model parameters of the second channel-wise convolutional layer branch, and the third model parameters of the first batch normalization layer branch in the first stage processing layer.

[0008] The first stage processing layer is converted into a single-path structure and fused into a channel-wise convolutional layer, wherein the model parameters of the channel-wise convolutional layer are obtained by superimposing the first model parameters, the second model parameters and the third model parameters;

[0009] Based on the model parameters, obtain the fourth model parameters of the convolutional layer branch in the second stage processing layer, and the fifth model parameters of the second batch normalization layer branch.

[0010] The second-stage processing layer is converted into a single-path structure and fused into a convolutional layer, wherein the model parameters of the convolutional layer are obtained by superimposing the fourth model parameters and the fifth model parameters;

[0011] The processing layer composed of the fused channel-wise convolutional layer and the fused convolutional layer replaces each processing layer in the model to be trained, and outputs the replaced target detection model.

[0012] Before obtaining the model parameters of each processing layer after training the model to be trained, the model training method further includes:

[0013] Remove the auxiliary head and its model parameters from the model to be trained;

[0014] Delete the network structure and model parameters of the processing layer corresponding to the auxiliary head.

[0015] The model training method further includes, after obtaining the model parameters of each processing layer of the model to be trained, as follows:

[0016] Obtain the combination of convolutional layers and batch normalization layers connected in the processing layer;

[0017] The convolutional layer and the batch normalization layer are merged into a single convolutional layer;

[0018] The eighth model parameter of the fused convolutional layer is calculated using the sixth model parameter of the convolutional layer and the seventh model parameter of the batch normalization layer within the combination.

[0019] The step of calculating the eighth model parameters of the fused convolutional layer using the sixth model parameters of the convolutional layer and the seventh model parameters of the batch normalization layer includes:

[0020] Obtain the convolutional layer weight parameters and convolutional layer bias parameters of the sixth model parameters;

[0021] Obtain the mean and variance of the batch normalization layer of the seventh model parameters;

[0022] The weight parameters in the eighth model parameters are obtained using the convolutional layer weight parameters, the variance, and the first fusion factor.

[0023] The bias parameters in the eighth model parameters are obtained using the convolutional layer bias parameters, the mean, the variance, and the second fusion factor.

[0024] The step of converting the first-stage processing layer into a single-path structure and fusing it into a channel-wise convolutional layer includes:

[0025] Merge several channel-wise convolutional layers in the first channel-wise convolutional layer branch into a single branch channel-wise convolutional layer;

[0026] The kernel size of the branch channel-wise convolutional layer, the kernel size of the second channel-wise convolutional layer branch, and the kernel size of the first batch normalization layer branch are filled to a uniform kernel size.

[0027] The filled branch of the channel-wise convolutional layer, the filled second channel-wise convolutional layer branch, and the filled first batch normalization layer branch are merged into a single channel-wise convolutional layer.

[0028] The model to be trained includes a backbone network module, a feature enhancement and extraction module, an auxiliary head, and a guide head. The backbone network module and the feature enhancement and extraction module each include several processing layers.

[0029] The model training method also includes:

[0030] The image to be trained is input into the backbone network module, and several processing layers stacked sequentially in the backbone network module are used to extract output feature maps with different downsampling resolutions, including a first output feature map, a second output feature map and a third output feature map.

[0031] The first output feature map, the second output feature map, and the third output feature map are input into the feature enhancement and extraction module. The third output feature map and the second output feature map are fused along the channel dimension to obtain the fourth output feature map.

[0032] The fourth output feature map is fused with the first output feature map along the channel dimension to obtain the first candidate feature map;

[0033] The first candidate feature map and the third output feature map are fused along the channel dimension to obtain the second candidate feature map;

[0034] The second candidate feature map and the fourth output feature map are fused along the channel dimension to obtain the third candidate feature map;

[0035] The processing layer is used to extract features from the first candidate feature map, the second candidate feature map, and the third candidate feature map to obtain the first predicted feature map, the second predicted feature map, and the third predicted feature map.

[0036] The processing layer is used to extract features from the first candidate feature map, the fourth output feature map, and the third output feature map, respectively, to obtain the fourth prediction feature map, the fifth prediction feature map, and the sixth prediction feature map.

[0037] The first predicted feature map, the second predicted feature map, and the third predicted feature map are input into the guide head for prediction, and the first loss function value of the guide head is obtained.

[0038] The fourth, fifth, and sixth predicted feature maps are input into the auxiliary head for prediction, and the second loss function value of the auxiliary head is obtained.

[0039] The model to be trained is trained using the first loss function value and the second loss function value to obtain a multi-path detection network.

[0040] The step of inputting the first predicted feature map, the second predicted feature map, and the third predicted feature map into the guidance head for prediction, and obtaining the first loss function value of the guidance head, includes:

[0041] The first prediction feature map, the second prediction feature map, and the third prediction feature map are respectively processed into a grid to obtain several predicted bounding box coordinates and prediction categories;

[0042] According to each prediction category, obtain the grid where the center point of the real bounding box is located and its first preset number of neighboring grids corresponding to the coordinates of each predicted target box, as guiding positive samples for matching.

[0043] Obtain the cost loss of each guiding positive sample, sort them according to the cost loss, obtain a first preset number of guiding positive samples and input them into the guiding head for prediction, and obtain the first loss function value of the guiding head.

[0044] The auxiliary positive samples for the input auxiliary head are obtained by searching for auxiliary matching positive samples in the fourth, fifth, and sixth prediction feature maps according to a third preset number of adjacent grids, and sorting them according to the cost loss to obtain a fourth preset number of auxiliary positive samples.

[0045] Wherein, the third preset quantity is greater than the first preset quantity, and the fourth preset quantity is greater than the second preset quantity.

[0046] Before performing gridding processing on the first predicted feature map, the second predicted feature map, and the third predicted feature map, the model training method further includes:

[0047] Obtain the number of prior anchor boxes and the number of categories in each of the first, second, and third predicted feature maps;

[0048] The number of channels for each predicted feature map is obtained according to the number of prior anchor boxes and the number of categories;

[0049] Each predicted feature map is processed by a convolutional layer according to the number of channels in each predicted feature map.

[0050] This application also provides a target detection method, the target detection method comprising:

[0051] Input the image to be detected into the mobile platform;

[0052] The detection result of the target to be detected is obtained using a preset target detection model in the mobile platform;

[0053] The target detection model is trained using the model training method described above.

[0054] This application also provides a target detection device, which includes a processor and a memory. The memory stores program data, and the processor executes the program data to implement the model training method and / or the target detection method described above.

[0055] This application also provides a computer-readable storage medium for storing program data, which, when executed by a processor, is used to implement the above-described model training method and / or object detection method.

[0056] The beneficial effects of this application are as follows: The object detection device obtains the model parameters of each processing layer after the training of the model to be trained, wherein the processing layer includes a first-stage processing layer and a second-stage processing layer; based on the model parameters, it obtains the first model parameters of the first channel-wise convolutional layer branch, the second model parameters of the second channel-wise convolutional layer branch, and the third model parameters of the first batch normalization layer branch in the first-stage processing layer; it converts the first-stage processing layer into a single-path structure and fuses it into a single channel-wise convolutional layer, wherein the model parameters of the single-path convolutional layer are obtained by superimposing the first model parameters, the second model parameters, and the third model parameters; based on the model parameters, it obtains the fourth model parameters of the convolutional layer branch in the second-stage processing layer and the fifth model parameters of the second batch normalization layer branch; it converts the second-stage processing layer into a single-path structure and fuses it into a single convolutional layer, wherein the model parameters of the convolutional layer are obtained by superimposing the fourth model parameters and the fifth model parameters; it replaces each processing layer in the model to be trained with the processing layer composed of the fused channel-wise convolutional layer and the fused convolutional layer, and outputs the replaced object detection model. In this way, the target detection device uses a multi-path structure during training to pursue high performance and high accuracy of the model; during inference, it reconstructs the network based on the idea of ​​structural reparameterization and converts it into a single-path structure, which retains the high performance advantages of multi-branch network structure and reduces the number of model parameters and the memory and computing resources required to run the model. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0058] Figure 1 This is a flowchart illustrating an embodiment of the model training method provided in this application;

[0059] Figure 2 This is a schematic diagram of the overall process of the lightweight target detection method for mobile platforms provided in this application;

[0060] Figure 3 This is a schematic diagram of the framework of the multi-path target detection training network provided in this application;

[0061] Figure 4 yes Figure 3 A schematic diagram of the MobileOne-Block framework;

[0062] Figure 5 This is a flowchart illustrating another embodiment of the model training method provided in this application;

[0063] Figure 6 This is a schematic diagram of the framework of the single-path target detection training network provided in this application;

[0064] Figure 7 This is a flowchart illustrating an embodiment of the target detection method provided in this application;

[0065] Figure 8 This is a schematic diagram of the structure of an embodiment of the target detection device provided in this application;

[0066] Figure 9 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0068] Please refer to details. Figure 1 and Figure 2 , Figure 1This is a flowchart illustrating an embodiment of the model training method provided in this application. Figure 2 This is a schematic diagram of the overall process of the lightweight target detection method for mobile platforms provided in this application.

[0069] The model training method of this application is applied to a target detection device, which can be a server or a system consisting of a server and a terminal device working together. Accordingly, the various parts of the target detection device, such as units, subunits, modules, and submodules, can be all located in the server, or they can be located separately in the server and the terminal device.

[0070] Furthermore, the aforementioned server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules, such as software or software modules used to provide distributed servers, or as a single software program or software module; no specific limitation is made here. In some possible implementations, the model training method of this application embodiment can be implemented by a processor calling computer-readable instructions stored in memory.

[0071] Specifically, such as Figure 1 As shown, the model training method in this embodiment of the application specifically includes the following steps:

[0072] Step S11: Input the image to be trained into the backbone network module, and use the processing layers stacked sequentially in the backbone network module to extract output feature maps with different downsampling resolutions, including the first output feature map, the second output feature map and the third output feature map.

[0073] In this embodiment of the application, the target detection device acquires several target image data, annotates the target detection bounding boxes and their target detection categories, thereby constructing a training dataset and a test dataset.

[0074] This application uses the YOLOv7 framework and MobileOne network as its foundation, and leverages the concept of structural reparameterization to construct a multi-path object detection training network. For details, please refer to [link / reference needed]. Figure 3 , Figure 3 This is a schematic diagram of the framework of the multi-path target detection training network provided in this application.

[0075] like Figure 3 As shown, the target detection training network in this embodiment includes a backbone network module, a feature enhancement and extraction module, and a detection module, wherein the detection module includes a guide head and an auxiliary head.

[0076] Specifically, the backbone network module of this application is based on the MobileOne network, and its core module is MobileOne-Block. An improved and optimized MobileOneLite network structure has been established, comprising five parts: Stage 1, Stage 2, Stage 3, Stage 4, and Stage 5. Detailed network structure parameters of the MobileOneLite network structure are shown in Table 1 below.

[0077] Table 1 MobileOneLite Network Structure Parameters

[0078]

[0079]

[0080] The backbone network module is based on MobileOne, and the number of MobileOne-Block channels in each stage has been reduced to avoid excessive redundant channel features. Furthermore, compared to the MobileOne network, the MobileOneLite network removes Stage 5 of the MobileOne network and increases the number of MobileOne-Blocks in its Stage 6, which is then incorporated as Stage 5 of MobileOneLite, to ensure deep feature extraction capabilities and reduce the number of model parameters.

[0081] The core of the backbone network module is MobileOne-Block, which integrates depthwise separable convolution and structural reparameterization. Its specific structure is as follows: Figure 4 As shown, each MobileOne-Block consists of two phases: the first phase includes a 1×1 depthwise convolution (DWConv) branch, a 3×3 DWConv branch controlled by the hyperparameter k, and a batch normalization (BN) layer branch. In the MobileOneLite network, the hyperparameter k = 4, and the activation function used is the ReLU function, which has low computational latency.

[0082] Furthermore, the second stage includes a 1×1 ordinary convolutional branch and a batch normalization (BN) layer branch, where the activation function used is also the ReLU function, which has low computational latency.

[0083] The backbone network module uses Stage 1 to Stage 5 to extract features from the input image data and outputs output feature maps with different downsampling resolutions. Specifically, the first output feature map F1 output by Stage 3 corresponds to 8 times downsampling resolution; the second output feature map F2 output by Stage 4 corresponds to 16 times downsampling resolution; and the third output feature map F3 output by Stage 5 corresponds to 32 times downsampling resolution.

[0084] Step S12: Input the first output feature map, the second output feature map and the third output feature map into the feature enhancement and extraction module, and fuse the third output feature map with the second output feature map in the channel dimension to obtain the fourth output feature map.

[0085] In this embodiment, the feature enhancement extraction module integrates the ideas of MobileOne-Block and the path aggregation network PANet, constructing a YOLOv7-Lite-Rep network structure. The MobileOne-Block hyperparameter k = 4.

[0086] Specifically, the feature enhancement extraction module uses a regular convolutional layer with a kernel of 1×1 and 48 channels to perform channel dimensionality reduction on the third output feature map F3 output by the backbone network module to obtain feature map S1-1; further, the feature map is upsampled by nearest neighbor interpolation to obtain feature map S1-2.

[0087] The feature enhancement and extraction module performs channel-dimensional fusion of feature map S1-2 and the second output feature map F2 from the backbone network module. Further, a MobileOne-Block layer with 48 channels and a stride of 1 is used for feature extraction. Then, a regular convolutional layer with 11 kernels and 48 channels is used to extract features from the output feature map, resulting in feature map S3-1. Finally, nearest-neighbor interpolation upsampling is performed on feature map S3-1 to obtain feature map S3-2, which is the fourth output feature map.

[0088] Step S13: Merge the fourth output feature map with the first output feature map along the channel dimension to obtain the first candidate feature map.

[0089] In this embodiment, the feature enhancement extraction module performs channel-dimensional fusion of the fourth output feature map and the first output feature map F1 output by the backbone network module. Furthermore, it uses a MobileOne-Block layer with 48 channels and a stride of 1 to extract features and outputs the first candidate feature map R1.

[0090] Step S14: Merge the first candidate feature map and the third output feature map along the channel dimension to obtain the second candidate feature map.

[0091] In this embodiment, the feature enhancement extraction module uses a MobileOne-Block layer with 48 channels and a stride of 2 to downsample and extract features from the first candidate feature map R1. Then, it performs channel-dimensional fusion with the feature map S1-1. Further, it uses a MobileOne-Block layer with 48 channels and a stride of 1 to extract features and outputs the second candidate predicted feature map R2.

[0092] Step S15: Merge the second candidate feature map and the fourth output feature map along the channel dimension to obtain the third candidate feature map.

[0093] In this embodiment, the feature enhancement extraction module uses a MobileOne-Block layer with 48 channels and a stride of 2 to downsample and extract features from the second candidate feature map R2. Then, it performs channel-dimensional fusion with the feature map S3-1. Further, it uses a MobileOne-Block layer with 48 channels and a stride of 1 to extract features and outputs the third candidate feature map R3.

[0094] Step S16: Use the processing layer to extract features from the first candidate feature map, the second candidate feature map, and the third candidate feature map respectively, and obtain the first predicted feature map, the second predicted feature map, and the third predicted feature map.

[0095] In this embodiment, the feature enhancement extraction module performs feature extraction on the first candidate feature map R1, the second candidate feature map R2, and the third candidate feature map R3 using a MobileOne-Block layer with 48 channels and a stride of 1, respectively, to obtain the first predicted feature map P1, the second predicted feature map P2, and the third predicted feature map P3.

[0096] Step S17: Use the processing layer to extract features from the first candidate feature map, the fourth output feature map, and the third output feature map respectively, to obtain the fourth prediction feature map, the fifth prediction feature map, and the sixth prediction feature map.

[0097] In this embodiment, the feature enhancement extraction module performs feature extraction on the first candidate feature maps R1, S3-1, and S1-1 using a MobileOne-Block layer with 48 channels and a stride of 1, respectively, to obtain the fourth predicted feature map P4, the fifth predicted feature map P5, and the fifth predicted feature map P6.

[0098] Step S18: Input the first predicted feature map, the second predicted feature map, and the third predicted feature map into the guide head for prediction, and obtain the first loss function value of the guide head.

[0099] Step S19: Input the fourth, fifth, and sixth predicted feature maps into the auxiliary head for prediction and obtain the second loss function value of the auxiliary head.

[0100] In this embodiment, the guiding head sequentially processes the first predicted feature map P1, the second predicted feature map P2, and the third predicted feature map P3 using convolutional layers with a kernel of 1×1 to obtain feature maps of different scales. The number of channels in this convolutional layer is determined based on Formula 1:

[0101] C = Num priors *(5+Num classes (1)

[0102] Where C represents the number of channels in this layer, and Num priors Num represents the number of prior anchor boxes assigned to each feature map layer. classes This indicates the number of categories contained in the target dataset. Similarly, the auxiliary head processes the fourth predicted feature map P4, the fifth predicted feature map P5, and the sixth predicted feature map P6 sequentially using convolutional layers with a kernel of 1×1.

[0103] Furthermore, the detection module performs gridding on each feature map to obtain several predicted target box coordinates and categories. Based on the prior anchor boxes and ground truth bounding boxes (GT), positive and negative samples are assigned according to the cross-grid matching strategy with aspect ratio threshold and the SimOTA sample allocation strategy. Then, the bounding box loss and category loss are calculated using the CIoU (Complete-IoU) loss function and the cross-entropy loss function, respectively.

[0104] In the positive and negative sample allocation strategy of the guide head, the aspect ratio threshold is set to 4. The two grid cells (top, bottom, left, and right) closest to the center point of the ground truth (GT) box outside the grid in the feature map are selected as the three matching positive samples. Further, SimOTA selects the top k (k=10) samples based on the cost loss, calculates the IoU sum, and rounds it down. Finally, it sorts the samples according to the cost loss to obtain several corresponding positive samples assigned to each GT.

[0105] Accordingly, in the positive and negative sample allocation strategy of the auxiliary head, the aspect ratio threshold remains unchanged. Five grids—the top, bottom, left, and right grids—are selected as matching positive samples when the center point of the ground truth bounding box is outside the grid in the feature map. Furthermore, SimOTA selects the top k (k=20) samples based on the cost loss, calculates the IoU sum, and rounds it down. Finally, it sorts the samples according to the cost loss to obtain several corresponding positive samples assigned to each ground truth bounding box.

[0106] Step S10: Train the model to be trained using the first loss function value and the second loss function value.

[0107] In this embodiment, the detection module adds the calculated losses of the auxiliary head and the guiding head at a ratio of 0.25:1, and updates the weight parameters of each network node using the stochastic gradient descent method until the loss value converges, thus obtaining a trained lightweight object detection training network with a multi-path structure.

[0108] Based on the above training process, this application can obtain a lightweight object detection training network with a multi-path structure. Furthermore, it is necessary to use the concept of structural reparameterization to convert the multi-path training network into a single-path inference network.

[0109] Please continue reading. Figure 5 , Figure 5 This is a flowchart illustrating another embodiment of the model training method provided in this application.

[0110] Specifically, such as Figure 5 As shown, the model training method in this embodiment of the application specifically includes the following steps:

[0111] Step S21: Obtain the model parameters of each processing layer after the model to be trained, wherein the processing layer includes the first-stage processing layer and the second-stage processing layer.

[0112] In the embodiments of this application, such as Figure 2 As shown, the target detection device utilizes the concept of structural reparameterization to transform the training network into a lightweight inference network with a single-path structure.

[0113] Specifically, after the target detection device has completed training the model to be trained, it removes the auxiliary head and the network structure and model parameters of the corresponding processing layer from the model to be trained. For details on the model structure before conversion, please refer to [link to relevant documentation]. Figure 3 ,like Figure 3 The auxiliary head connects three MobileOne-Block layers. The target detection device needs to delete the network structure and weight parameters of the three MobileOne-Block layers. At the same time, the multi-path training network is converted into a single-path inference network using the structural reparameterization concept. The core technology is to add the weights and bias parameters of multiple convolutional kernels in MobileOne-Block and assign them to a brand new single convolutional kernel of the same size.

[0114] Furthermore, the object detection device fuses the convolutional layers and batch normalization (BN) layers in the MobileOne-Block used in the training network using Equation 2, while also fusing the ordinary convolutional layers and BN layers.

[0115]

[0116] Among them, Wi μ represents the parameters of the convolutional layer before transformation. i σ represents the mean of the BN layer. i γ represents the variance of the BN layer. i and β i W represents the scale factor and offset factor of the BN layer, respectively. i ′ and b i ′ These represent the weights and biases of the convolution after fusion, respectively.

[0117] Step S22: Based on the model parameters, obtain the first model parameters of the first channel-wise convolutional layer branch, the second model parameters of the second channel-wise convolutional layer branch, and the third model parameters of the first batch normalization layer branch in the first stage processing layer.

[0118] Step S23: Merge the first-stage processing layers into a single channel-wise convolutional layer, wherein the model parameters of the channel-wise convolutional layer are obtained by superimposing the first model parameters, the second model parameters, and the third model parameters.

[0119] In this embodiment, the target detection device superimposes the weights and biases of the k fused 3×3 DWConv layers contained in the MobileOne-Block; simultaneously, it converts the fused 1×1 DWConv layer convolutional kernel contained in the MobileOne-Block into a 3×3 convolutional kernel through zero padding; furthermore, it treats the BN layer as a 1×1 unit convolutional kernel with a weight value of 1 and converts it into a 3×3 convolutional kernel through zero padding.

[0120] The target detection device sums the weights W and biases b of all branches of MobileOne-Block to obtain a new 3×3DWConv layer with weights W. ′ The bias is b ′ The calculation formula is shown in Formula 3.

[0121]

[0122] further, For the input, There is an output, and its final correspondence is shown in Formula 4. For have:

[0123] X : ′ ,i,:,: =(X*W ′ ) :,i,:,: +b ′ (4)

[0124] Step S24: Obtain the fourth model parameters of the convolutional layer branch in the second-stage processing layer and the fifth model parameters of the second batch normalization layer branch based on the model parameters.

[0125] Step S25: Merge the second-stage processing layers into a single convolutional layer, wherein the model parameters of the convolutional layer are obtained by superimposing the fourth and fifth model parameters.

[0126] In this embodiment, the target detection device superimposes the weights and biases of the k fused 1×1 ordinary convolutional layers contained in the MobileOne-Block; simultaneously, it maps the BN layer to a 1×1 unit convolutional kernel with a weight value of 1. Further, the weights and biases of the two branches are superimposed to obtain a newly fused 1×1 ordinary convolutional layer.

[0127] Step S26: Replace each processing layer in the model to be trained with the processing layer composed of the fused channel-wise convolutional layer and the fused convolutional layer, and output the replaced object detection model.

[0128] In this embodiment, the target detection device saves the auxiliary head of the deletion detection module and the fused network model structure and weight parameters to obtain the final single-path heavy parameter lightweight inference network, the final network structure diagram of which is shown below. Figure 6 As shown. The MobileOne-Block structure, after reparameter fusion, is transformed into... Figure 4 The structure displayed during the inference stage is a depthwise separable convolution.

[0129] In this embodiment, the object detection device obtains the model parameters of each processing layer after training of the model to be trained. The processing layer includes a first-stage processing layer and a second-stage processing layer. Based on the model parameters, it obtains the first model parameters of the first channel-wise convolutional layer branch, the second model parameters of the second channel-wise convolutional layer branch, and the third model parameters of the first batch normalization layer branch in the first-stage processing layer. The first-stage processing layer is then fused into a single channel-wise convolutional layer, where the model parameters of the channel-wise convolutional layer are obtained by superimposing the first, second, and third model parameters. Based on the model parameters, it obtains the fourth model parameters of the convolutional layer branch in the second-stage processing layer, and the fifth model parameters of the second batch normalization layer branch. The second-stage processing layer is then fused into a single convolutional layer, where the model parameters of the convolutional layer are obtained by superimposing the fourth and fifth model parameters. Finally, the fused channel-wise convolutional layer and the fused convolutional layer are used to replace each processing layer in the model to be trained, and the replaced object detection model is output. In this way, the target detection device uses a multi-path structure during training to pursue high performance and high accuracy of the model; during inference, it reconstructs the network based on the idea of ​​structural reparameterization and converts it into a single-path structure, which retains the high performance advantages of multi-branch network structure and reduces the number of model parameters and the memory and computing resources required to run the model.

[0130] The model training method in this application uses MobileOne-Block, which integrates depthwise separable convolutions and structural reparameterization. Unlike typical structural reparameterization convolutional layers, it improves the 3x3 convolutional layers used during inference by replacing them with 3x3 depthwise separable convolutional layers. Furthermore, by controlling the hyperparameter k, the feature extraction capability of the convolutional layers is enhanced, further reducing the number of model parameters. Based on MobileOne-Block, the concept of the path aggregation network PANet is incorporated to construct the YOLOv7-Lite-Rep feature enhancement and extraction module, performing multi-scale feature extraction on the output of the backbone module. During the inference phase, MobileOne-Block can be converted to a single-path structure. Compared to simply using depthwise separable convolutions, YOLOv7-Lite-Rep improves the model's feature representation capability while maintaining high inference speed, without changing the model parameters.

[0131] Please continue reading. Figure 7 , Figure 7 This is a flowchart illustrating an embodiment of the target detection method provided in this application.

[0132] Specifically, such as Figure 7 As shown, the target detection method in this application embodiment specifically includes the following steps:

[0133] Step S31: Input the image to be detected into the mobile platform.

[0134] Step S32: Obtain the detection result of the target to be detected using the preset target detection model in the mobile platform.

[0135] In this embodiment, the target detection device performs model transformation, quantization, and other operations on the single-path heavy-parameter lightweight inference network and its weight parameters trained by the above model training method, and further deploys it on a mobile platform to complete the detection of target data and output the results.

[0136] In one specific implementation, the single-path network structure used by the final inference network has approximately 1.02 MB of parameters and approximately 1.2 GFLOPS of floating-point computations. It has low requirements for the memory and computing resources of the mobile platform, making it easy to deploy and apply.

[0137] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0138] To implement the model training method and / or object detection method described in the above embodiments, this application also proposes an object detection device, which can be found in detail below. Figure 8 , Figure 8 This is a schematic diagram of an embodiment of the target detection device provided in this application.

[0139] The target detection device 400 of this application embodiment includes a memory 41 and a processor 42, wherein the memory 41 and the processor 42 are coupled together.

[0140] The memory 41 is used to store program data, and the processor 42 is used to execute the program data to implement the model training method and / or target detection method described in the above embodiments.

[0141] In this embodiment, processor 42 can also be referred to as a CPU (Central Processing Unit). Processor 42 may be an integrated circuit chip with signal processing capabilities. Processor 42 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 42 can be any conventional processor.

[0142] To implement the model training method and / or object detection method of the above embodiments, this application also provides a computer-readable storage medium, such as... Figure 9 As shown, the computer-readable storage medium 500 is used to store program data 51, which, when executed by a processor, is used to implement the model training method and / or object detection method as described in the above embodiments.

[0143] This application also provides a computer program product, wherein the computer program product includes a computer program operable to cause a computer to perform the model training method and / or object detection method as described in the embodiments of this application. The computer program product may be a software installation package.

[0144] The model training method and / or object detection method described in the above embodiments of this application, when implemented as a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0145] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A model training method, characterized in that, The model training method includes: Obtain the model parameters of each processing layer after the model to be trained is trained, wherein the processing layer includes a first-stage processing layer and a second-stage processing layer. Based on the model parameters, obtain the first model parameters of the first channel-wise convolutional layer branch, the second model parameters of the second channel-wise convolutional layer branch, and the third model parameters of the first batch normalization layer branch in the first stage processing layer. The first stage processing layer is converted into a single-path structure and fused into a channel-wise convolutional layer, wherein the model parameters of the channel-wise convolutional layer are obtained by superimposing the first model parameters, the second model parameters and the third model parameters; Based on the model parameters, obtain the fourth model parameters of the convolutional layer branch in the second stage processing layer, and the fifth model parameters of the second batch normalization layer branch. The second-stage processing layer is converted into a single-path structure and fused into a convolutional layer, wherein the model parameters of the convolutional layer are obtained by superimposing the fourth model parameters and the fifth model parameters; The processing layer composed of the fused channel-wise convolutional layer and the fused convolutional layer replaces each processing layer in the model to be trained, and outputs the replaced target detection model; The model to be trained includes a backbone network module, a feature enhancement and extraction module, an auxiliary head, and a guide head, wherein the backbone network module and the feature enhancement and extraction module each include several processing layers; The model training method also includes: The image to be trained is input into the backbone network module, and several processing layers stacked sequentially in the backbone network module are used to extract output feature maps with different downsampling resolutions, including a first output feature map, a second output feature map and a third output feature map. The first output feature map, the second output feature map, and the third output feature map are input into the feature enhancement and extraction module. The third output feature map and the second output feature map are fused along the channel dimension to obtain the fourth output feature map. The fourth output feature map is fused with the first output feature map along the channel dimension to obtain the first candidate feature map; The first candidate feature map and the third output feature map are fused along the channel dimension to obtain the second candidate feature map; The second candidate feature map and the fourth output feature map are fused along the channel dimension to obtain the third candidate feature map; The processing layer is used to extract features from the first candidate feature map, the second candidate feature map, and the third candidate feature map to obtain the first predicted feature map, the second predicted feature map, and the third predicted feature map. The processing layer is used to extract features from the first candidate feature map, the fourth output feature map, and the third output feature map, respectively, to obtain the fourth prediction feature map, the fifth prediction feature map, and the sixth prediction feature map. The first predicted feature map, the second predicted feature map, and the third predicted feature map are input into the guide head for prediction, and the first loss function value of the guide head is obtained. The fourth, fifth, and sixth predicted feature maps are input into the auxiliary head for prediction, and the second loss function value of the auxiliary head is obtained. The model to be trained is trained using the first loss function value and the second loss function value to obtain a multi-path detection network.

2. The model training method according to claim 1, characterized in that, Before obtaining the model parameters of each processing layer after training of the model to be trained, the model training method further includes: Remove the auxiliary head and its model parameters from the model to be trained; Delete the network structure and model parameters of the processing layer corresponding to the auxiliary head.

3. The model training method according to claim 1, characterized in that, After obtaining the model parameters of each processing layer of the model to be trained, the model training method further includes: Obtain the combination of convolutional layers and batch normalization layers connected in the processing layer; The convolutional layer and the batch normalization layer are merged into a single convolutional layer; The eighth model parameter of the fused convolutional layer is calculated using the sixth model parameter of the convolutional layer and the seventh model parameter of the batch normalization layer within the combination.

4. The model training method according to claim 3, characterized in that, The calculation of the eighth model parameters of the fused convolutional layer using the sixth model parameters of the convolutional layer and the seventh model parameters of the batch normalization layer includes: Obtain the convolutional layer weight parameters and convolutional layer bias parameters of the sixth model parameters; Obtain the mean and variance of the batch normalization layer of the seventh model parameters; The weight parameters in the eighth model parameters are obtained using the convolutional layer weight parameters, the variance, and the first fusion factor. The bias parameters in the eighth model parameters are obtained using the convolutional layer bias parameters, the mean, the variance, and the second fusion factor.

5. The model training method according to claim 1, characterized in that, The step of converting the first-stage processing layer into a single-path structure and fusing it into a channel-wise convolutional layer includes: Merge several channel-wise convolutional layers in the first channel-wise convolutional layer branch into a single branch channel-wise convolutional layer; The kernel size of the branch channel-wise convolutional layer, the kernel size of the second channel-wise convolutional layer branch, and the kernel size of the first batch normalization layer branch are filled to a uniform kernel size. The filled branch of the channel-wise convolutional layer, the filled second channel-wise convolutional layer branch, and the filled first batch normalization layer branch are merged into a single channel-wise convolutional layer.

6. The model training method according to claim 1, characterized in that, The step of inputting the first predicted feature map, the second predicted feature map, and the third predicted feature map into the guidance head for prediction, and obtaining the first loss function value of the guidance head, includes: The first prediction feature map, the second prediction feature map, and the third prediction feature map are respectively processed into a grid to obtain several predicted bounding box coordinates and prediction categories; According to each prediction category, obtain the grid where the center point of the real bounding box is located and its first preset number of neighboring grids corresponding to the coordinates of each predicted target box, as guiding positive samples for matching. Obtain the cost loss of each guiding positive sample, sort them according to the cost loss, obtain a second preset number of guiding positive samples and input them into the guiding head for prediction, and obtain the first loss function value of the guiding head.

7. The model training method according to claim 6, characterized in that, The auxiliary positive samples for the input auxiliary head are obtained by searching for auxiliary matching positive samples in the fourth prediction feature map, the fifth prediction feature map, and the sixth prediction feature map according to a third preset number of adjacent grids, and sorting them according to the cost loss to obtain a fourth preset number of auxiliary positive samples. Wherein, the third preset quantity is greater than the first preset quantity, and the fourth preset quantity is greater than the second preset quantity.

8. The model training method according to claim 6, characterized in that, Before performing gridding processing on the first predicted feature map, the second predicted feature map, and the third predicted feature map, the model training method further includes: Obtain the number of prior anchor boxes and the number of categories in each of the first, second, and third predicted feature maps; The number of channels for each predicted feature map is obtained according to the number of prior anchor boxes and the number of categories; Each predicted feature map is processed by a convolutional layer according to the number of channels in each predicted feature map.

9. A target detection method, characterized in that, The target detection method includes: Input the image to be detected into the mobile platform; The detection results of the image to be detected are obtained using a preset target detection model in the mobile platform; The target detection model is trained using the model training method described in any one of claims 1 to 8.

10. A target detection device, characterized in that, The target detection device includes a processor and a memory, the memory storing program data, and the processor executing the program data to implement the model training method as described in any one of claims 1-8, and / or the target detection method as described in claim 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program data, which, when executed by a processor, is used to implement the model training method according to any one of claims 1-8, and / or the target detection method according to claim 9.

Images