Detection model training methods, target detection methods, devices, equipment, and media

Abstract

The present disclosure provides a method for training a detection model, which relates to the field of artificial intelligence technology, especially to natural language processing, deep learning, and pre-training model technology, and can be applied in smart city and smart government scenarios. The specific implementation scheme is: inputting a sample image into a first detection model to obtain a first backbone network output feature; performing at least one convolution process on the first backbone network output feature to obtain a first query feature, a first key feature, and a first value feature; obtaining a query loss based on the second query feature and the first query feature; obtaining a key loss based on the second key feature and the first key feature; obtaining a value loss based on the second value feature and the first value feature, wherein the parameter amount of the second detection model is greater than the parameter amount of the first detection model; and training the first detection model based on the query loss, key loss, and value loss. The present disclosure also provides a target detection method, device, electronic device, and storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly natural language processing, deep learning, and pre-trained model technologies, which can be applied in smart city and smart government scenarios. More specifically, this disclosure provides a detection model training method, target detection method, device, electronic device, and storage medium. Background Art

[0002] With the development of artificial intelligence technology, the application scenarios of deep learning models are constantly increasing. Model distillation technology can be used to improve the object detection capabilities of detection models. Summary of the Invention

[0003] The present disclosure provides a detection model training method, a target detection method, an apparatus, an electronic device, and a storage medium.

[0004] According to one aspect of the present disclosure, a method for training a detection model is provided, the method comprising: inputting a sample image into a first detection model to obtain a first backbone network output feature; performing at least one convolution process on the first backbone network output feature to obtain a first query feature, a first key feature, and a first value feature; obtaining a query loss based on the second query feature and the first query feature; obtaining a key loss based on the second key feature and the first key feature; obtaining a value loss based on the second value feature and the first value feature, wherein the second query feature, the second key feature, and the second value feature are obtained by processing the sample image using a second detection model, and the parameter amount of the second detection model is greater than the parameter amount of the first detection model; and training the first detection model based on the query loss, the key loss, and the value loss.

[0005] According to another aspect of the present disclosure, a target detection method is provided, which includes: inputting a target image into a first detection model to obtain a target detection result, wherein the target detection result includes a target detection frame of the target object in the target image, and the target detection frame is used to indicate the area where the target object is located, wherein the first detection model is trained using the method provided by the present disclosure.

[0006] According to another aspect of the present disclosure, a training device for a detection model is provided, which includes: a first acquisition module for inputting a sample image into a first detection model to obtain a first backbone network output feature; a convolution processing module for performing at least one convolution processing on the first backbone network output feature to obtain a first query feature, a first key feature and a first value feature; a second acquisition module for obtaining a query loss based on the second query feature and the first query feature; a third acquisition module for obtaining a key loss based on the second key feature and the first key feature; a fourth acquisition module for obtaining a value loss based on the second value feature and the first value feature, wherein the second query feature, the second key feature and the second value feature are obtained by processing the sample image using the second detection model, and the parameter amount of the second detection model is greater than the parameter amount of the first detection model; and a training module for training the first detection model based on the query loss, the key loss and the value loss.

[0007] According to another aspect of the present disclosure, a target detection device is provided, the device comprising: a fifth acquisition module for inputting a target image into a first detection model to obtain a target detection result, wherein the target detection result includes a target detection frame of a target object in the target image, and the target detection frame is used to indicate the area where the target object is located, wherein the first detection model is trained using the device provided by the present disclosure.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided according to the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method provided according to the present disclosure when executed by a processor.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0013] Figure 1is a schematic diagram of a detection model according to one embodiment of the present disclosure;

[0014] Figure 2 is a flowchart of a method for training a detection model according to an embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram of a backbone network of a detection model according to an embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram of an intermediate network of a detection model according to an embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram of a detection model according to an embodiment of the present disclosure;

[0018] Figure 6 is a flow chart of a target detection method according to one embodiment of the present disclosure;

[0019] Figure 7 is a block diagram of a training apparatus for a detection model according to an embodiment of the present disclosure;

[0020] Figure 8 is a block diagram of an object detection apparatus according to an embodiment of the present disclosure; and

[0021] Figure 9 1 is a block diagram of an electronic device to which a detection model training method and / or a target detection method can be applied according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] The teacher model can be a large model with more parameters and better performance. However, the teacher model has too many parameters and is slow to respond. This makes the teacher model difficult to apply in scenarios where performance requirements, such as high throughput and fast processing, are high.

[0024] Model distillation, also known as knowledge distillation, involves learning useful knowledge from a teacher model to train a student model. Model distillation allows for model compression without sacrificing performance. During the distillation process, the student model can learn the generalization capabilities of the teacher model and achieve performance similar to that of the teacher. While preserving model accuracy, model distillation allows for model compression and improved processing speed. The following describes some approaches to model distillation, using examples.

[0025] In some embodiments, the features and results output by the teacher model can be aligned with those output by the student model. For example, the results output by the teacher model can be used as labels for the student model. For another example, the loss value of the student model can be determined based on the data distribution differences between the results output by the teacher model and the results output by the student model.

[0026] In some embodiments, convolutional layers based on channel attention and spatial attention, respectively, can be added to the student model, and the feature maps output by these convolutional layers can be aligned with the feature maps output by the teacher model for model distillation.

[0027] In some embodiments, the global features output by the teacher model and the global features output by the student model can be aligned.

[0028] It is understood that the structure of the teacher model can be the same as that of the learning model. For example, the backbone network of the teacher model can be built based on the Transformer model. The backbone network of the student model can also be built based on the Transformer model. This allows for efficient model distillation. However, the structure of the teacher model can also differ from that of the student model. In the case of different structures, the efficiency of the model distillation mentioned above may be lower.

[0029] The overall architecture of the model distillation method disclosed herein will be described below in conjunction with relevant embodiments.

[0030] Figure 1 is a schematic diagram of a detection model according to an embodiment of the present disclosure.

[0031] In some embodiments, the detection model may include a backbone network (Backbone), an intermediate network (Neck) and a detection network (Head). Figure 1As shown, the first detection model 110 may include a first backbone network 111, a first intermediate network 112 and a first detection network 113. The second detection model 120 may include a second backbone network 121, a second intermediate network 122 and a second detection network 123. For example, the first backbone network 111 may be a convolutional neural network (CNN). The second backbone network 121 may include multiple Transformer coding blocks (Transformer Block). It can be understood that the structure of the first backbone network 111 and the structure of the second backbone network 121 are different. It can also be understood that the first detection model 110 can be used as a student model. The second detection model 120 can be used as a teacher model. The number of parameters of the second detection model 120 can be greater than the number of parameters of the first detection model 110.

[0032] In the embodiment of the present disclosure, the sample image can be input into the first detection model to obtain the first backbone network output feature, the first intermediate network output feature and the first detection result. For example, the sample image is input into the first backbone network to obtain the first backbone network output feature. The first backbone network output feature is input into the first intermediate network to obtain the first intermediate network output feature. The first intermediate network output feature is input into the first detection network to obtain the first detection result. Figure 1 As shown, the sample image 1001 is input into the first backbone network 111 to obtain the first backbone network output features. The first backbone network output features are input into the first intermediate network 112 to obtain the first intermediate network output features. The first intermediate network output features are input into the first detection network 113 to obtain the first detection result.

[0033] like Figure 1 As shown, the sample image 1001 is input into the second backbone network 121 to obtain the second backbone network output features. The second backbone network output features are input into the second intermediate network 122 to obtain the second intermediate network output features. The second intermediate network output features are input into the second detection network 123 to obtain the second detection result. It can be understood that the second detection model 120 can be used to process the sample image 1001 before it is input into the first detection model 110.

[0034] In the disclosed embodiments, the label of a sample image may include a labeled detection box of a sample object in the sample image and a sample category. The labeled detection box indicates the area where the sample object is located. The sample category may indicate the category of the sample object. The category of the sample object may be animal, plant, or other example.

[0035] The following will be combined Figure 2 Some implementations for enabling the first backbone network to achieve performance close to that of the second backbone network will be described.

[0036] Figure 2 4 is a flowchart of a method for training a detection model according to an embodiment of the present disclosure.

[0037] like Figure 2 As shown, the method 200 may include operations S210 to S260.

[0038] In operation S210, a sample image is input into a first detection model to obtain a first backbone network output feature.

[0039] In the embodiment of the present disclosure, the first detection model may be a student model. For example, the first detection model may be the first detection model 110 described above. The sample image is input into the first backbone network of the first detection model to obtain the first backbone network output features.

[0040] In operation S220, at least one convolution process is performed on the first backbone network output features to obtain a first query feature, a first key feature, and a first value feature.

[0041] In the embodiment of the present disclosure, the first backbone network may be a convolutional neural network.

[0042] In the embodiment of the present disclosure, three convolutional layers can be used to perform three convolution processes on the first backbone network features to obtain the first query feature, the first key feature and the first value feature.

[0043] In the disclosed embodiment, the second detection model may be a teacher model. For example, the second backbone network of the second detection model may include multiple cascaded Transformer encoding blocks. The number of parameters of the second detection model may be greater than the number of parameters of the first detection model.

[0044] In the embodiment of the present disclosure, the second query feature, the second key feature and the second value feature are obtained by processing the sample image using the second detection model. For example, the sample image can be processed using multiple cascaded Transformer encoding blocks. Among the multiple cascaded Transformer encoding blocks, the last Transformer encoding block can output the query feature, key feature and value feature of the last level. Based on the attention mechanism, the TransfOrmer encoding block can also fuse the query feature, key feature and value feature of the last level to obtain a fused feature as the output feature of the second backbone network. The above-mentioned query feature, key feature and value feature of the last level can be used as the second query feature, the second key feature and the second value feature, respectively.

[0045] In operation S230 , a query loss is obtained according to the second query feature and the first query feature.

[0046] For example, various loss functions can be used to determine the query loss.

[0047] In operation S240, a key loss is obtained based on the second key feature and the first key feature.

[0048] For example, bond loss can be determined using various loss functions.

[0049] In operation S250 , a value loss is obtained according to the second value feature and the first value feature.

[0050] For example, the value loss can be determined using various loss functions.

[0051] In operation S260 , a first detection model is trained according to the query loss, the key loss, and the value loss.

[0052] In the embodiment of the present disclosure, the backbone network loss can be obtained based on the query loss, key loss and value loss. For example, the query loss, key loss and value loss can be weighted or summed to obtain the backbone network loss.

[0053] In the disclosed embodiment, the parameters of the first detection model can be adjusted so that the backbone network loss converges. It is understood that H rounds of training can be performed to obtain H backbone network losses respectively. H is an integer greater than 1. If the backbone network loss of the hth round is less than or equal to the backbone network loss of the h-1th round and the backbone network loss of the hth round is less than or equal to the backbone network loss of the h+1th round, it can be determined that the backbone network loss converges after h rounds of training. h can be an integer greater than 1 and less than H.

[0054] Through the disclosed embodiments, the output features of the first backbone network are convolved at least once to obtain a feature form close to that of the backbone network of the teacher model. This allows the first backbone network to achieve performance close to that of the second backbone network, achieving knowledge distillation across model structures.

[0055] In addition, the teacher model can be built based on a large number of Transformer coding blocks, so that the teacher model can have higher accuracy and generalization ability. However, directly using the teacher model for target detection requires a high time cost. In order to reduce the time cost, a student model can be built based on a convolutional neural network. Through the embodiments of the present disclosure, the student model built based on the convolutional neural network can learn the data processing capabilities of the teacher model built based on the Transformer model, which helps to improve the accuracy of target detection.

[0056] It is understandable that the above Figure 2 The method flow of the present invention is described below. Figure 3Some ways in which the present disclosure implements knowledge distillation across model structures are further described.

[0057] Figure 3 Schematic diagram of a backbone network of a detection model according to an embodiment of the present disclosure.

[0058] like Figure 3 As shown, the first detection model 310 may include a first backbone network 311, a first intermediate network 312, and a first detection network 313. The second detection model 320 may include a second backbone network 321, a second intermediate network 322, and a second detection network 323. For a detailed description of the first detection model 310 and the second detection model 320, reference may be made to the first detection model 110 and the second detection model 120 described above, and will not be repeated herein.

[0059] In the embodiment of the present disclosure, in the above-mentioned operation S220, performing at least one convolution process on the first backbone network output feature to obtain the first query feature, the first key feature, and the first value feature may include: performing a first convolution process on the first backbone network output feature to obtain the first query feature. Performing a second convolution process on the first backbone network output feature to obtain the first key feature. Performing a third convolution process on the first backbone network output feature to obtain the first value feature.

[0060] For example, the first convolution layer 3141 can be used to perform a first convolution process on the first backbone network output feature to obtain a first query feature. The second convolution layer 3142 can be used to perform a second convolution process on the first backbone network output feature to obtain a first key feature. The third convolution layer 3143 can be used to perform a third convolution process on the first backbone network output feature to obtain a first value feature.

[0061] Next, the above operations S230 to S250 may be performed to obtain a query loss based on the first query feature and the second query feature, a key loss based on the first key feature and the second key feature, and a value loss based on the first value feature and the second value feature.

[0062] In the disclosed embodiment, in the aforementioned operation S260, backbone network loss 301 can be obtained based on the query loss, key loss, and value loss. The parameters of the first detection model 310 can be adjusted based on backbone network loss 301. Through the disclosed embodiment, the performance differences between backbone networks with different structures can be reduced, the feature extraction capability of the first detection model can be improved, and the target detection capability of the first detection model can be improved.

[0063] It can be understood that the above describes some ways to achieve knowledge distillation across model structures in conjunction with relevant embodiments. Figure 4Some ways of enabling the first intermediate network to achieve performance close to that of the second intermediate network are described.

[0064] Figure 4 is a schematic diagram of an intermediate network of a detection model according to an embodiment of the present disclosure.

[0065] like Figure 4 As shown, first detection model 410 may include a first backbone network 411, a first intermediate network 412, and a first detection network 413. Second detection model 420 may include a second backbone network 421, a second intermediate network 422, and a second detection network 423. For detailed descriptions of first detection model 410 and second detection model 420, reference may be made to first detection model 110 and second detection model 120 described above, and will not be repeated herein.

[0066] In some embodiments, in the above-mentioned operation S260, training the first detection model according to the query loss, key loss and value loss may further include: obtaining the intermediate network output loss according to the second intermediate network output feature and the first intermediate network output feature.

[0067] In the disclosed embodiment, the second intermediate network output feature is obtained by processing the second backbone network output feature using the second detection model. The second backbone network output feature is obtained by fusing the second query feature, the second key feature, and the second value feature. For example, the second intermediate network output feature is obtained by processing the second backbone network output using the second intermediate network 420.

[0068] like Figure 4 As shown, the first processing layer 415 can be used to process the first intermediate network output features to obtain processed first intermediate network output features. The second processing layer 416 can also be used to process the second intermediate network output features to obtain processed second intermediate network output features. It is understood that the first processing layer 415 and the second processing layer 416 can be constructed by various neural networks. In one example, the first processing layer 415 and the second processing layer 416 can be constructed based on feature pyramid networks (FPNs).

[0069] In an embodiment of the present disclosure, obtaining the intermediate network output loss based on the second intermediate network output feature and the first intermediate network output feature may include: normalizing the first intermediate network output feature to obtain the first intermediate normalized feature. Normalizing the second intermediate network output feature to obtain the second intermediate normalized feature. For example, the normalization layer 431 may be used to perform channel-wise normalization on the processed first intermediate network output feature to obtain the first intermediate normalized feature. The normalization layer 431 may also be used to perform channel-wise normalization on the processed second intermediate network output feature to obtain the second intermediate normalized feature. As mentioned above, the parameter quantity of the second detection model is greater than the parameter quantity of the first detection model. For the first detection model and the second detection model, the magnitude of the parameter quantities of the two may be different, the numerical scale of the parameters of the two may also be inconsistent, and the structure of the backbone network of the two may also be inconsistent. Through the embodiments disclosed herein, the output features of the first intermediate network and the output features of the second intermediate network are normalized in terms of channel dimensions, which can reduce the impact of inconsistent magnitudes of parameter quantities and inconsistent numerical scales of parameters, and especially can further reduce the impact of inconsistent structures of the backbone network, so that a learning model with different backbone network results from the teacher model can more fully learn the generalization ability of the teacher model, and further improve the target detection ability of the first detection model as a student model.

[0070] In the embodiment of the present disclosure, the intermediate network output loss can be obtained according to the first intermediate normalized feature and the second intermediate normalized feature, which will be described in detail below.

[0071] In the embodiment of the present disclosure, obtaining the intermediate network output loss based on the first intermediate normalized feature and the second intermediate normalized feature may include: performing global context processing on the first intermediate normalized feature to obtain the first global feature; and performing global context processing on the second intermediate normalized feature to obtain the second global feature.

[0072] In an embodiment of the present disclosure, the first intermediate normalized feature can be processed based on the channel attention mechanism to obtain the first channel attention feature. The first channel attention feature can be processed based on the spatial attention mechanism to obtain the first spatial attention feature. The first spatial attention feature can be subjected to global context processing to obtain the first global feature. For example, the first intermediate normalized feature can be input into the channel attention (Channel Attention) processing layer 432 to obtain the first channel attention feature. The first channel attention feature can be input into the spatial attention (Spatial Attention) processing layer 433 to obtain the first spatial attention feature. The first spatial attention feature can be input into the global context processing layer (Globalcontext Block, GcBlock) 434 to obtain the first global feature.

[0073] In an embodiment of the present disclosure, the second intermediate normalized feature can be processed based on the channel attention mechanism to obtain a second channel attention feature. The second channel attention feature can be processed based on the spatial attention mechanism to obtain a second spatial attention feature. The second spatial attention feature can be subjected to global context processing to obtain a second global feature. For example, the second intermediate normalized feature can be input into the channel attention processing layer 432 to obtain a second channel attention feature. The second channel attention feature can be input into the spatial attention processing layer 433 to obtain a second spatial attention feature. The second spatial attention feature can be input into the global context processing layer 434 to obtain a second global feature.

[0074] In the embodiment of the present disclosure, the first intermediate network output sub-loss can be obtained according to the first global feature and the second global feature.

[0075] In addition, in an embodiment of the present disclosure, obtaining the intermediate network output loss based on the second intermediate network output feature and the first intermediate network output feature may further include: performing masking processing on the first intermediate network output feature according to the position of the labeled detection box indicated by the label of the sample image to obtain a first intermediate mask feature. Performing global context processing on the first intermediate mask feature to obtain a first global mask feature. For example, the processed first intermediate network output feature can be input into the first mask processing layer 416 to obtain the first intermediate mask feature. The first intermediate mask feature can be input into the global context processing layer 434 to obtain the first global mask feature. It is understood that the processed first intermediate network output feature can be implemented as a first feature map. A first mapping relationship between the first feature map and the sample image can be determined. The label of the sample image can include a labeled detection box. The first mask processing layer 416 can perform the following operations: Based on the first mapping relationship, the labeled detection box can be mapped to the first feature map to obtain a first mapping box. In the first feature map, feature values ​​outside the first mapping box are set to a preset value (e.g., 0) to implement masking.

[0076] In an embodiment of the present disclosure, obtaining the intermediate network output loss based on the second intermediate network output feature and the first intermediate network output feature may further include: performing masking processing on the second intermediate network output feature according to the position of the labeled detection box indicated by the label of the sample image to obtain a second intermediate mask feature. Performing global context processing on the second intermediate mask feature to obtain a second global mask feature. For example, the processed second intermediate network output feature can be input into the second mask processing layer 426 to obtain the second intermediate mask feature. The second intermediate mask feature can be input into the global context processing layer 434 to obtain the second global mask feature. It can be understood that the processed second intermediate network output feature can be implemented as a second feature map. A second mapping relationship between the second feature map and the sample image can be determined. The label of the sample image can include a labeled detection box. The second mask processing layer 426 can perform the following operations: Based on the second mapping relationship, the labeled detection box can be mapped to the second feature map to obtain a second mapping box. In the second feature map, the feature values ​​outside the second mapping box are set to a preset value (e.g., 0) to implement masking.

[0077] Next, according to the first global mask feature and the second global mask feature, the second intermediate network output sub-loss can be obtained.

[0078] In the disclosed embodiment, the intermediate network output loss can be obtained based on the first intermediate network output sub-loss. Furthermore, the intermediate network output loss can be obtained based on the second intermediate network output sub-loss. For example, the intermediate network output loss 402 can be obtained based on the first intermediate network output sub-loss and the second intermediate network output sub-loss. The disclosed embodiment can reduce the impact of different backbone network structures on other networks, further improving the feature extraction capability of the first detection model and, consequently, the target detection capability of the first detection model.

[0079] It is understandable that the above Figure 4 Some methods for making the first intermediate network achieve performance close to that of the second intermediate network are described below. Figure 5 Some methods of training the first detection model are further explained.

[0080] Figure 5 is a schematic diagram of a detection model according to an embodiment of the present disclosure.

[0081] like Figure 5 As shown, the first detection model 510 may include a first backbone network 511, a first intermediate network 512, and a first detection network 513. The second detection model 520 may include a second backbone network 521, a second intermediate network 522, and a second detection network 523. For a detailed description of the first detection model 510 and the second detection model 520, reference may be made to the first detection model 110 and the second detection model 120 described above, and will not be repeated herein.

[0082] In an embodiment of the present disclosure, training the first detection model based on the backbone network output loss and the intermediate network output loss may include: obtaining a distillation loss based on the second detection result and the first detection result; obtaining a detection loss based on the label of the sample image and the first detection result; and training the first detection model based on the backbone network output loss, the intermediate network output loss, the distillation loss, and the detection loss.

[0083] like Figure 5 As shown, backbone network loss 501 can be obtained based on the first backbone network output feature output by the first backbone network 511 and the second backbone network output feature output by the second backbone network 521. It can be understood that for a detailed description of backbone network loss 501, reference can be made to the aforementioned backbone network loss 301, and this disclosure will not elaborate on it here.

[0084] Based on the first intermediate network output characteristic output by the first intermediate network 512 and the second intermediate network output characteristic output by the second intermediate network 522, an intermediate network loss 502 can be obtained. It is understood that for a detailed description of the intermediate network loss 502, reference can be made to the aforementioned intermediate network loss 402, and this disclosure will not elaborate on this further.

[0085] The features output by the first intermediate network are input into the first detection network 513 to obtain a first detection result. The features output by the second intermediate network are input into the second detection network 523 to obtain a second detection result. Based on the first and second detection results, a distillation loss 503 is obtained. Based on the first detection result and the label 5002 of the sample image 5001, a detection loss 504 is obtained.

[0086] Next, the total loss can be obtained according to the backbone network loss 501, the intermediate network loss 502, the distillation loss 503 and the detection loss 504. The parameters of the first detection model can be adjusted so that the total loss converges to train the first detection model.

[0087] Figure 6 is a flowchart of a target detection method according to an embodiment of the present disclosure.

[0088] like Figure 6 As shown, the method 600 may include operation S610.

[0089] In operation S310 , a target image is input into a first detection model to obtain a target detection result.

[0090] In the disclosed embodiments, the target detection result includes a target detection frame of the target object in the target image. The target detection frame is used to indicate the area where the target object is located. For example, the target image may include one or more objects. For another example, the target object is any one of the one or more objects. For another example, the object may be various animals or plants.

[0091] In an embodiment of the present disclosure, the first detection model is trained using the method provided by the present disclosure. For example, the first detection model is trained using method 200.

[0092] Figure 7 4 is a block diagram of a detection model training apparatus according to an embodiment of the present disclosure.

[0093] like Figure 7 As shown, the apparatus 700 may include a first obtaining module 710 , a convolution processing module 720 , a second obtaining module 730 , a third obtaining module 740 , a fourth obtaining module 750 and a training module 760 .

[0094] The first acquisition module 710 is used to input the sample image into the first detection model to obtain the first backbone network output features.

[0095] The convolution processing module 720 is used to perform at least one convolution process on the first backbone network output features to obtain a first query feature, a first key feature and a first value feature.

[0096] The second obtaining module 730 is configured to obtain the query loss according to the second query feature and the first query feature.

[0097] The third obtaining module 740 is configured to obtain the key loss according to the second key feature and the first key feature.

[0098] The fourth obtaining module 750 is configured to obtain a value loss based on the second value feature and the first value feature. For example, the second query feature, the second key feature, and the second value feature are obtained by processing the sample image using a second detection model, and the number of parameters of the second detection model is greater than the number of parameters of the first detection model.

[0099] The training module 760 is used to train the first detection model according to the query loss, the key loss and the value loss.

[0100] In some embodiments, the convolution processing module includes: a first convolution processing module, used to perform a first convolution processing on the first backbone network output feature to obtain a first query feature; a second convolution processing module, used to perform a second convolution processing on the first backbone network output feature to obtain a first key feature; and a third convolution processing module, used to perform a third convolution processing on the first backbone network output feature to obtain a first value feature.

[0101] In some embodiments, the first acquisition module includes: a first acquisition submodule, which is used to input the sample image into the first detection model to obtain the first backbone network output feature, the first intermediate network output feature and the first detection result.

[0102] In some embodiments, the first detection model includes a first backbone network, a first intermediate network and a first detection network, the second detection model includes a second backbone network, a second intermediate network and a second detection network, the second query feature, the second key feature and the second value feature are obtained by processing the sample image using the second backbone network, and the structure of the first backbone network is different from the structure of the second backbone network.

[0103] In some embodiments, the first acquisition submodule includes: a first acquisition unit, used to input the sample image into the first backbone network to obtain the first backbone network output feature; a second acquisition unit, used to input the first backbone network output feature into the first intermediate network to obtain the first intermediate network output feature; and a third acquisition unit, used to input the first intermediate network output feature into the first detection network to obtain the first detection result.

[0104] In some embodiments, the training module includes: a second acquisition submodule, used to obtain the backbone network output loss based on the query loss, key loss and value loss; a third acquisition submodule, used to obtain the intermediate network output loss based on the second intermediate network output feature and the first intermediate network output feature, wherein the second intermediate network output feature is obtained by processing the second backbone network output feature using the second detection model, and the second backbone network output feature is obtained by fusing the second query feature, the second key feature and the second value feature; and a training submodule, used to train the first detection model based on the backbone network output loss and the intermediate network output loss.

[0105] In some embodiments, the third acquisition submodule includes: a first normalization unit, used to normalize the first intermediate network output feature to obtain a first intermediate normalized feature; a second normalization unit, used to normalize the second intermediate network output feature to obtain a second intermediate normalized feature; and a fourth acquisition unit, used to obtain the intermediate network output loss based on the first intermediate normalized feature and the second intermediate normalized feature.

[0106] In some embodiments, the fourth obtaining unit includes: a first global context processing subunit, used to perform global context processing on the first intermediate normalized feature to obtain a first global feature; a second global context processing subunit, used to perform global context processing on the second intermediate normalized feature to obtain a second global feature; a first obtaining subunit, used to obtain a first intermediate network output sub-loss based on the first global feature and the second global feature; and a second obtaining subunit, used to obtain an intermediate network output loss based on the first intermediate network output sub-loss.

[0107] In some embodiments, the first global context processing subunit is also used to: process the first intermediate normalized feature based on the channel attention mechanism to obtain a first channel attention feature; process the first channel attention feature based on the spatial attention mechanism to obtain a first spatial attention feature; and perform global context processing on the first spatial attention feature to obtain a first global feature.

[0108] In some embodiments, the second global context processing subunit is also used to: process the second intermediate normalized feature based on the channel attention mechanism to obtain a second channel attention feature; process the second channel attention feature based on the spatial attention mechanism to obtain a second spatial attention feature; and perform global context processing on the second spatial attention feature to obtain a second global feature.

[0109] In some embodiments, the third acquisition submodule includes: a first mask processing unit, which is used to mask the first intermediate network output feature according to the position of the labeled detection box indicated by the label of the sample image, so as to obtain a first intermediate mask feature; a second mask processing unit, which is used to mask the second intermediate network output feature according to the position of the labeled detection box indicated by the label of the sample image, so as to obtain a second intermediate mask feature; a first global context processing unit, which is used to perform global context processing on the first intermediate mask feature, so as to obtain a first global mask feature; a second global context processing unit, which is used to perform global context processing on the second intermediate mask feature, so as to obtain a second global mask feature; a fifth acquisition unit, which is used to obtain the second intermediate network output sub-loss according to the first global mask feature and the second global mask feature; and a sixth acquisition unit, which is used to obtain the intermediate network output loss according to the second intermediate network output sub-loss.

[0110] In some embodiments, the training submodule includes: a seventh acquisition unit, used to obtain the distillation loss based on the second detection result and the first detection result; an eighth acquisition unit, used to obtain the detection loss based on the label of the sample image and the first detection result, wherein the label includes a labeled detection box of the sample object in the sample image, and the labeled detection box is used to indicate the area where the sample object is located; and a training unit, used to train the first detection model based on the backbone network output loss, the intermediate network output loss, the distillation loss and the detection loss.

[0111] Figure 8 is a block diagram of an object detection apparatus according to another embodiment of the present disclosure.

[0112] like Figure 8 As shown, the apparatus 800 may include a fifth obtaining module 810

[0113] The fifth obtaining module 810 is used to input the target image into the first detection model to obtain the target detection result.

[0114] For example, the target detection result includes a target detection frame of the target object in the target image, and the target detection frame is used to indicate the area where the target object is located.

[0115] For example, the first detection model is trained using the apparatus provided by the present disclosure.

[0116] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0117] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0118] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0119] like Figure 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0120] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0121] The computing unit 901 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the training method of the detection model and / or the target detection method. For example, in some embodiments, the training method of the detection model and / or the target detection method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the training method of the detection model and / or the target detection method described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the detection model training method and / or the target detection method in any other appropriate manner (for example, by means of firmware).

[0122] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0124] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) display or an LCD (liquid crystal display)) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0126] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0127] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0128] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0129] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for training a detection model, comprising: Inputting the sample image into the first detection model to obtain the first backbone network output features; Performing at least one convolution process on the first backbone network output features to obtain a first query feature, a first key feature, and a first value feature; Obtaining a query loss based on the second query feature and the first query feature; Obtaining a bond loss according to the second bond characteristic and the first bond characteristic; Obtaining a value loss based on a second value feature and the first value feature, wherein the second query feature, the second key feature, and the second value feature are obtained by processing the sample image using a second detection model, and an amount of parameters of the second detection model is greater than an amount of parameters of the first detection model; as well as The first detection model is trained according to the query loss, the key loss, and the value loss.

2. The method according to claim 1, wherein The step of performing at least one convolution process on the first backbone network output features to obtain the first query feature, the first key feature, and the first value feature comprises: Performing a first convolution process on the first backbone network output feature to obtain the first query feature; Performing a second convolution process on the first backbone network output feature to obtain the first key feature; and Perform a third convolution process on the first backbone network output feature to obtain the first value feature.

3. The method according to claim 1, wherein Inputting the sample image into the first detection model to obtain the first backbone network output features includes: The sample image is input into the first detection model to obtain the first backbone network output feature, the first intermediate network output feature and the first detection result.

4. The method according to claim 3, wherein: The first detection model includes a first backbone network, a first intermediate network and a first detection network. The second detection model includes a second backbone network, a second intermediate network and a second detection network. The second query feature, the second key feature, and the second value feature are obtained by processing the sample image using the second backbone network. The structure of the first backbone network is different from that of the second backbone network.

5. The method according to claim 4, wherein Inputting the sample image into the first detection model to obtain the first backbone network output feature, the first intermediate network output feature, and the first detection result includes: Inputting the sample image into the first backbone network to obtain output features of the first backbone network; Inputting the first backbone network output feature into the first intermediate network to obtain the first intermediate network output feature; and The output feature of the first intermediate network is input into the first detection network to obtain the first detection result.

6. The method according to claim 3, wherein: The training of the first detection model according to the query loss, the key loss, and the value loss includes: Obtaining a backbone network output loss according to the query loss, the key loss, and the value loss; Obtaining an intermediate network output loss based on a second intermediate network output feature and the first intermediate network output feature, wherein the second intermediate network output feature is obtained by processing a second backbone network output feature using the second detection model, and the second backbone network output feature is obtained by fusing the second query feature, the second key feature, and the second value feature; and The first detection model is trained according to the backbone network output loss and the intermediate network output loss.

7. The method according to claim 6, wherein: The obtaining of the intermediate network output loss according to the second intermediate network output feature and the first intermediate network output feature includes: Normalizing the first intermediate network output feature to obtain a first intermediate normalized feature; Normalizing the second intermediate network output feature to obtain a second intermediate normalized feature; and The intermediate network output loss is obtained according to the first intermediate normalized features and the second intermediate normalized features.

8. The method according to claim 7, wherein: The obtaining of the intermediate network output loss according to the first intermediate normalized feature and the second intermediate normalized feature includes: Performing global context processing on the first intermediate normalized feature to obtain a first global feature; performing global context processing on the second intermediate normalized feature to obtain a second global feature; Obtaining a first intermediate network output sub-loss according to the first global feature and the second global feature; and The intermediate network output loss is obtained according to the first intermediate network output sub-loss.

9. The method according to claim 8, wherein The performing global context processing on the first intermediate normalized feature to obtain a first global feature includes: Processing the first intermediate normalized feature based on the channel attention mechanism to obtain a first channel attention feature; Processing the first channel attention feature based on a spatial attention mechanism to obtain a first spatial attention feature; and Perform global context processing on the first spatial attention feature to obtain the first global feature.

10. The method according to claim 8, wherein The performing global context processing on the second intermediate normalized feature to obtain the second global feature includes: Processing the second intermediate normalized feature based on the channel attention mechanism to obtain a second channel attention feature; Processing the second channel attention feature based on the spatial attention mechanism to obtain a second spatial attention feature; and Perform global context processing on the second spatial attention feature to obtain the second global feature.

11. The method according to claim 6, wherein: The obtaining of the intermediate network output loss according to the second intermediate network output feature and the first intermediate network output feature includes: performing mask processing on the first intermediate network output feature according to the position of the annotated detection box indicated by the label of the sample image to obtain a first intermediate mask feature; performing mask processing on the second intermediate network output feature according to the position of the annotated detection box indicated by the label of the sample image to obtain a second intermediate mask feature; Performing global context processing on the first intermediate mask feature to obtain a first global mask feature; Performing global context processing on the second intermediate mask feature to obtain a second global mask feature; Obtaining a second intermediate network output sub-loss according to the first global mask feature and the second global mask feature; and The intermediate network output loss is obtained according to the second intermediate network output sub-loss.

12. The method according to claim 6, wherein: The training of the first detection model according to the backbone network output loss and the intermediate network output loss includes: Obtaining a distillation loss according to the second detection result and the first detection result; Obtaining a detection loss based on a label of the sample image and the first detection result, wherein the label includes a labeled detection box of a sample object in the sample image, the labeled detection box being used to indicate an area where the sample object is located; and The first detection model is trained according to the backbone network output loss, the intermediate network output loss, the distillation loss and the detection loss.

13. A target detection method, comprising: Input the target image into the first detection model to obtain the target detection result. The target detection result includes a target detection frame of the target object in the target image, and the target detection frame is used to indicate the area where the target object is located. Wherein, the first detection model is trained using the method according to any one of claims 1 to 12.

14. A training device for a detection model, comprising: A first acquisition module is used to input the sample image into the first detection model to obtain the first backbone network output features; A convolution processing module, configured to perform at least one convolution process on each of the first backbone network output features to obtain a first query feature, a first key feature, and a first value feature; A second obtaining module, configured to obtain a query loss according to a second query feature and the first query feature; a third obtaining module, configured to obtain a key loss according to the second key feature and the first key feature; a fourth obtaining module, configured to obtain a value loss based on a second value feature and the first value feature, wherein the second query feature, the second key feature, and the second value feature are obtained by processing the sample image using a second detection model, and a parameter amount of the second detection model is greater than a parameter amount of the first detection model; as well as A training module is configured to train the first detection model according to the query loss, the key loss, and the value loss.

15. The device according to claim 14, wherein The convolution processing module includes: A first convolution processing module, configured to perform a first convolution processing on the first backbone network output feature to obtain the first query feature; a second convolution processing module, configured to perform a second convolution processing on the first backbone network output feature to obtain the first key feature; and The third convolution processing module is used to perform a third convolution processing on the first backbone network output feature to obtain the first value feature.

16. The device according to claim 14, wherein The first obtaining module includes: The first acquisition submodule is used to input the sample image into the first detection model to obtain the first backbone network output feature, the first intermediate network output feature and the first detection result.

17. The device according to claim 16, wherein The first detection model includes a first backbone network, a first intermediate network and a first detection network. The second detection model includes a second backbone network, a second intermediate network and a second detection network. The second query feature, the second key feature, and the second value feature are obtained by processing the sample image using the second backbone network. The structure of the first backbone network is different from that of the second backbone network.

18. The device according to claim 17, wherein The first obtaining submodule includes: A first obtaining unit, configured to input the sample image into the first backbone network to obtain output features of the first backbone network; a second obtaining unit, configured to input the first backbone network output feature into the first intermediate network to obtain the first intermediate network output feature; and The third obtaining unit is configured to input the output feature of the first intermediate network into the first detection network to obtain the first detection result.

19. The device according to claim 16, wherein The training module includes: A second obtaining submodule is configured to obtain a backbone network output loss according to the query loss, the key loss, and the value loss; a third obtaining submodule, configured to obtain an intermediate network output loss based on a second intermediate network output feature and the first intermediate network output feature, wherein the second intermediate network output feature is obtained by processing the second backbone network output feature using the second detection model, and the second backbone network output feature is obtained by fusing the second query feature, the second key feature, and the second value feature; and A training submodule is used to train the first detection model according to the backbone network output loss and the intermediate network output loss.

20. The device according to claim 19, wherein The third obtaining submodule includes: a first normalization unit, configured to perform normalization processing on the first intermediate network output feature to obtain a first intermediate normalized feature; a second normalization unit, configured to perform normalization processing on the second intermediate network output feature to obtain a second intermediate normalized feature; and A fourth obtaining unit is used to obtain the intermediate network output loss according to the first intermediate normalized feature and the second intermediate normalized feature.

21. The device according to claim 20, wherein The fourth obtaining unit includes: a first global context processing subunit, configured to perform global context processing on the first intermediate normalized feature to obtain a first global feature; a second global context processing subunit, configured to perform global context processing on the second intermediate normalized feature to obtain a second global feature; a first obtaining subunit, configured to obtain a first intermediate network output sub-loss according to the first global feature and the second global feature; and The second obtaining subunit is configured to obtain the intermediate network output loss according to the first intermediate network output sub-loss.

22. The device according to claim 21, wherein The first global context processing subunit is further configured to: Processing the first intermediate normalized feature based on the channel attention mechanism to obtain a first channel attention feature; Processing the first channel attention feature based on a spatial attention mechanism to obtain a first spatial attention feature; as well as Perform global context processing on the first spatial attention feature to obtain the first global feature.

23. The device according to claim 21, wherein The second global context processing subunit is further configured to: Processing the second intermediate normalized feature based on the channel attention mechanism to obtain a second channel attention feature; Processing the second channel attention feature based on the spatial attention mechanism to obtain a second spatial attention feature; as well as Perform global context processing on the second spatial attention feature to obtain the second global feature.

24. The apparatus according to claim 19, wherein The third obtaining submodule includes: a first mask processing unit, configured to perform mask processing on the first intermediate network output feature according to the position of the annotated detection box indicated by the label of the sample image, to obtain a first intermediate mask feature; a second mask processing unit, configured to perform mask processing on the second intermediate network output feature according to the position of the annotated detection box indicated by the label of the sample image, to obtain a second intermediate mask feature; a first global context processing unit, configured to perform global context processing on the first intermediate mask feature to obtain a first global mask feature; a second global context processing unit, configured to perform global context processing on the second intermediate mask feature to obtain a second global mask feature; a fifth obtaining unit, configured to obtain a second intermediate network output sub-loss according to the first global mask feature and the second global mask feature; and A sixth obtaining unit is configured to obtain the intermediate network output loss according to the second intermediate network output sub-loss.

25. The apparatus according to claim 19, wherein The training submodule includes: a seventh obtaining unit, configured to obtain a distillation loss according to the second detection result and the first detection result; an eighth obtaining unit, configured to obtain a detection loss based on a label of the sample image and the first detection result, wherein the label includes a labeled detection frame of a sample object in the sample image, the labeled detection frame being used to indicate an area where the sample object is located; and A training unit is used to train the first detection model according to the backbone network output loss, the intermediate network output loss, the distillation loss and the detection loss.

26. A target detection device comprising: The fifth acquisition module is used to input the target image into the first detection model to obtain the target detection result. The target detection result includes a target detection frame of the target object in the target image, and the target detection frame is used to indicate the area where the target object is located. Wherein, the first detection model is trained using the device described in any one of claims 14 to 25.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 13.

29. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images