Target detection network training and target detection method and device, and electronic equipment

Abstract

The present disclosure provides a target detection network training method and a target detection method, device and electronic equipment. The training method is as follows: inputting each training image in a training image set into a first neural network to obtain first feature map information of each training image; obtaining first target detection information of each training image through an extraction module according to the first feature map information; inputting each training image into a second neural network to obtain second feature map information of each training image; obtaining second target detection information of each training image through the extraction module according to the second feature map information and the first feature map information; training the second neural network according to the second target detection information and the first target detection information of each training image, and obtaining a target detection network under the condition that a preset condition is met, thereby realizing a general knowledge distillation.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to target detection network training and target detection methods, apparatus, and electronic equipment. Background Technology

[0002] Object detection is the process of finding targets or objects of interest in an image, and it is an important problem in computer vision. In practical deployments, due to the computing power limitations of different hardware devices, knowledge distillation algorithms can be used to compress neural networks. For example, a large neural network can be used to train a small neural network. However, in related technologies, knowledge distillation algorithms are usually only applied to homogeneous networks, requiring that the large and small neural networks have the same detectors and the same backbone network structure, which reduces the flexibility and applicable scenarios of knowledge distillation algorithms. Summary of the Invention

[0003] This disclosure provides at least object detection network training and object detection methods, apparatus, and electronic devices.

[0004] In a first aspect, embodiments of this disclosure provide a method for training an object detection network, including:

[0005] Obtain a training image set, wherein the training image set includes each training image;

[0006] Each training image is input into a first neural network to obtain first feature map information of each training image. Based on the first feature map information, first target detection information of each training image is obtained through an extraction module, wherein the extraction module is obtained based on the training of the first neural network.

[0007] The training images are input into the second neural network to obtain the second feature map information of each training image. Based on the second feature map information and the first feature map information, the extraction module obtains the second target detection information of each training image.

[0008] Based on the second target detection information and the first target detection information of each training image, the second neural network is trained, and a target detection network is obtained under preset conditions.

[0009] In this embodiment, each training image in the training image set is input into a first neural network, and the first target detection information of each training image is obtained through an extraction module. Then, each training image is input into a second neural network, and the second target detection information of each training image is obtained through an extraction module. Based on the second and first target detection information of each training image, the second neural network is trained to obtain a target detection network. Thus, by extracting relevant detection knowledge from the first neural network through the extraction module, and then using the extracted relevant detection knowledge to guide the training of the second neural network, the knowledge distillation process is completed. Furthermore, this scheme uses the extraction module as an intermediary, without restricting the network structures of the first and second neural networks. Therefore, it can be applied to knowledge distillation of any type of detector and across skeleton networks, improving application flexibility and also enhancing the target detection accuracy of the second neural network.

[0010] In one optional implementation, based on the first feature map information, the extraction module obtains the first target detection information for each training image, including:

[0011] Based on the target initial vector and the first feature map information in the extraction module, the query vector is obtained through the first target cross-attention layer of the extraction module;

[0012] Based on the query vector and the first feature map information, the first target detection information is obtained through the second target cross-attention layer of the extraction module.

[0013] In this embodiment of the disclosure, the target initial vector and the first feature map information are subjected to cross-attention processing to obtain the query vector, and then cross-attention processing is performed with the first feature map information to obtain the relevant detection knowledge in the first neural network. Through cross-attention processing, more attention can be paid to the target part in the training image, thereby improving accuracy.

[0014] In one optional implementation, based on the second feature map information and the first feature map information, the extraction module obtains second target detection information for each training image, including:

[0015] Based on the target initial vector and the first feature map information in the extraction module, the query vector is obtained through the first target cross-attention layer of the extraction module;

[0016] Based on the query vector and the second feature map information, the second target detection information is obtained through the second target cross-attention layer of the extraction module.

[0017] In this embodiment of the disclosure, the output of the first neural network is used to obtain a query vector, and then cross-attention is performed based on the query vector and the second feature map information to obtain the second target detection information of the second neural network, which can improve the performance of the second neural network in learning the detection knowledge in the first neural network.

[0018] In one optional implementation, the first target detection information includes first target content information and first target location information, and the second target detection information includes second target content information and second target location information;

[0019] The extraction module includes a first network branch and a second network branch. The first network branch is used to extract the first target content information or the second target content information. The network structure of the first network branch includes at least a first cross-attention layer and a second cross-attention layer.

[0020] Furthermore, the second network branch is used to extract the first target location information or the second target location information, and the network structure of the second network branch includes at least a third cross-attention layer and a fourth cross-attention layer.

[0021] In this embodiment of the disclosure, the relevant detection knowledge may include content knowledge and location knowledge, and is extracted through two independent first network branches and second network branches of the extraction module. This allows for the extraction of richer relevant detection knowledge from the first neural network, thereby improving the accuracy of training the second neural network.

[0022] In one optional implementation, satisfying the preset conditions includes:

[0023] The number of training iterations reaches a threshold, or the target loss function satisfies the convergence condition;

[0024] The target loss function includes at least a weighted sum of a first loss function and a second loss function. The first loss function represents the loss function between the first target content information and the second target content information, and the second loss function represents the loss function between the first target location information and the second target location information.

[0025] In this embodiment, the extraction module includes two network branches to obtain content knowledge and location knowledge respectively. Then, based on the weighted sum of the loss functions of the two network branches, the final target loss function can be determined to improve training accuracy.

[0026] In one optional implementation, based on the first feature map information, the extraction module obtains the first target detection information for each training image, including:

[0027] The first feature map information is input into the first network branch. Based on the first initial vector and the first feature map information in the first network branch, the first query vector is obtained through the first cross-attention layer. Based on the first query vector and the first feature map information, the first target content information is obtained through the second cross-attention layer.

[0028] The first feature map information is input into the second network branch. Based on the second initial vector in the second network branch and the first feature map information, the second query vector is obtained through the third cross-attention layer. Based on the second query vector and the first feature map information, the first target location information is obtained through the fourth cross-attention layer. The second initial vector is related to the position of the target candidate box in each training image.

[0029] In this embodiment of the disclosure, the first target content information and the second target location information of the first neural network are extracted by the first network branch and the second network branch of the extraction module, respectively, thereby improving the accuracy and richness of the relevant detection knowledge extraction.

[0030] In one optional implementation, based on the second feature map information and the first feature map information, the extraction module obtains second target detection information for each training image, including:

[0031] The first feature map information and the second feature map information are input into the first network branch. Based on the first initial vector and the first feature map information in the first network branch, the first query vector is obtained through the first cross-attention layer. Based on the first query vector and the second feature map information, the second target content information is obtained through the second cross-attention layer.

[0032] The first feature map information and the second feature map information are input into the second network branch. Based on the second initial vector in the second network branch and the first feature map information, the second query vector is obtained through the third cross-attention layer. Based on the second query vector and the second feature map information, the second target location information is obtained through the fourth cross-attention layer. The second initial vector is related to the position of the target candidate box in each training image.

[0033] In this embodiment of the disclosure, a first query vector and a second query vector are obtained based on the feature output of the first neural network. Then, based on the first network branch and the second network branch of the extraction module, the second target content information and the second target location information of the second neural network are obtained, so that the second neural network imitates the first neural network, thereby achieving the purpose of training the second neural network and improving the training accuracy of the second neural network.

[0034] In one optional implementation, the training method of the extraction module includes the following steps:

[0035] Obtain a second training image set, wherein the second training image set includes each second training image;

[0036] The second training images are input into the first neural network to obtain the third feature map information of each second training image, and the third target detection information of each second training image is obtained through the extraction module based on the third feature map information.

[0037] Based on the third object detection information, the extraction module is trained. When the number of iterations reaches a threshold or the extraction learning loss function meets the convergence condition, the trained extraction module is obtained. The extraction learning loss function includes the loss between the predicted label and the real label. The predicted label is obtained after object detection based on the third object detection information.

[0038] In this embodiment of the disclosure, the extraction module can be attached to the feature output of the first neural network, and then the extraction module can be obtained by training the first neural network, so that the extraction module can extract detection knowledge related to the detection task.

[0039] In one optional implementation, the third target detection information includes third target content information and third target location information. Then, based on the third feature map information, the extraction module obtains the third target detection information for each of the second training images, including:

[0040] The third feature map information is input into the first network branch of the extraction module to obtain the third target content information of each second training image;

[0041] The third feature map information is input into the second network branch of the extraction module to obtain the third target location information of each second training image.

[0042] In this embodiment of the disclosure, the extraction module can be divided into a first network branch and a second network branch, and these two network branches are trained separately with independent network parameters, which can improve the accuracy of content knowledge and location knowledge extraction.

[0043] In one optional implementation, the predicted label includes a predicted target location label and a predicted category label, and the predicted target location label and the predicted category label are obtained based on the third target content information or the third target location information; the real label includes a real target location label and a real category label.

[0044] The extraction learning loss function includes the loss between the predicted target location label and the real target location label, and the sum of the losses between the predicted category label and the real target category label.

[0045] In this embodiment of the disclosure, the training of the extraction module based on two network branches can include a position loss function and a category loss function, thereby improving training accuracy.

[0046] Secondly, embodiments of this disclosure also provide a target detection method, characterized in that it includes:

[0047] Acquire the image to be detected;

[0048] The target detection network is used to detect targets in the image to be detected to obtain target categories. The target detection network is obtained based on the target detection network training method described in the first aspect or any possible implementation of the first aspect.

[0049] In this embodiment, the first target detection information of the first neural network is extracted by the extraction module, and the second target detection information of the second neural network is extracted, thereby training to obtain a target detection network. Based on the trained target detection information, target detection can be performed on the image to be detected to obtain the target category detected from the image to be detected. In this way, the second neural network is lighter than the first neural network and has a similar target detection function to the first neural network. Therefore, the second neural network can be directly deployed to the corresponding application scenario. The second neural network also has lower requirements for hardware devices, which improves the application flexibility and can also ensure the performance of target detection.

[0050] Thirdly, embodiments of this disclosure also provide a target detection network training apparatus, comprising:

[0051] The first acquisition module is used to acquire a training image set, wherein the training image set includes each training image;

[0052] A first processing module is configured to input each training image into a first neural network to obtain first feature map information of each training image, and based on the first feature map information, to obtain first target detection information of each training image through an extraction module, wherein the extraction module is obtained based on the training of the first neural network;

[0053] The second processing module is used to input each training image into the second neural network to obtain the second feature map information of each training image, and to obtain the second target detection information of each training image through the extraction module based on the second feature map information and the first feature map information.

[0054] The first training module is used to train the second neural network based on the second target detection information and the first target detection information of each training image, and to obtain the target detection network under preset conditions.

[0055] Fourthly, embodiments of this disclosure also provide a target detection device, comprising:

[0056] The second acquisition module is used to acquire the image to be detected;

[0057] The detection module is used to perform target detection on the image to be detected using a target detection network to obtain the target category. The target detection network is obtained based on the target detection network training method described in the first aspect or any possible implementation of the first aspect.

[0058] For a description of the effects of the aforementioned target detection network or target detection device, electronic equipment, and computer-readable storage medium, please refer to the description of the aforementioned target detection network or target detection method; it will not be repeated here.

[0059] Fifthly, an optional implementation of this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the steps of any of the possible implementations of the first aspect or the second aspect described above.

[0060] In a sixth aspect, an optional implementation of this disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the possible implementations of the first aspect or the second aspect described above.

[0061] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure.

[0062] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0063] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0064] Figure 1 A flowchart of a target detection network training method provided in an embodiment of this disclosure is shown;

[0065] Figure 2 A logical schematic diagram of the extraction module training process provided in an embodiment of this disclosure is shown;

[0066] Figure 3 A logical schematic diagram of the second neural network training process provided in an embodiment of this disclosure is shown;

[0067] Figure 4 A flowchart of a target detection method provided by an embodiment of this disclosure is shown;

[0068] Figure 5 A schematic diagram of a target detection network training apparatus provided in an embodiment of this disclosure is shown;

[0069] Figure 6 A schematic diagram of a target detection device provided in an embodiment of this disclosure is shown;

[0070] Figure 7 A schematic diagram of an electronic device provided in an embodiment of the present disclosure is shown. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown herein can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0072] To facilitate understanding of the technical solutions disclosed herein, the technical terms used in the embodiments of this disclosure will first be explained:

[0073] Knowledge distillation: This refers to the distillation of knowledge. For neural networks, knowledge is contained in the trained model parameters. Knowledge distillation is a method of model compression. In knowledge distillation, the supervision information of a large neural network with better performance is extracted to construct a small neural network, which also has good performance and accuracy. The large neural network can be called the teacher network, and the small neural network can be called the student network. That is, the knowledge learned in the teacher network is transferred to the student network.

[0074] Backbone network: The backbone network is the main network of a neural network, mainly used for feature extraction and generating feature maps for use by subsequent network modules. For example, the backbone network is a convolutional neural network. In the field of computer vision, feature extraction of images is usually performed first through the backbone network. This part is the basis for subsequent downstream tasks, which need to be based on the extracted features, such as classification tasks.

[0075] Feature Pyramid Network (FPN): It mainly solves the multi-scale problem in object detection, and utilizes the high resolution of low-level features and the high semantic information of high-level features to achieve the prediction effect by fusing the features of these different layers.

[0076] Research has shown that object detection is a crucial problem in computer vision, with wide applications in fields such as intelligent surveillance, smart cities, smart homes, autonomous driving, and robotics. In practical deployments, due to the computing power limitations of different hardware devices, knowledge distillation algorithms can be used to compress neural networks to meet the requirements of different hardware devices regarding neural network parameters, inference speed, etc. However, in related technologies, knowledge distillation algorithms are usually only applied to homogeneous networks, requiring large and small neural networks to belong to the same detector and have the same backbone network structure, which reduces the flexibility and applicable scenarios of knowledge distillation algorithms.

[0077] Based on the above research, this disclosure provides a method for training an object detection network. An adaptive extraction module is designed, which is obtained by training a first neural network. Then, when training a second neural network based on a training image set, each training image is input into the first neural network, and the extraction module obtains the first object detection information for each training image. The second training image is then input into the second neural network, and the extraction module obtains the second object detection information for each training image. Based on the second and first object detection information of each training image, the second neural network is trained. Thus, in this embodiment, the extraction module extracts relevant object detection knowledge from the first neural network and transmits it to the second neural network without restricting the backbone network or detector network architecture of the first and second neural networks. Therefore, the application is more flexible and can be applied to knowledge distillation across detector network architectures and backbone network architectures. Furthermore, the training of the second neural network is guided by the relevant object detection knowledge extracted from the first neural network, ensuring the performance of the second neural network.

[0078] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.

[0079] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0080] To facilitate understanding of this embodiment, a detailed description of the object detection network training method disclosed in this disclosure is provided first. The execution entity of the object detection network training method provided in this disclosure is generally an electronic device with a certain computing capability. This electronic device may include, for example, a terminal device, a server, or other processing devices. The terminal device can be a user equipment (UE), mobile device, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. A personal digital assistant is a handheld electronic device that possesses some functions of a computer. It can be used to manage personal information, browse the internet, send and receive emails, etc., and generally does not have a keyboard; it can also be called a handheld computer. In some possible implementations, this object detection network training method can be implemented by a processor calling computer-readable instructions stored in memory.

[0081] The following describes the target detection network training method provided in this embodiment, taking the server as the execution subject as an example.

[0082] See Figure 1 The diagram shows a flowchart of a target detection network training method provided in this embodiment of the present disclosure. The method includes:

[0083] S101: Obtain the training image set, which includes each training image.

[0084] In this embodiment of the disclosure, the application scenarios of the target detection method and the target detection network training method are not limited. The training image set can be training images in different scenarios, such as intelligent transportation, robots, autonomous driving and other application scenarios.

[0085] The training images in the training image set can be, for example, labeled images, meaning each training image includes the location and category labels of the labeled targets. These images can be used to calculate the location and category loss functions during the training of the second neural network. For example, since the training of the second neural network is based on the first neural network in this embodiment, the second neural network can also directly mimic the output of the first neural network. Therefore, the location and category output by the second neural network can be used as the location and category labels during the training process of the second neural network. This embodiment does not impose any limitations on this approach.

[0086] S102: Input each training image into the first neural network to obtain the first feature map information of each training image. Based on the first feature map information, obtain the first target detection information of each training image through the extraction module. The extraction module is obtained based on the training of the first neural network.

[0087] In this embodiment of the disclosure, a general knowledge distillation method is provided, which includes an additional extraction module. This extraction module is first trained based on a first neural network to extract relevant target detection knowledge from the first neural network. Then, the extraction module can be connected to a second neural network to guide the training of the second neural network using the relevant target detection knowledge extracted from the first neural network.

[0088] For example, if the first neural network is a Teacher network and the second neural network is a Student network, when training the extraction module, the network parameters of other network structures of the first neural network can be fixed, and only the extraction module can be trained. Compared with training the complete first neural network, this can greatly improve efficiency.

[0089] For ease of understanding, the extraction module is briefly introduced here. The extraction module can be understood as a relatively independent module. The network structure of the extraction module mainly includes two parts, where the first part is used to obtain the query vector and the second part is used to extract the target detection information. The output of the first part is the input of the second part, and the network structures of the two parts can be the same or similar. In this embodiment of the disclosure, there are no restrictions on the network structure of the extraction module.

[0090] For example, one possible network structure for the extraction module is as follows: the network structure of the first part consists of an input layer, a first target self-attention layer, a residual connection and normalization layer, a first target cross-attention layer, a residual connection and normalization layer, a feedforward network layer, and a residual connection and normalization layer; the network structure of the second part consists of a second target self-attention layer, a residual connection and normalization layer, a second target cross-attention layer, a residual connection and normalization layer, a feedforward network layer, a residual connection and normalization layer, and an output layer.

[0091] The input layer of the first part can be either the first feature map information obtained through the first neural network or the second feature map information obtained through the second neural network. Taking the first feature map information as an example, the subsequent processing logic is explained as follows: Then, the target initial vector is fused and extracted through the first target self-attention layer. The output of the first target self-attention layer and the target initial vector are subjected to residual connection and normalization processing through the residual connection and normalization layer. Then, the output of the first target cross-attention layer and the first feature map information are subjected to cross-feature fusion through the first target cross-attention layer. Then, the input and output of the first target cross-attention layer are subjected to residual connection and normalization processing through another residual connection and normalization layer. Then, the feature nonlinearity is processed through the feedforward network layer. Then, the input and output of the feedforward network layer are subjected to residual connection and normalization processing through another residual connection and normalization layer to obtain the query vector. The query vector is then input into the second part and processed by the network layers of the second part, which are similar to the processing of the network layers in the first part. Finally, the first target detection information can be output.

[0092] When specifically performing step S102, this disclosure provides a possible implementation method:

[0093] 1) Input each training image into the first neural network to obtain the first feature map information of each training image.

[0094] Specifically, each training image is input into the first neural network, and features are extracted from each training image through the backbone network of the first neural network to obtain the first feature map information of each training image.

[0095] Furthermore, in this embodiment of the disclosure, the backbone network can be combined with the FPN method to obtain first feature map information at multiple different scales for use in the training of the subsequent extraction module, which can improve the robustness of the feature extraction and detection algorithm to detect targets at different scales.

[0096] 2) Based on the initial target vector and the first feature map information in the extraction module, the query vector is obtained through the first target cross-attention layer of the extraction module.

[0097] The initial target vector can be understood as the parameter information obtained after pre-training in the extraction module.

[0098] 3) Based on the query vector and the first feature map information, the first target detection information is obtained through the second target cross-attention layer of the extraction module.

[0099] In this embodiment of the present disclosure, when training the second neural network based on the first neural network, various parameters of the trained first neural network and the trained extraction module are fixed, so as to extract the first target detection information in the first neural network based on the trained extraction module.

[0100] S103: Input each training image into the second neural network to obtain the second feature map information of each training image. Based on the second feature map information and the first feature map information, obtain the second target detection information of each training image through the extraction module.

[0101] When performing step S103, this embodiment of the present disclosure also provides a possible implementation:

[0102] 1) Input each training image into the second neural network to obtain the second feature map information of each training image.

[0103] Specifically, this includes: inputting each training image into a second neural network, extracting features from each training image through the backbone network of the second neural network, and obtaining the second feature map information of each training image.

[0104] Similarly, in this embodiment of the present disclosure, FPN can also be applied to the backbone network in the second neural network to obtain second feature map information at different scales, thereby improving the target detection effect of the second neural network on images of different scales.

[0105] 2) Based on the initial target vector and the first feature map information in the extraction module, the query vector is obtained through the first target cross-attention layer of the extraction module.

[0106] In this embodiment of the present disclosure, the query vector is obtained by using the first feature map information output by the first neural network, which is then used for target detection information extraction by the second neural network. This allows the second neural network to learn the intermediate feature information of the first neural network. Instead of directly transmitting the first feature map information, the second neural network combines the first feature map information with cross attention. Cross attention can better reflect the activation of the backbone network, thereby improving the performance of the second neural network in knowledge distillation.

[0107] 3) Based on the query vector and the second feature map information, the second target detection information is obtained through the second target cross-attention layer of the extraction module.

[0108] S104: Train the second neural network based on the second target detection information and the first target detection information of each training image, and obtain the target detection network under the condition of satisfying the preset conditions.

[0109] In one possible embodiment, the object detection network is the trained second neural network. Training the second neural network means training to obtain the values ​​of each network parameter of the second neural network, such as the network parameters of the backbone network of the second neural network. Compared with the first neural network, the second neural network is a lighter and smaller network model. Since the second neural network is trained based on the first neural network, the second neural network can have the functions of the first neural network. The second neural network can be deployed on hardware devices with lower requirements or edge computing devices to meet the needs of different hardware devices.

[0110] Additionally, it should be noted that in this embodiment of the disclosure, the second neural network is trained based on the first target detection information and the second target detection information, so that the second neural network can absorb detection-related knowledge from the first neural network and imitate the first target detection information output by the first neural network. The loss function between the first target detection information and the second target detection information can also be called the knowledge distillation loss function. Of course, in this embodiment of the disclosure, during the training of the second neural network, the target loss function also includes the loss of the detection algorithm itself, such as classification loss and target localization loss.

[0111] In this embodiment, a training image set is acquired, and each training image in the training image set is input into a first neural network to obtain first feature map information for each training image. Based on the first feature map information, first target detection information for each training image is obtained through an extraction module. Then, each training image is input into a second neural network to obtain second feature map information for each training image. Based on the second and first feature map information, second target detection information for each training image is obtained through an extraction module. Furthermore, based on the second and first target detection information for each training image, the second neural network is trained to obtain a target detection network. Thus, by designing an extraction module, training the first neural network to obtain the extraction module, and then extracting the first target detection information from the first neural network, the extracted first target detection information is used to guide the training of the second neural network. Using this scheme, knowledge distillation between any detector architecture and any skeleton network structure can be achieved, reducing the adaptation cost of the first and second neural networks to meet the detector knowledge distillation requirements, and improving the accuracy and performance of the second neural network.

[0112] Based on the above embodiments, in this embodiment of the disclosure, relevant detection knowledge is first extracted from the first neural network, and then the extracted knowledge is used to guide the training of the second neural network. In one possible embodiment, the extracted first target detection information includes first target content information and first target location information, and the second target detection information includes second target content information and second target location information. That is, content knowledge and location knowledge are extracted from the first neural network to guide the training of the second neural network, which improves the knowledge distillation performance. Accordingly, this embodiment of the disclosure provides corresponding extraction and training methods.

[0113] In this embodiment of the disclosure, the extraction module includes a first network branch and a second network branch. The first network branch is used to extract first target content information or second target content information, and the network structure of the first network branch includes at least a first cross-attention layer and a second cross-attention layer. The second network branch is used to extract first target location information or second target location information, and the network structure of the second network branch includes at least a third cross-attention layer and a fourth cross-attention layer.

[0114] Regarding step S102 above, which involves extracting the first target detection information of each training image through the first neural network based on the first feature map information, a possible implementation method is provided:

[0115] 1) Input the first feature map information into the first network branch. Based on the first initial vector and the first feature map information in the first network branch, obtain the first query vector through the first cross-attention layer. Based on the first query vector and the first feature map information, obtain the first target content information through the second cross-attention layer.

[0116] For example, the first feature map information is F T The first cross-attention layer is denoted as f. content (·), the second cross-attention layer is denoted as f extract1 (·), the first initial vector is E, and the first query vector is denoted as E. The first target content information is denoted as Specifically, we can obtain:

[0117] First query vector

[0118] The first target content information is:

[0119] Among them, W content and θ content The learnable parameters in the first and second cross-attention layers are respectively determined through pre-training, and the first initial vector E can also be obtained through pre-training.

[0120] 2) Input the first feature map information into the second network branch. Based on the second initial vector and the first feature map information in the second network branch, obtain the second query vector through the third cross-attention layer. Based on the second query vector and the first feature map information, obtain the first target location information through the fourth cross-attention layer. The second initial vector is related to the position of the target candidate box in each training image.

[0121] For example, the first feature map information is F T The third cross-attention layer is denoted as f. pos (·), the fourth cross-attention layer is denoted as f. extract2 (·), the second initial vector is E′, and the second query vector is denoted as (·). The location information of the first target is denoted as Specifically, we can obtain:

[0122] Second query vector

[0123] The first target location information is:

[0124] Among them, W pos and θ pos The learnable parameters in the third and fourth cross-attention layers are determined through pre-training. The second initial vector differs from the first initial vector; it is related to positional information and can be generated through fully connected layers (FC) using the randomly jittered positions of real target candidate boxes. This indicates that the network parameters of the FC layer include real target candidate boxes and are obtained by random jittering based on real target candidate boxes. The network parameters of the FC layer can also be obtained through pre-training.

[0125] In this embodiment of the disclosure, two independent first network branches and second network branches are used. The network parameters of the second network branches and the second network branches are not shared. The main difference is that the first initial vector in the first network branch can be obtained by a set of learnable parameters, which does not include target location information. In the second network branch, the second initial vector is related to the target location information. However, the outputs of the first network branch and the second network branch can be used for the detection task to obtain the candidate box position and category of the detected target.

[0126] In step S103 above, based on the second feature map information and the first feature map information, the extraction module obtains the second target detection information for each training image, specifically including:

[0127] 1) Input the first feature map information and the second feature map information into the first network branch. Based on the first initial vector and the first feature map information in the first network branch, obtain the first query vector through the first cross-attention layer. Based on the first query vector and the second feature map information, obtain the second target content information through the second cross-attention layer.

[0128] In this embodiment of the disclosure, when dealing with the second target content information and the second target location information extracted by the second neural network, both the first query vector and the second query vector are obtained using the first feature map information output by the first neural network. Furthermore, during the training of the second neural network, all network parameters of the extraction module are fixed in order to train the second neural network more accurately and make the second neural network more accurately mimic the first neural network.

[0129] For example, the first query vector is denoted as The second feature map information is F S The second target content information is denoted as Specifically, we can obtain:

[0130] The second target content information is:

[0131] 2) Input the first feature map information and the second feature map information into the second network branch. Based on the second initial vector and the first feature map information in the second network branch, obtain the second query vector through the third cross attention layer. Based on the second query vector and the second feature map information, obtain the second target location information through the fourth cross attention layer. The second initial vector is related to the position of the target candidate box in each training image.

[0132] For example, the second feature map information is F S The second query vector is The location information of the second target is denoted as Specifically, we can obtain:

[0133] The location information of the second target is:

[0134] Furthermore, based on the first and second network branches included in the extraction module, when training the second neural network, the preset training conditions include reaching a threshold number of iterations or the target loss function satisfying a convergence condition. The target loss function includes at least a weighted sum of the first and second loss functions. The first loss function represents the loss between the first target content information and the second target content information, and the second loss function represents the loss between the first target location information and the second target location information. For example, the loss function between the first target detection information and the second target detection information in the target loss function for training the second neural network can also be called the knowledge distillation loss function, denoted as... but

[0135] in, Let λ represent the first loss function, and λ1 represent the weights of the first loss function. Let λ represent the second loss function, and λ2 represent the weights of the second loss function.

[0136] Among them, During the calculation process, positive and negative predictions can be decoupled, that is, predictions containing the target and predictions not containing the target are calculated separately. Specifically:

[0137]

[0138] "*" indicates that it can be either content or pos, N ′ NN represents the number of true target candidate boxes. ′ The number of candidate boxes generated by random jittering based on real target candidate boxes, i.e., N, can be represented by E or E0. ′ The final number of query vectors. Indicates the positive set, The negative set is represented by the mean squared error (MSE), which is the calculation of the mean squared error.

[0139] Thus, in this embodiment of the disclosure, detection-related knowledge, namely content knowledge and location knowledge, is extracted through the first query vector and the second query vector. Furthermore, during the training of the second neural network, the network parameters of the extraction module are fixed and directly applied to the feature output of the second neural network to mimic the output of the first neural network, thereby improving training efficiency. This achieves a general knowledge distillation method that can be applied not only to different types of detectors but also to knowledge distillation scenarios across skeleton networks, reducing adaptation costs and improving flexibility.

[0140] The training process of the extraction module in this embodiment is described below. In this embodiment, the extraction module can be attached to the feature output of the first neural network and trained based on the first neural network. To improve the training efficiency and accuracy of the extraction module, other network parameters of the first neural network can be fixed during the training process. Specifically, this disclosure provides a possible implementation method for the training of the extraction module:

[0141] 1) Obtain the second training image set, wherein the second training image set includes each second training image.

[0142] 2) Input each second training image into the first neural network to obtain the third feature map information of each second training image, and obtain the third target detection information of each second training image through the extraction module based on the third feature map information.

[0143] The third target detection information includes third target content information and third target location information. That is, during the training process of the extraction module, the first and second network branches of the extraction module are trained separately. For this step, this disclosure provides a possible implementation method:

[0144] The third feature map information is input into the first network branch of the extraction module to obtain the third target content information of each second training image; the third feature map information is input into the second network branch of the extraction module to obtain the third target location information of each second training image.

[0145] In the training of the extraction module, the process by which the first and second network branches extract the third target content information and the third target location information is similar to the process of extracting the first target content information and the first target location information during the training of the second neural network in the above embodiment. However, in this case, the network parameters of the extraction module need to be trained and learned, rather than being fixed. For example, the network parameters trained and learned in the extraction module include E and W. content θ content FC, W pos and θ pos wait.

[0146] 3) Based on the third object detection information, train the extraction module. When the number of iterations reaches the threshold or the extraction learning loss function meets the convergence condition, obtain the extraction module after training. The extraction learning loss function includes the loss between the predicted label and the real label. The predicted label is obtained after object detection based on the third object detection information.

[0147] Specifically, the predicted labels include predicted target location labels and predicted category labels, which are obtained based on third target content information or third target location information. The true labels include true target location labels and true category labels. The extraction learning loss function includes the loss between predicted target location labels and true target location labels, and the sum of the losses between predicted category labels and true target category labels.

[0148] In this embodiment, after extracting the third target detection information, to ensure the relevance of the extracted third target detection information to the detection task, a bipartite graph matching algorithm can be used to assign real target boxes to the third target detection information for training the detection task. Specifically, based on the third target content information and the third target location information, a set of N predictions of a fixed size can be obtained through the feed-forward network (FFN) layer of the extraction module. Each prediction can be whether a target was found, or the candidate box location and category of the found target. For the first network branch, since the correspondence between the location and the query vector is not predefined in the first network branch, bipartite graph matching can be performed to divide the query vector into positive sets. and negative set σ content () is defined as the index of the i-th query vector. If the i-th query vector belongs to the positive set, then This represents the location of the true target bounding box, specifically the label of the true target location corresponding to the i-th query vector in the first network branch. This represents the true target category label corresponding to the i-th query vector in the first network branch; otherwise, if the i-th query vector belongs to the negative set, there is no target box, and... This refers to the background label (i.e., no target); however, for the second network branch, the query vector is related to location information, meaning the correspondence between the query vector and the location is known. The second network branch... pos The definitions in () and the first network branch are similar, and correspondingly This represents the true target location label corresponding to the i-th query vector in the second network branch. Let represent the true target category label corresponding to the i-th query vector in the second network branch. Therefore, the extraction learning loss function of the extraction module is denoted as . It can be represented as:

[0149]

[0150] Where "*" indicates that it can be either content or pos, and N is the total number of positive and negative sets. ′ Let be the number of positive sets. Indicates the positive set, Represents the negative set. This represents the loss function between the predicted bounding box location labels and the ground truth bounding box location labels. This represents the loss function between the predicted class label and the true class label. Correspondingly, this indicates the predicted target location label or the predicted category label.

[0151] In this embodiment of the disclosure, the extraction module is trained based on the first neural network, so that the extraction module can extract detection knowledge related to the detection task from the first neural network. In addition, during the training of the extraction module, other network parameters of the first neural network can be fixed to improve training efficiency.

[0152] It should be noted that the network structure of the extraction module is not limited in this embodiment. For example, a deformable transformer can be used to extract detection knowledge from the first neural network and use the extracted detection knowledge to guide the training of the second neural network. Therefore, the detector and backbone network of the second neural network and the first neural network are not limited. For example, they can be deformable end-to-end target detection (Detection Transformer, DETR), residual network (ResNet), etc. This embodiment does not impose any restrictions.

[0153] Based on the above embodiments, the logical principle of the training process of the object detection network will be briefly explained below. The training process of the object detection network in this embodiment can be divided into two aspects: the first aspect is the training of the extraction module, and the second aspect is the training of the second neural network, which will be described below.

[0154] The first aspect: training of the extraction module.

[0155] See Figure 2 The diagram shown is a logical principle diagram of the extraction module training process in an embodiment of this disclosure. Figure 2 As shown, a second training image set is obtained, and each second training image in the second training image set is input into the first neural network to obtain the third feature map information of each training image. During the training extraction module, other network parameters of the second neural network are fixed, for example, such as... Figure 2 As shown, the backbone network and FPN network of the first neural network are fixed. It can be understood that the other network parameters of the second neural network are also pre-trained at this time.

[0156] like Figure 2 As shown, the third feature map information is input into the first network branch, and then processed through the first initial vector E, the third feature map information, and the first cross-attention layer f. content (·), to obtain the first query vector Here, E can be randomly generated from a set of learnable parameters, and then passed through a second cross-attention layer f. extract1 (·), obtain the third target content information.

[0157] Similarly, the third feature map information is input into the second network branch, and then processed through the second initial vector E′, the third feature map information, and the third cross-attention layer f. pos (·), to obtain the second query vector In the second network branch, E′ is obtained based on the randomly jittered true target bounding box. It is generated by combining FC, and then passed through the fourth cross-attention layer f. extract2 (·), obtain the location information of the third target.

[0158] Then, the detection task is trained based on the third target content information and the second target location information to obtain the detection results, such as whether the target was detected, or the category and location of the detected target. The extraction module is trained based on the predicted box position and the real target box position, as well as the loss function between the preset category label and the real category label, so as to obtain the trained extraction module and determine the values ​​of each network parameter in the extraction module.

[0159] The second aspect: training the second neural network.

[0160] See Figure 3 The diagram shown is a logical schematic of the second neural network training process in an embodiment of this disclosure. Figure 3 As shown, each training image in the training image set is input into the first neural network and the second neural network respectively to obtain the first feature map information and the second feature map information of each training image. During the training process of the second neural network, the network parameters of the first neural network and the extraction module are fixed, while the network parameters of the second neural network are trained.

[0161] Furthermore, such as Figure 3 As shown, the first feature map information is input into the extraction module of the first neural network and the extraction module of the second neural network, respectively, and... Figure 2The extraction module's processing is the same. Based on the first feature map information, a first query vector and a second query vector are obtained. These first and second query vectors are then subjected to cross-attention processing with the first and second feature map information, respectively. This ultimately yields the first target content information and the first target location information of the first neural network, and the second target content information and the second target location information of the second neural network. Based on the loss functions between the first and second target content information, and between the first and second target location information, the second neural network is trained, resulting in the trained second neural network, i.e., the object detection network. The training of the second neural network may also include position and category loss functions for the detection task itself (this part is discussed in...). Figure 3 (Not shown in the image).

[0162] Thus, in this embodiment of the disclosure, a general knowledge distillation method is realized by designing an extraction module. It does not restrict the network structure of the detector and backbone network of the first and second neural networks, and can be applied to any detector architecture and any backbone network architecture. This improves flexibility, reduces the adaptation cost of the first and second neural networks to meet the knowledge distillation requirements, and can also improve the accuracy of the second neural network.

[0163] Compared to knowledge distillation methods in related technologies, which use a first feature representation output by a first neural network and a second feature representation output by a second neural network to calculate a knowledge distillation loss function for learning, this learning method requires that the feature pixels of the first and second neural networks be aligned and matched. It also requires that the first and second neural networks be isomorphic. When the first and second neural networks have different structures, the second neural network will fail or make mistakes in learning from the first neural network due to different semantic differences. However, in this embodiment, knowledge is extracted from the first neural network by an extraction module, and then the extraction module guides the training and learning of the second neural network based on the knowledge information extracted from the first neural network. Therefore, this embodiment does not need to limit the network structure of the first and second neural networks, and can realize a general knowledge distillation method with a wider range of applications.

[0164] In this embodiment of the disclosure, after the second neural network is trained, it can be deployed in an electronic device. Since the second neural network is smaller and lighter than the first neural network, it has lower requirements for electronic devices. For example, the second neural network can be deployed in some edge computing devices to improve task processing efficiency.

[0165] Furthermore, object detection can be performed based on the trained second neural network, namely the object detection network. Accordingly, this disclosure also provides an object detection method, see reference. Figure 4 The diagram shown is a flowchart of a target detection method in an embodiment of this disclosure, including:

[0166] S401: Acquire the image to be detected.

[0167] S402: Use the trained object detection network to perform object detection on the image to be detected and obtain the object category.

[0168] The target detection network in this embodiment can be obtained by the target detection network training method in the above embodiments. The specific implementation method is the same as in the above embodiments, and will not be described again here.

[0169] In this embodiment, the application scenarios of the target detection method and the training method of the target detection network are not limited. The target detection network obtained by training on the training image samples corresponding to different application scenarios can be applied to the corresponding application scenarios for target detection. For example, in the field of intelligent monitoring, the target detection network can be used to detect human bodies in the monitoring image to be detected, determine whether there is a human body in the monitoring image to be detected, and determine whether the human body is the target user if it is determined to exist.

[0170] In this embodiment of the disclosure, the extraction module applies the extracted features to the feature output of the second neural network to mimic the output of the first neural network, thus completing knowledge distillation. This enables the second neural network to also achieve the target detection function of the first neural network, allowing the second neural network to be directly applied for target detection. This reduces the requirements for deployment hardware and improves the performance of target detection.

[0171] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0172] Based on the same inventive concept, this disclosure also provides an object detection network training device corresponding to the object detection network training method. Since the principle of the device in this disclosure for solving the problem is similar to the object detection network training method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0173] Reference Figure 5 The diagram shown is a schematic of a target detection network training device provided in an embodiment of this disclosure. The device includes:

[0174] The first acquisition module 51 is used to acquire a training image set, wherein the training image set includes each training image;

[0175] The first processing module 52 is used to input each training image into the first neural network to obtain the first feature map information of each training image, and to obtain the first target detection information of each training image through the extraction module based on the first feature map information, wherein the extraction module is obtained based on the training of the first neural network;

[0176] The second processing module 53 is used to input each training image into the second neural network to obtain the second feature map information of each training image, and to obtain the second target detection information of each training image through the extraction module based on the second feature map information and the first feature map information.

[0177] The first training module 54 is used to train the second neural network based on the second target detection information and the first target detection information of each training image, and obtain the target detection network under preset conditions.

[0178] In an optional implementation, when obtaining the first target detection information of each training image through the extraction module based on the first feature map information, the first processing module 52 is used to:

[0179] Based on the target initial vector and the first feature map information in the extraction module, the query vector is obtained through the first target cross-attention layer of the extraction module;

[0180] Based on the query vector and the first feature map information, the first target detection information is obtained through the second target cross-attention layer of the extraction module.

[0181] In an optional implementation, when the extraction module obtains the second target detection information of each training image based on the second feature map information and the first feature map information, the second processing module 53 is used to:

[0182] Based on the target initial vector and the first feature map information in the extraction module, the query vector is obtained through the first target cross-attention layer of the extraction module;

[0183] Based on the query vector and the second feature map information, the second target detection information is obtained through the second target cross-attention layer of the extraction module.

[0184] In one optional implementation, the first target detection information includes first target content information and first target location information, and the second target detection information includes second target content information and second target location information;

[0185] The extraction module includes a first network branch and a second network branch. The first network branch is used to extract the first target content information or the second target content information. The network structure of the first network branch includes at least a first cross-attention layer and a second cross-attention layer.

[0186] Furthermore, the second network branch is used to extract the first target location information or the second target location information, and the network structure of the second network branch includes at least a third cross-attention layer and a fourth cross-attention layer.

[0187] In one optional implementation, the first training module 54 is used to: reach a threshold number of iterations for training, or satisfy the convergence condition for the target loss function;

[0188] The target loss function includes at least a weighted sum of a first loss function and a second loss function. The first loss function represents the loss function between the first target content information and the second target content information, and the second loss function represents the loss function between the first target location information and the second target location information.

[0189] In an optional implementation, when obtaining the first target detection information of each training image through the extraction module based on the first feature map information, the first processing module 52 is used to:

[0190] The first feature map information is input into the first network branch. Based on the first initial vector and the first feature map information in the first network branch, the first query vector is obtained through the first cross-attention layer. Based on the first query vector and the first feature map information, the first target content information is obtained through the second cross-attention layer.

[0191] The first feature map information is input into the second network branch. Based on the second initial vector in the second network branch and the first feature map information, the second query vector is obtained through the third cross-attention layer. Based on the second query vector and the first feature map information, the first target location information is obtained through the fourth cross-attention layer. The second initial vector is related to the position of the target candidate box in each training image.

[0192] In an optional implementation, when the extraction module obtains the second target detection information of each training image based on the second feature map information and the first feature map information, the second processing module 53 is used to:

[0193] The first feature map information and the second feature map information are input into the first network branch. Based on the first initial vector and the first feature map information in the first network branch, the first query vector is obtained through the first cross-attention layer. Based on the first query vector and the second feature map information, the second target content information is obtained through the second cross-attention layer.

[0194] The first feature map information and the second feature map information are input into the second network branch. Based on the second initial vector in the second network branch and the first feature map information, the second query vector is obtained through the third cross-attention layer. Based on the second query vector and the second feature map information, the second target location information is obtained through the fourth cross-attention layer. The second initial vector is related to the position of the target candidate box in each training image.

[0195] In an optional implementation, a second training module 55 is further included, which performs the following steps in response to the training method of the extraction module:

[0196] Obtain a second training image set, wherein the second training image set includes each second training image;

[0197] The second training images are input into the first neural network to obtain the third feature map information of each second training image, and the third target detection information of each second training image is obtained through the extraction module based on the third feature map information.

[0198] Based on the third object detection information, the extraction module is trained. When the number of iterations reaches a threshold or the extraction learning loss function meets the convergence condition, the trained extraction module is obtained. The extraction learning loss function includes the loss between the predicted label and the real label. The predicted label is obtained after object detection based on the third object detection information.

[0199] In an optional implementation, the third target detection information includes third target content information and third target location information. Then, when obtaining the third target detection information for each of the second training images based on the third feature map information through the extraction module, the second training module 55 is used for:

[0200] The third feature map information is input into the first network branch of the extraction module to obtain the third target content information of each second training image;

[0201] The third feature map information is input into the second network branch of the extraction module to obtain the third target location information of each second training image.

[0202] In one optional implementation, the predicted label includes a predicted target location label and a predicted category label, and the predicted target location label and the predicted category label are obtained based on the third target content information or the third target location information; the real label includes a real target location label and a real category label.

[0203] The extraction learning loss function includes the loss between the predicted target location label and the real target location label, and the sum of the losses between the predicted category label and the real target category label.

[0204] Based on the same inventive concept, this disclosure also provides a target detection device corresponding to the target detection method. Since the principle by which the device solves the problem is similar to the target detection method described above, the implementation of the device can be referred to the implementation of the method, and repeated details will not be elaborated further. See also... Figure 6 The diagram shown is a schematic representation of a target detection device provided in an embodiment of this disclosure. The device includes:

[0205] The second acquisition module 61 is used to acquire the image to be detected;

[0206] The detection module 62 is used to perform target detection on the image to be detected using a trained target detection network to obtain the target category.

[0207] The trained target detection network is generated based on the target detection network training method in the embodiments of this disclosure.

[0208] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0209] This disclosure also provides an electronic device, such as... Figure 7 The diagram shown is a schematic representation of the structure of an electronic device provided in an embodiment of this disclosure, including:

[0210] A processor 71 and a memory 72; the memory 72 stores machine-readable instructions executable by the processor 71, and the processor 71 executes the machine-readable instructions stored in the memory 72. When the machine-readable instructions are executed by the processor 71, the processor 71 performs the following steps:

[0211] Obtain a training image set, wherein the training image set includes each training image;

[0212] Each training image is input into a first neural network to obtain first feature map information of each training image. Based on the first feature map information, first target detection information of each training image is obtained through an extraction module, wherein the extraction module is obtained based on the training of the first neural network.

[0213] The training images are input into the second neural network to obtain the second feature map information of each training image. Based on the second feature map information and the first feature map information, the extraction module obtains the second target detection information of each training image.

[0214] Based on the second target detection information and the first target detection information of each training image, the second neural network is trained, and a target detection network is obtained under preset conditions.

[0215] Alternatively, the processor 71 may also perform the following steps:

[0216] Acquire the image to be detected;

[0217] The trained object detection network is used to perform object detection on the image to be detected, and the object category is obtained.

[0218] The aforementioned memory 72 includes a main memory 721 and an external memory 722; the main memory 721, also known as internal memory, is used to temporarily store the computational data in the processor 71, as well as the data exchanged with external memory 722 such as a hard disk. The processor 71 exchanges data with the external memory 722 through the main memory 721.

[0219] The specific execution process of the above instructions can be referred to the steps of the target detection network training method or target detection method described in the embodiments of this disclosure, and will not be repeated here.

[0220] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the object detection network training method or object detection method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0221] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the target detection network training method or the target detection method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0222] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0223] If the technical solution disclosed herein involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution disclosed herein involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0224] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0225] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0226] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0227] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0228] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A method for training an object detection network, characterized in that, include: Obtain a training image set, wherein the training image set includes each training image; Each training image is input into a first neural network to obtain first feature map information of each training image. Based on the first feature map information, first target detection information of each training image is obtained through an extraction module, wherein the extraction module is obtained based on the training of the first neural network. The training images are input into the second neural network to obtain the second feature map information of each training image. Based on the second feature map information and the first feature map information, the extraction module obtains the second target detection information of each training image. Based on the second target detection information and the first target detection information of each training image, the second neural network is trained, and a target detection network is obtained under preset conditions; wherein, obtaining the first target detection information of each training image through an extraction module based on the first feature map information includes: Based on the target initial vector and the first feature map information in the extraction module, the query vector is obtained through the first target cross-attention layer of the extraction module; Based on the query vector and the first feature map information, the first target detection information is obtained through the second target cross-attention layer of the extraction module; The step of obtaining the second target detection information of each training image through the extraction module based on the second feature map information and the first feature map information includes: Based on the target initial vector and the first feature map information in the extraction module, the query vector is obtained through the first target cross-attention layer of the extraction module; Based on the query vector and the second feature map information, the second target detection information is obtained through the second target cross-attention layer of the extraction module.

2. The method according to claim 1, characterized in that, The first target detection information includes first target content information and first target location information, and the second target detection information includes second target content information and second target location information; The extraction module includes a first network branch and a second network branch. The first network branch is used to extract the first target content information or the second target content information. The network structure of the first network branch includes at least a first cross-attention layer and a second cross-attention layer. Furthermore, the second network branch is used to extract the first target location information or the second target location information, and the network structure of the second network branch includes at least a third cross-attention layer and a fourth cross-attention layer.

3. The method according to claim 2, characterized in that, The conditions for satisfying the preset conditions include: The number of training iterations reaches a threshold, or the target loss function satisfies the convergence condition; The target loss function includes at least a weighted sum of a first loss function and a second loss function. The first loss function represents the loss function between the first target content information and the second target content information, and the second loss function represents the loss function between the first target location information and the second target location information.

4. The method according to claim 2, characterized in that, The step of obtaining the first target detection information of each training image based on the first feature map information through the extraction module includes: The first feature map information is input into the first network branch. Based on the first initial vector and the first feature map information in the first network branch, the first query vector is obtained through the first cross-attention layer. Based on the first query vector and the first feature map information, the first target content information is obtained through the second cross-attention layer. The first feature map information is input into the second network branch. Based on the second initial vector in the second network branch and the first feature map information, the second query vector is obtained through the third cross-attention layer. Based on the second query vector and the first feature map information, the first target location information is obtained through the fourth cross-attention layer. The second initial vector is related to the position of the target candidate box in each training image.

5. The method according to claim 2, characterized in that, The step of obtaining the second target detection information of each training image through the extraction module based on the second feature map information and the first feature map information includes: The first feature map information and the second feature map information are input into the first network branch. Based on the first initial vector and the first feature map information in the first network branch, the first query vector is obtained through the first cross-attention layer. Based on the first query vector and the second feature map information, the second target content information is obtained through the second cross-attention layer. The first feature map information and the second feature map information are input into the second network branch. Based on the second initial vector in the second network branch and the first feature map information, the second query vector is obtained through the third cross-attention layer. Based on the second query vector and the second feature map information, the second target location information is obtained through the fourth cross-attention layer. The second initial vector is related to the position of the target candidate box in each training image.

6. The method according to any one of claims 1-5, characterized in that, The training method for the extraction module includes the following steps: Obtain a second training image set, wherein the second training image set includes each second training image; The second training images are input into the first neural network to obtain the third feature map information of each second training image, and the third target detection information of each second training image is obtained through the extraction module based on the third feature map information. Based on the third object detection information, the extraction module is trained. When the number of iterations reaches a threshold or the extraction learning loss function meets the convergence condition, the trained extraction module is obtained. The extraction learning loss function includes the loss between the predicted label and the real label. The predicted label is obtained after object detection based on the third object detection information.

7. The method according to claim 6, characterized in that, The third target detection information includes third target content information and third target location information. Based on the third feature map information, the extraction module obtains the third target detection information for each of the second training images, including: The third feature map information is input into the first network branch of the extraction module to obtain the third target content information of each second training image; The third feature map information is input into the second network branch of the extraction module to obtain the third target location information of each second training image.

8. The method according to claim 7, characterized in that, The predicted tags include predicted target location tags and predicted category tags, and the predicted target location tags and predicted category tags are obtained based on the third target content information or the third target location information. The real tags include real target location tags and real category tags. The extraction learning loss function includes the loss between the predicted target location label and the real target location label, and the sum of the losses between the predicted category label and the real category label.

9. A target detection method, characterized in that, include: Acquire the image to be detected; The target detection network is used to detect targets in the image to be detected and to obtain the target category. The target detection network is obtained based on the target detection network training method according to any one of claims 1 to 8.

10. A target detection network training device, characterized in that, include: The first acquisition module is used to acquire a training image set, wherein the training image set includes each training image; A first processing module is configured to input each training image into a first neural network to obtain first feature map information of each training image, and, based on the first feature map information, obtain first target detection information of each training image through an extraction module, wherein the extraction module is obtained based on training of the first neural network; when obtaining the first target detection information of each training image through the extraction module based on the first feature map information, the first processing module is configured to: obtain a query vector through a first target cross-attention layer of the extraction module based on the target initial vector in the extraction module and the first feature map information; and obtain the first target detection information through a second target cross-attention layer of the extraction module based on the query vector and the first feature map information. The second processing module is configured to input each training image into a second neural network to obtain second feature map information of each training image, and obtain second target detection information of each training image through the extraction module based on the second feature map information and the first feature map information; when obtaining the second target detection information of each training image through the extraction module based on the second feature map information and the first feature map information, the second processing module is configured to: obtain a query vector through the first target cross-attention layer of the extraction module based on the target initial vector in the extraction module and the first feature map information; and obtain the second target detection information through the second target cross-attention layer of the extraction module based on the query vector and the second feature map information. The first training module is used to train the second neural network based on the second target detection information and the first target detection information of each training image, and to obtain the target detection network under preset conditions.

11. A target detection device, characterized in that, include: The second acquisition module is used to acquire the image to be detected; A detection module is used to perform target detection on the image to be detected using a target detection network to obtain the target category, wherein the target detection network is obtained based on the target detection network training method according to any one of claims 1 to 8.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-8 or 9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-8 or 9.

Images