Target detection model training and target detection methods, devices, equipment and media

Abstract

The present disclosure provides a target detection model training and target detection method, device, equipment and medium, which relates to the field of artificial intelligence technology, specifically to the technical fields of computer vision, deep learning, etc. The target detection model includes: a feature extraction network and multiple auxiliary prediction networks, and the training method includes: obtaining training data; using the feature extraction network to obtain image features of image samples in the training data; obtaining auxiliary prediction results based on the image features and each auxiliary prediction network; constructing a sub-loss function based on the auxiliary prediction results, the real results in the training data and the target matching relationship, wherein the target matching relationship is determined based on the initial matching relationship shared by the multiple auxiliary prediction networks; constructing a total loss function based on the sub-loss function, and using the total loss function to adjust the model parameters. The present disclosure can improve the accuracy of target detection.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as computer vision and deep learning, and more particularly to target detection model training and target detection methods, devices, equipment, and media. Background Art

[0002] Object detection is an important research direction in the field of computer vision and is also the basis for other complex visual tasks.

[0003] How to improve target detection accuracy is a problem that needs to be solved. Summary of the Invention

[0004] The present disclosure provides a target detection model training and target detection method, apparatus, device and medium.

[0005] According to one aspect of the present disclosure, a training method for a target detection model is provided, wherein the target detection model includes: a feature extraction network and multiple auxiliary prediction networks, and the method includes: obtaining training data, wherein the training data includes: an image sample and a real result of a target in the image sample; using the feature extraction network to perform feature extraction processing on the image sample to obtain image features of the image sample; based on the image features and each auxiliary prediction network in the multiple auxiliary prediction networks, obtaining an auxiliary prediction result of the target corresponding to each auxiliary prediction network; for each auxiliary prediction network, based on the auxiliary prediction result, the real result and the target matching relationship, constructing a sub-loss function corresponding to each auxiliary prediction network, wherein the target matching relationship is determined based on an initial matching relationship shared by the multiple auxiliary prediction networks; constructing a total loss function based on the sub-loss functions corresponding to each auxiliary prediction network, and using the total loss function to adjust the model parameters of the target detection model.

[0006] According to another aspect of the present disclosure, a target detection method is provided, comprising: acquiring an image; processing the image using a target detection model to obtain a detection result of a target in the image; wherein the target detection model is obtained using any training method described in any one of the above aspects.

[0007] According to another aspect of the present disclosure, a training device for a target detection model is provided, wherein the target detection model includes: a feature extraction network and multiple auxiliary prediction networks, and the device includes: an acquisition module for acquiring training data, wherein the training data includes: image samples and real results of targets in the image samples; a feature extraction module for using the feature extraction network to perform feature extraction processing on the image samples to obtain image features of the image samples; an auxiliary prediction module for obtaining auxiliary prediction results of the targets corresponding to the respective auxiliary prediction networks based on the image features and each auxiliary prediction network in the multiple auxiliary prediction networks; an auxiliary construction module for constructing, for each auxiliary prediction network, a sub-loss function corresponding to the respective auxiliary prediction networks based on the auxiliary prediction results, the real results and the target matching relationship, wherein the target matching relationship is determined based on the initial matching relationship shared by the multiple auxiliary prediction networks; and an adjustment module for constructing a total loss function based on the sub-loss functions corresponding to the respective auxiliary prediction networks, and using the total loss function to adjust the model parameters of the target detection model.

[0008] According to another aspect of the present disclosure, a target detection device is provided, comprising: an acquisition module for acquiring an image; a detection module for processing the image using a target detection model to obtain a detection result of the target in the image; wherein the target detection model is obtained using the training method described in any one of the above aspects.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any one of the methods described in any one of the above aspects.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to any one of the above aspects.

[0011] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of the above aspects.

[0012] According to the technical solution disclosed in the present invention, the target detection accuracy can be improved.

[0013] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0015] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0016] Figure 2 is a schematic diagram of an application scenario provided according to an embodiment of the present disclosure;

[0017] Figure 3 is a schematic diagram of the structure of the target detection model provided according to an embodiment of the present disclosure;

[0018] Figure 4 is a schematic diagram of the structure of multiple prediction networks provided according to an embodiment of the present disclosure;

[0019] Figure 5 1 is a schematic diagram of a process for obtaining an initial matching relationship according to an embodiment of the present disclosure;

[0020] Figure 6 is a schematic diagram according to a second embodiment of the present disclosure;

[0021] Figure 7 is a schematic diagram according to a third embodiment of the present disclosure;

[0022] Figure 8 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0023] Figure 9 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0024] Figure 10 Schematic diagram of an electronic device used to implement the target detection model training method or target detection method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] Figure 1is a schematic diagram according to the first embodiment of the present disclosure. This embodiment provides a method for training a target detection model, wherein the target detection model includes: a feature extraction network and multiple auxiliary prediction networks. The method includes:

[0027] 101. Acquire training data, where the training data includes: image samples and real results of targets in the image samples.

[0028] 102. Use the feature extraction network to perform feature extraction processing on the image sample to obtain image features of the image sample.

[0029] 103. Based on the image feature and each auxiliary prediction network in the multiple auxiliary prediction networks, obtain an auxiliary prediction result of the target corresponding to each auxiliary prediction network.

[0030] 104. For each of the auxiliary prediction networks, construct a sub-loss function corresponding to each of the auxiliary prediction networks based on the auxiliary prediction results, the true results, and the target matching relationship, wherein the target matching relationship is determined based on an initial matching relationship shared by the multiple auxiliary prediction networks.

[0031] 105. Construct a total loss function based on the sub-loss functions corresponding to the respective auxiliary prediction networks, and use the total loss function to adjust the model parameters of the target detection model.

[0032] The image samples may be images in an existing image set, and the image samples may be annotated manually or in other ways to obtain the true results of the targets in the image samples.

[0033] An image sample may include one or more objects. Taking multiple objects as an example, the true result of each object can be represented by Y1, Y2, Y3, etc. The true result of each object can include location information (box) and category information (class).

[0034] The feature extraction network can be a convolutional neural network (CNN), such as the Resnet series network.

[0035] The input of the feature extraction network is an image sample, and the output is the image features of the image sample. The image features can be single-scale image features or multi-scale image features. For example, the multi-scale image features include the three-scale image features output by the last three stages of ResNet (which can be represented by P3, P4, and P5).

[0036] After obtaining the image features, a prediction network can be used to obtain the prediction results of the target in the image. The input of the prediction network is the image features, and the output is the prediction results of the target.

[0037] Prediction networks are typically multiple (or multi-level or multi-layered). For two adjacent prediction networks, the output features of the previous prediction network serve as the input features of the next prediction network. Furthermore, prediction networks can be divided into primary prediction networks and auxiliary prediction networks. For example, if there are N prediction networks, the first (N-1) prediction network can be used as the auxiliary prediction network, and the last (i.e., the Nth) prediction network can be used as the primary prediction network.

[0038] The prediction result of the target obtained based on the auxiliary prediction network can be called an auxiliary prediction result, and the prediction result of the target obtained based on the main prediction network can be called a main prediction result.

[0039] In this embodiment, for a scenario where there are multiple auxiliary prediction networks, a corresponding sub-loss function can be constructed for each auxiliary prediction network based on the corresponding auxiliary prediction results and the actual result of the target.

[0040] After obtaining the sub-loss functions, a total loss function can be constructed based on the sub-loss functions, and the model parameters can be adjusted using the total loss function until the preset conditions are met (such as reaching a preset number of times, the total loss function is less than a preset value, the difference in model parameters between two iterations is less than a preset difference, etc.). Specifically, the back propagation (BP) algorithm can be used to adjust the model parameters.

[0041] For the sub-loss function, taking a certain auxiliary prediction network as an example, the auxiliary prediction results obtained by the auxiliary prediction network can be expressed as X1, X2, X3... When calculating the sub-loss function, it is necessary to know the matching relationship between the auxiliary prediction results and the true results, for example, the above-mentioned X1 matches Y1, X2 matches Y2, etc., so as to construct a sub-loss function based on the mutually matching auxiliary prediction results and true results.

[0042] To distinguish, the matching relationship between the auxiliary prediction results and the true results can be called the target matching relationship, and the matching relationship between the main prediction results and the true results can be called the main matching relationship. For different auxiliary prediction networks, the corresponding target matching relationships may be different.

[0043] Assuming that there are (N-1) auxiliary prediction networks, if each auxiliary prediction network calculates the target matching relationship once, (N-1) matching operations are required.

[0044] Since the target matching relationships calculated by each auxiliary prediction network may be inconsistent, this will cause model instability during the training process, which in turn leads to longer training time; since the target matching relationships of each auxiliary prediction network need to be calculated separately, this also increases the amount of calculation.

[0045] To address the aforementioned instability and high computational complexity issues, in this embodiment, the matching relationship can be calculated only once, and multiple (e.g., N-1) auxiliary prediction networks can all use this calculation result. That is, the matching relationship is calculated once and used (N-1) times.

[0046] The result of this matching operation can be called an initial matching relationship, and specifically, the Hungarian algorithm can be used for the matching operation. The (N-1) auxiliary prediction networks can each use this initial matching relationship to obtain their corresponding target matching relationships, and then construct a sub-loss function based on the target matching relationships.

[0047] In a specific implementation, the training method can be executed by a graphics processing unit (GPU). After the GPU calculates the initial matching relationship, the initial matching relationship can be stored in a hardware memory outside the GPU. When a sub-loss function needs to be constructed (such as through instructions, events, codes, etc. indicating that a sub-loss function needs to be constructed), the GPU can read the initial matching relationship from the hardware memory, and based on the initial matching relationship, determine the target matching relationship corresponding to each auxiliary prediction network in parallel, and based on the auxiliary prediction results, the true results and the target matching relationship corresponding to each auxiliary prediction network, construct the sub-loss functions corresponding to each auxiliary prediction network in parallel.

[0048] In this embodiment, the target detection model includes multiple auxiliary prediction networks, and a total loss function is constructed based on the sub-loss functions corresponding to the multiple auxiliary prediction networks, thereby realizing multi-level optimization and improving the model accuracy. The target matching relationship is obtained based on the initial matching relationship shared by multiple auxiliary prediction networks, and only one matching operation can be performed without the need for each auxiliary prediction network to calculate the matching relationship separately. This can reduce the amount of calculation, avoid instability problems, increase training speed, reduce hardware resource consumption, and improve resource utilization.

[0049] In order to better understand the embodiments of the present disclosure, application scenarios to which the embodiments of the present disclosure are applicable are described below.

[0050] like Figure 2As shown, the user terminal 201 can capture an image and send the image to the server 202. The server 202 recognizes the information of the target in the image through a pre-trained target detection model. The target information includes location information and category information. The user terminal 201 identifies the target in the image based on the target information and displays it to the user. For example, a box is used to mark the location of the target, and the corresponding box identifies the category of the target. The user terminal includes, for example: a personal computer (PC), a laptop, a mobile device (such as a mobile phone), etc. The server can be a local server or a cloud server. The user terminal and the server can communicate through a wired network and / or a wireless network. The above takes the server performing target detection as an example. If the user terminal has target detection capabilities, the target detection model can also be installed locally on the user terminal to perform target detection locally on the user terminal.

[0051] The object detection model consists of a feature extraction network and a prediction network. The feature extraction network is used to extract image features; the prediction network is used to obtain prediction results of objects in the image based on the image features. The prediction results include the location information (box) and category information (class) of the object.

[0052] The feature extraction network is the backbone network of the target detection model, usually a CNN, such as the Resnet series network.

[0053] Prediction networks are typically multiple (or multi-level or multi-layered). For two adjacent prediction networks, the output features of the previous prediction network serve as the input features of the next prediction network. Furthermore, prediction networks can be divided into primary prediction networks and auxiliary prediction networks. For example, if there are N prediction networks, the first (N-1) prediction network can be used as the auxiliary prediction network, and the last (i.e., the Nth) prediction network can be used as the primary prediction network.

[0054] The prediction result of the target obtained based on the auxiliary prediction network can be called an auxiliary prediction result, and the prediction result of the target obtained based on the main prediction network can be called a main prediction result.

[0055] Generally speaking, a loss function is constructed based on the main prediction result, the actual result of the target, and the matching relationship between the two (main matching relationship), and then the model parameters are adjusted based on the loss function.

[0056] However, in order to improve the performance of the model, a multi-level optimization method can be adopted, that is, based on each prediction network (including the main prediction network and the auxiliary prediction network), the corresponding prediction results are obtained, and the sub-loss functions corresponding to each prediction network are constructed based on the predicted results and the actual results and the matching relationship between the two. Then, the total loss function is constructed based on the sub-loss functions corresponding to each prediction network, and the model parameters are adjusted based on the total loss function.

[0057] For the case of multiple auxiliary prediction networks, the matching relationship between the auxiliary prediction results of each auxiliary prediction network and the true result can be called a target matching relationship, and the target matching relationship can be obtained based on the initial matching relationship shared by multiple auxiliary detection networks.

[0058] Wherein, an initial prediction result of the target may be obtained based on the image features; and matching processing is performed on the initial prediction result and the true result to obtain the initial matching relationship.

[0059] In this embodiment, by matching the initial prediction results with the actual results, an accurate initial matching relationship can be obtained, thereby improving the model accuracy.

[0060] Specifically, the target detection model can be a Transformer-based detection (DETR) series model, such as the DETR model, deformable DETR (deformable_DETR) model, denoising DETR (DeNoising, DN_DETR) model, DETR with improved denoising anchor boxes (DINO) model, group DETR (group_DETR) model, etc.

[0061] Take the DINO model as an example, Figure 3 As shown, the target detection model includes: a feature extraction network 301, an encoder (Encoder) 302, a query selection (Query Sellection, QS) module 303 and N (N is a positive integer greater than 2) prediction networks ( Figure 3 denoted by prediction network xN in the figure), each prediction network includes a decoder 304 and a prediction head 305. The network structure of each prediction network can be the same, but the parameters are not shared.

[0062] The feature extraction network 301 is used to extract image features of an image, such as CNN.

[0063] The encoder 302 is used to encode the image features to obtain encoded features. The network structure can be the same as the Transformer encoder.

[0064] The QS module 303 is used to select query features from the coding features. Specifically, assuming that the dimension of the coding features is M, K (K is a preset value) features can be selected from the M features as query features. Specifically, the selection can be made in sequence, such as selecting the first K features from the M features as query features; or, the objectivity score of each feature in the M features can be calculated, and K query features can be selected in descending order of the objectivity score. The specific calculation of the objectivity score can be found in the calculation process in Efficient DETR.

[0065] The decoder 304 is used to obtain decoding features based on the encoding features and the query features. The network structure can be the same as that of the Transformer decoder.

[0066] The prediction head 305 is used to obtain a prediction result of the target in the image based on the decoded features, and the prediction result includes location information (box) and category information (class). The prediction head can be a feedforward network (FFN), which can specifically include a fully connected layer.

[0067] The multiple prediction networks can be divided into a main prediction network and multiple auxiliary prediction networks. Assuming that there are N prediction networks, the 1st to (N-1)th prediction networks can be called auxiliary prediction networks, and the Nth prediction network can be called the main prediction network.

[0068] Among them, the decoder and prediction head included in the main prediction network can be called the main decoder and main prediction head, and the prediction result can be called the main prediction result; the decoder and prediction head included in the auxiliary prediction network can be called the auxiliary decoder and auxiliary prediction head, and the prediction result can be called the auxiliary prediction result.

[0069] like Figure 4 As shown, the auxiliary decoders are represented by auxiliary decoder 1 to auxiliary decoder (N-1), the auxiliary prediction heads are represented by auxiliary prediction head 1 to auxiliary prediction head (N-1), and the auxiliary prediction results are represented by prediction result 1 to prediction result (N-1). The main decoder is represented by main decoder N, the main prediction head is represented by main prediction head N, the main prediction result is represented by prediction result N, and the sub-loss functions obtained by each prediction network are represented by sub-loss function 1 to sub-loss function N.

[0070] In a specific implementation, the output features of the previous decoder can be used as the input features of the next decoder. For example, the input features of auxiliary decoder 2 are the output features of auxiliary decoder 1. Alternatively, in the iterative bounding box optimization scenario, the input information of the next decoder includes: the output features of the previous decoder and the prediction results obtained by the previous decoder, such as Figure 4In the example, the input information of auxiliary decoder 2 includes the output features of auxiliary decoder 1 and prediction result 1.

[0071] In the usual training process, the loss function is constructed based on the main prediction result (prediction result N) and its matching true result.

[0072] In this embodiment, a multi-level optimization training method is used. In this training method, loss functions are also constructed for the auxiliary prediction networks. Accordingly, for each prediction network, a sub-loss function is constructed based on the corresponding prediction result and its matched real result. A total loss function is then constructed based on these sub-loss functions. For example, the total loss function is obtained by summing all the sub-loss functions.

[0073] like Figure 5 As shown, during the training phase, query features obtained by the QS module 501 are input into the feedforward network (FFN) 502. After processing by the FFN, an initial prediction result of the target in the image is output. The matching module 503 is used to match the initial prediction result with the actual result to obtain an initial matching relationship. The network structure of the FFN can be the same as the network structure of the prediction head in the prediction network, but the parameters are not shared. The matching module can be applied during the training phase and is no longer included in the inference phase.

[0074] After obtaining the initial matching relationship, the target matching relationship corresponding to each auxiliary prediction network can be obtained based on the initial matching relationship, which is expressed as target matching relationship 1 to target matching relationship (N-1).

[0075] After obtaining each target matching relationship, the corresponding sub-loss function can be obtained based on the corresponding predicted result, true result and target matching relationship. For example, sub-loss function 1 is constructed based on predicted result 1, true result and target matching relationship 1, and so on, sub-loss function (N-1) is constructed based on predicted result (N-1), true result and target matching relationship (N-1).

[0076] Regarding the initial matching relationship and the target matching relationship, since the bounding box is optimized step by step, the target matching relationship can be obtained based on the initial matching relationship based on the step-by-step optimization relationship. Taking auxiliary decoder 1 as an example, the input information of auxiliary decoder 1 includes: the encoding features output by the encoder, the query features output by the QS module, and the initial prediction results output by the FFN. The prediction result 1 output by auxiliary decoder 1 is optimized based on the initial prediction results. For example, for multiple targets, the initial prediction results include: X1, X2, X3..., and auxiliary decoder 1 optimizes X1, X2, X3..., and the obtained auxiliary prediction results (prediction result 1) include: X1', X2', X3'..., and X1' corresponds to X1, X2' corresponds to X2, and X3' corresponds to X3. Assuming that the initial matching relationship is X1 matches Y1, then since X1' corresponds to X1, it can be determined that X1' matches Y1 in the target matching relationship.

[0077] Multiple auxiliary prediction networks can share the same initial matching relationship. To improve model accuracy, the main prediction network no longer shares the same initial matching relationship. Specifically, for the main prediction network, the main matching relationship between the main prediction result and the true result can be calculated. Based on the main matching relationship, the main prediction result, and the true result, a sub-loss function (sub-loss function N) corresponding to the main prediction network is constructed. Then, the sub-loss function corresponding to the main prediction network and the sub-loss functions corresponding to each auxiliary prediction network are added together to obtain the total loss function. The initial matching relationship and the main matching relationship can both be matched using the Hungarian algorithm.

[0078] In combination with the above application scenarios, the present disclosure also provides a method for training a target detection model.

[0079] Figure 6 2 is a schematic diagram according to the second embodiment of the present disclosure. This embodiment provides a method for training an object detection model. The object detection model includes: a feature extraction network, an encoder, multiple auxiliary prediction networks and a main prediction network. Each auxiliary prediction network includes: an auxiliary decoder and an auxiliary prediction head. The main prediction network includes: a main decoder and a main prediction head. The method includes:

[0080] 601. Acquire training data, where the training data includes: image samples and real results of targets in the image samples.

[0081] 602. Use the feature extraction network to perform feature extraction processing on the image sample to obtain image features of the image sample.

[0082] 603. Use the encoder to encode the image features to obtain encoded features.

[0083] 604. Select and process the encoding feature to obtain a query feature.

[0084] Among them, combined Figure 3 ,QS module can be used to select the encoding features to obtain the query features.

[0085] 605. Obtain an initial prediction result of the target based on the query feature.

[0086] Among them, combined Figure 5 , FFN can be used to process the query features to obtain the initial prediction results.

[0087] In this embodiment, by encoding and selecting image features, more effective query features can be obtained. Initial prediction results are obtained based on the more effective query features, which can improve the accuracy of the initial prediction results and thus improve the accuracy of the model.

[0088] 606. Perform matching processing on the initial prediction result and the actual result to obtain an initial matching relationship.

[0089] Among them, combined Figure 5 , the matching module can be used to process the query features to obtain the initial prediction results.

[0090] 607. Based on the initial matching relationship, determine the target matching relationship corresponding to each auxiliary prediction network.

[0091] As described above, the matching relationships of the respective objects may be determined based on the optimized relationship of the bounding boxes.

[0092] 608. For each auxiliary prediction network, obtain an auxiliary prediction result corresponding to each auxiliary prediction network based on the encoding feature, the query feature, and the initial prediction result.

[0093] Specifically, the auxiliary decoder can be used to obtain the decoding features corresponding to the various auxiliary prediction networks based on the encoding features, the query features and the initial prediction results; and the auxiliary prediction head can be used to perform prediction processing on the decoding features corresponding to the various auxiliary prediction networks to obtain the auxiliary prediction results corresponding to the various auxiliary prediction networks.

[0094] In this embodiment, for each auxiliary prediction network, a corresponding auxiliary prediction result is obtained based on the encoding feature, the query feature and the initial prediction result, and the prediction result can be optimized step by step to improve the model performance.

[0095] 609. For each auxiliary prediction network, construct a sub-loss function corresponding to each auxiliary prediction network based on the auxiliary prediction result, the true result, and the target matching relationship corresponding to each auxiliary prediction network.

[0096] 610. For the main prediction network, obtain a main prediction result corresponding to the main prediction network based on the encoding feature, the query feature, and the initial prediction result.

[0097] Specifically, the main decoder can be used to obtain the decoding features corresponding to the main prediction network based on the encoding features, the query features and the initial prediction results; and the main prediction head can be used to perform prediction processing on the decoding features corresponding to the main prediction network to obtain the main prediction results corresponding to the main prediction network.

[0098] In this embodiment, the corresponding main prediction result is obtained based on the encoding feature, the query feature and the initial prediction result, and the prediction result can be optimized step by step to improve the model performance.

[0099] 611. For the main prediction network, perform matching processing on the main prediction result and the true result to obtain a main matching relationship between the main prediction result and the true result.

[0100] In this embodiment, for the main prediction network, the main prediction results and the actual results are matched, and the matching relationship can be calculated separately for the main prediction network to improve the model accuracy.

[0101] 612. For the main prediction network, construct a sub-loss function corresponding to the main prediction network based on the main prediction result, the true result and the main matching relationship.

[0102] Among them, there is no time sequence restriction relationship between 606-609 and 610-612.

[0103] 613. Construct a total loss function based on the sub-loss functions corresponding to the auxiliary prediction networks and the sub-loss function corresponding to the main prediction network.

[0104] In this embodiment, a total loss function is constructed based on the sub-loss functions corresponding to each auxiliary prediction network and the sub-loss function corresponding to the main prediction network. The information of each prediction network can be referred to to improve the accuracy of the total loss function, thereby improving the accuracy of the model.

[0105] 614. Use the total loss function to adjust model parameters of the target detection model.

[0106] Among them, the adjusted model parameters specifically include: model parameters of the feature extraction network, model parameters of the encoder, model parameters of each auxiliary decoder, model parameters of each auxiliary prediction head, model parameters of the main decoder and model parameters of the main prediction head. In addition, the model parameters of the FFN used to obtain the initial prediction results can also be adjusted.

[0107] In this embodiment, a total loss function is constructed based on the sub-loss functions corresponding to the main prediction network and the sub-loss functions corresponding to each auxiliary prediction network. This allows for multi-level optimization training and improves model accuracy. By sharing the initial matching relationship among multiple auxiliary prediction networks, a single matching calculation can be used multiple times, stabilizing the training process, reducing the amount of computation required, and increasing training speed while maintaining accuracy.

[0108] Figure 7 is a schematic diagram according to a third embodiment of the present disclosure. This embodiment provides a target detection method, the method comprising:

[0109] 701. Acquire an image.

[0110] 702. Use a target detection model to process the image to obtain a detection result of the target in the image.

[0111] The target detection model is obtained by using any of the above training methods.

[0112] Specifically, combined Figures 3 to 5 The network structure shown can use the feature extraction network in the target detection model to perform feature extraction processing on the image to obtain image features; use the encoder in the target detection model to encode the image features to obtain encoding features; use the QS module in the target detection model to select query features from the encoding features; use the FFN in the target detection model to predict the query features to obtain an initial prediction result of the target in the image; use the prediction network in the target detection model to obtain the detection result of the target in the image based on the encoding features, the query features and the initial prediction result, and the detection result includes the location information and category information of the target.

[0113] Among them, the prediction network has multiple levels. For the prediction network of a certain level, its input information includes: coding features, query features and output information of the previous level. The output information of the previous level can specifically be the output features of the decoder of the previous level, or the output features of the decoder of the previous level and the prediction results obtained by the previous level. The output information of the previous level corresponding to the prediction network of the first level (the network where the auxiliary decoder 1 is located) includes the initial prediction results obtained by FFN. The prediction results obtained by the prediction network of the last level (the network where the main decoder N is located) are used as the detection results.

[0114] In this embodiment, since an accurate target detection model can be obtained, the target detection accuracy can be improved when the target detection model is used for target detection.

[0115] Figure 8This is a schematic diagram according to the fourth embodiment of the present disclosure. This embodiment provides a training device for a target detection model, wherein the target detection model includes: a feature extraction network and multiple auxiliary prediction networks, and the device includes: an acquisition module 801, a feature extraction module 802, an auxiliary prediction module 803, an auxiliary construction module 804, and an adjustment module 805.

[0116] The acquisition module 801 is used to acquire training data, and the training data includes: image samples and real results of targets in the image samples; the feature extraction module 802 is used to use the feature extraction network to perform feature extraction processing on the image samples to obtain image features of the image samples; the auxiliary prediction module 803 is used to obtain the auxiliary prediction results of the targets corresponding to the auxiliary prediction networks based on the image features and each auxiliary prediction network in the multiple auxiliary prediction networks; the auxiliary construction module 804 is used to construct the sub-loss functions corresponding to the auxiliary prediction networks based on the auxiliary prediction results, the real results and the target matching relationship, wherein the target matching relationship is determined based on the initial matching relationship shared by the multiple auxiliary prediction networks; the adjustment module 805 is used to construct a total loss function based on the sub-loss functions corresponding to the auxiliary prediction networks, and use the total loss function to adjust the model parameters of the target detection model.

[0117] In this embodiment, the target detection model includes multiple auxiliary prediction networks, and a total loss function is constructed based on the sub-loss functions corresponding to the multiple auxiliary prediction networks, thereby realizing multi-level optimization and improving the model accuracy. The target matching relationship is obtained based on the initial matching relationship shared by multiple auxiliary prediction networks, and only one matching operation can be performed without the need for each auxiliary prediction network to calculate the matching relationship separately. This can reduce the amount of calculation, avoid instability problems, increase training speed, reduce hardware resource consumption, and improve resource utilization.

[0118] In some embodiments, the device 800 further includes: a determination module for obtaining an initial prediction result of the target based on the image features; and a matching module for matching the initial prediction result and the true result to obtain the initial matching relationship.

[0119] In this embodiment, by matching the initial prediction results with the actual results, an accurate initial matching relationship can be obtained, thereby improving the model accuracy.

[0120] In some embodiments, the object detection model further includes: an encoder;

[0121] The determination module is further configured to: use the encoder to perform encoding processing on the image features to obtain encoding features; perform selection processing on the encoding features to obtain query features; and obtain an initial prediction result of the target based on the query features.

[0122] In this embodiment, by encoding and selecting image features, more effective query features can be obtained. Initial prediction results are obtained based on the more effective query features, which can improve the accuracy of the initial prediction results and thus improve the accuracy of the model.

[0123] In some embodiments, each auxiliary prediction network includes: an auxiliary decoder and an auxiliary prediction head; the auxiliary prediction module 803 is further used to: for each auxiliary prediction network, use the auxiliary decoder to obtain the decoding features corresponding to each auxiliary prediction network based on the encoding features, the query features and the initial prediction results; for each auxiliary prediction network, use the auxiliary prediction head to perform prediction processing on the decoding features corresponding to each auxiliary prediction network to obtain the auxiliary prediction results corresponding to each auxiliary prediction network.

[0124] In this embodiment, for each auxiliary prediction network, a corresponding auxiliary prediction result is obtained based on the encoding feature, the query feature and the initial prediction result, and the prediction result can be optimized step by step to improve the model performance.

[0125] In some embodiments, the target detection module also includes: a main prediction network; the device also includes: a main prediction module, used to adopt the main prediction network, based on the encoding features, the query features and the initial prediction results, to obtain the main prediction results corresponding to the main prediction network; a main construction module, used to construct the sub-loss function corresponding to the main prediction network based on the main prediction results; accordingly, the adjustment module is further used to: construct the total loss function based on the sub-loss functions corresponding to the each auxiliary prediction network and the sub-loss function corresponding to the main prediction network.

[0126] In this embodiment, a total loss function is constructed based on the sub-loss functions corresponding to each auxiliary prediction network and the sub-loss function corresponding to the main prediction network. The information of each prediction network can be referred to to improve the accuracy of the total loss function, thereby improving the accuracy of the model.

[0127] In some embodiments, the main prediction network includes: a main decoder and a main prediction head; the main prediction module is further used to: use the main decoder to obtain the decoding features corresponding to the main prediction network based on the encoding features, the query features and the initial prediction results; use the main prediction head to perform prediction processing on the decoding features corresponding to the main prediction network to obtain the main prediction results corresponding to the main prediction network.

[0128] In this embodiment, the corresponding main prediction result is obtained based on the encoding feature, the query feature and the initial prediction result, and the prediction result can be optimized step by step to improve the model performance.

[0129] In some embodiments, the main construction module is further used to: perform matching processing on the main prediction result and the true result to obtain a main matching relationship between the main prediction result and the true result; and construct a sub-loss function corresponding to the main prediction network based on the main prediction result, the true result and the main matching relationship.

[0130] In this embodiment, for the main prediction network, the main prediction results and the true results are matched, and the matching relationship can be calculated separately for the main prediction network to improve the model accuracy.

[0131] Figure 9 is a schematic diagram according to the fifth embodiment of the present disclosure. This embodiment provides a target detection device, and the device 900 includes: an acquisition module 901 and a detection module 902.

[0132] The acquisition module 901 is used to acquire an image; the detection module 902 is used to use a target detection model to process the image to obtain a detection result of the target in the image; wherein the target detection model is obtained using any of the above-mentioned training methods.

[0133] In this embodiment, since an accurate target detection model can be obtained, the target detection accuracy can be improved when the target detection model is used for target detection.

[0134] It can be understood that in the embodiments of the present disclosure, the same or similar contents in different embodiments can be referenced to each other.

[0135] It can be understood that the terms “first”, “second”, etc. in the embodiments of the present disclosure are only used for distinction and do not indicate the degree of importance, time sequence, etc.

[0136] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0137] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0138] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. Electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 1000 can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0139] like Figure 10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the electronic device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0140] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0141] The computing unit 1001 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the training method of the target detection model or the target detection method. For example, in some embodiments, the training method of the target detection model or the target detection method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the training method of the target detection model or the target detection method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute a target detection model training method or a target detection method in any other appropriate manner (e.g., by means of firmware).

[0142] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable load balancing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0147] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within a cloud computing service ecosystem that addresses the management difficulties and limited business scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0148] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0149] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for training a target detection model, the target detection model comprising: Feature extraction network and a plurality of auxiliary prediction networks, the method comprising: Acquire training data, the training data including: image samples and real results of targets in the image samples; Using the feature extraction network, performing feature extraction processing on the image sample to obtain image features of the image sample; Based on the image features and each auxiliary prediction network in the plurality of auxiliary prediction networks, obtaining an auxiliary prediction result of the target corresponding to each auxiliary prediction network; For each of the auxiliary prediction networks, constructing a sub-loss function corresponding to each of the auxiliary prediction networks based on the auxiliary prediction results, the true results, and a target matching relationship, wherein the target matching relationship is determined based on an initial matching relationship shared by the multiple auxiliary prediction networks; Constructing a total loss function based on the sub-loss functions corresponding to the respective auxiliary prediction networks, and using the total loss function to adjust model parameters of the target detection model; Also includes: Based on the image features, obtaining an initial prediction result of the target; Performing matching processing on the initial prediction result and the actual result to obtain the initial matching relationship; The target detection model further includes: an encoder; The obtaining of an initial prediction result of the target based on the image feature includes: Using the encoder, encoding the image features to obtain encoding features; Selecting and processing the coding features to obtain query features; Based on the query features, obtaining an initial prediction result of the target; Each auxiliary prediction network includes: an auxiliary decoder and an auxiliary prediction head; The obtaining, based on the image features and each auxiliary prediction network in the plurality of auxiliary prediction networks, an auxiliary prediction result of the target corresponding to each auxiliary prediction network includes: For each auxiliary prediction network, using the auxiliary decoder, based on the encoding feature, the query feature, and the initial prediction result, obtain decoding features corresponding to each auxiliary prediction network; For each auxiliary prediction network, the auxiliary prediction head is used to perform prediction processing on the decoding features corresponding to each auxiliary prediction network to obtain the auxiliary prediction results corresponding to each auxiliary prediction network.

2. The method according to claim 1, wherein The target detection module also includes: a main prediction network; The method further comprises: Using the main prediction network, based on the encoding feature, the query feature and the initial prediction result, obtain a main prediction result corresponding to the main prediction network; Based on the main prediction result, construct a sub-loss function corresponding to the main prediction network; Accordingly, the total loss function is constructed based on the sub-loss functions corresponding to the respective auxiliary prediction networks, including: The total loss function is constructed based on the sub-loss functions corresponding to the auxiliary prediction networks and the sub-loss function corresponding to the main prediction network.

3. The method according to claim 2, wherein: The main prediction network includes: a main decoder and a main prediction head; The adopting the main prediction network to obtain a main prediction result corresponding to the main prediction network based on the encoding feature, the query feature, and the initial prediction result includes: Using the primary decoder, based on the encoding feature, the query feature, and the initial prediction result, obtain a decoding feature corresponding to the primary prediction network; The main prediction head is used to perform prediction processing on the decoding features corresponding to the main prediction network to obtain a main prediction result corresponding to the main prediction network.

4. The method according to claim 2, wherein: The constructing of a sub-loss function corresponding to the main prediction network based on the main prediction result includes: Performing matching processing on the main prediction result and the true result to obtain a main matching relationship between the main prediction result and the true result; Based on the main prediction result, the true result and the main matching relationship, a sub-loss function corresponding to the main prediction network is constructed.

5. A target detection method, comprising: Acquire images; Using an object detection model to process the image to obtain a detection result of the object in the image; The target detection model is obtained by using the training method described in any one of claims 1 to 4.

6. A training device for a target detection model, the target detection model comprising: Feature extraction network and a plurality of auxiliary prediction networks, the device comprising: An acquisition module is used to acquire training data, wherein the training data includes: image samples and real results of targets in the image samples; A feature extraction module, configured to perform feature extraction processing on the image sample using the feature extraction network to obtain image features of the image sample; an auxiliary prediction module, configured to obtain, based on the image features and each of the plurality of auxiliary prediction networks, an auxiliary prediction result of the target corresponding to each of the auxiliary prediction networks; an auxiliary construction module, configured to construct, for each of the auxiliary prediction networks, a sub-loss function corresponding to the auxiliary prediction network based on the auxiliary prediction result, the true result, and a target matching relationship, wherein the target matching relationship is determined based on an initial matching relationship shared by the multiple auxiliary prediction networks; An adjustment module, configured to construct a total loss function based on the sub-loss functions corresponding to the respective auxiliary prediction networks, and to adjust the model parameters of the target detection model using the total loss function; Also includes: A determination module, configured to obtain an initial prediction result of the target based on the image features; A matching module, configured to perform matching processing on the initial prediction result and the actual result to obtain the initial matching relationship; The target detection model further includes: an encoder; The determining module is further configured to: Using the encoder, encoding the image features to obtain encoding features; Selecting and processing the coding features to obtain query features; Based on the query features, obtaining an initial prediction result of the target; Each auxiliary prediction network includes: an auxiliary decoder and an auxiliary prediction head; The auxiliary prediction module is further configured to: For each auxiliary prediction network, using the auxiliary decoder, based on the encoding feature, the query feature, and the initial prediction result, obtain decoding features corresponding to each auxiliary prediction network; For each auxiliary prediction network, the auxiliary prediction head is used to perform prediction processing on the decoding features corresponding to each auxiliary prediction network to obtain the auxiliary prediction results corresponding to each auxiliary prediction network.

7. The device according to claim 6, wherein The target detection module also includes: a main prediction network; The device further comprises: A main prediction module, configured to use the main prediction network to obtain a main prediction result corresponding to the main prediction network based on the encoding feature, the query feature and the initial prediction result; A main construction module, configured to construct a sub-loss function corresponding to the main prediction network based on the main prediction result; The adjustment module is further configured to: The total loss function is constructed based on the sub-loss functions corresponding to the auxiliary prediction networks and the sub-loss function corresponding to the main prediction network.

8. The device according to claim 7, wherein The main prediction network includes: a main decoder and a main prediction head; The main prediction module is further configured to: Using the primary decoder, based on the encoding feature, the query feature, and the initial prediction result, obtain a decoding feature corresponding to the primary prediction network; The main prediction head is used to perform prediction processing on the decoding features corresponding to the main prediction network to obtain a main prediction result corresponding to the main prediction network.

9. The device according to claim 7, wherein The main building block is further configured to: Performing matching processing on the main prediction result and the true result to obtain a main matching relationship between the main prediction result and the true result; Based on the main prediction result, the true result and the main matching relationship, a sub-loss function corresponding to the main prediction network is constructed.

10. A target detection device comprising: An acquisition module, used for acquiring images; A detection module, configured to process the image using a target detection model to obtain a detection result of the target in the image; The target detection model is obtained by using the training method described in any one of claims 1 to 4.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

13. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images