Model training method and device, target detection method and device and electronic equipment

Abstract

The embodiment of the invention provides a model training method and device, a target detection method and device and electronic equipment, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a sample image and a label; for each sample image, utilizing an encoder in the target detection model to obtain a first sample feature vector; the model further comprises a decoder; the decoder comprises at least one partial cross attention module; for each part of the cross attention module, splitting input data in a channel dimension, and processing a first input part and a second input part by utilizing a self-attention layer and a cross attention layer of the module; processing results of the self-attention layer and the cross attention layer are combined to obtain output data of the module; obtaining a detection result by using output data of the last module; and calculating a loss value based on the detection result and the label, and adjusting model parameters to obtain a trained target detection model. Therefore, the time delay of target detection can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a model training method, a target detection method, a device and an electronic device. Background Art

[0002] In the field of computer vision technology, object detection models can be used to detect objects in images. In some real-time object detection scenarios, it is necessary to obtain image detection results with low latency. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to provide a model training method, a target detection method, a device and an electronic device to reduce the delay of target detection and improve the efficiency of target detection. The specific technical solution is as follows:

[0004] In a first aspect of an embodiment of the present application, a model training method is first provided, the method comprising:

[0005] Obtain sample images and labels of each sample image; wherein the label of a sample image is used to represent each object in the sample image;

[0006] For each sample image, extract features of the sample image using an encoder in the target detection model in the initial state to obtain a first sample feature vector; wherein the target detection model in the initial state further includes a decoder; the decoder includes at least one partial cross attention module, and a partial cross attention module includes a self-attention layer and a cross attention layer connected in parallel;

[0007] For each cross-attention module, the input data of the cross-attention module is split in the channel dimension to obtain a first input part and a second input part; wherein the input data of the first cross-attention module is the first sample feature vector;

[0008] Using the self-attention layer and the cross-attention layer of the cross-attention module, the first input part and the second input part are processed respectively;

[0009] Combining the processing results of the self-attention layer and the cross-attention layer of this part of the cross-attention module, the output data of this part of the cross-attention module is obtained;

[0010] Use the output data of the last partial cross attention module to perform detection and obtain the detection results of each object in the sample image;

[0011] Based on the difference between the detection result of each object in the sample image and the label of the sample image, a loss value is calculated, and based on the obtained loss value, the model parameters of the initial state of the target detection model are adjusted until convergence is reached to obtain a trained target detection model.

[0012] In some embodiments, the decoder comprises: a plurality of serially connected partial cross-attention modules.

[0013] In some embodiments, a partial crisscross attention module further comprises: a residual gate network;

[0014] The processing results of the self-attention layer and the cross-attention layer of the part of the cross-attention module are combined to obtain the output data of the part of the cross-attention module, including:

[0015] Combining the processing results of the self-attention layer and the cross-attention layer of this part of the cross-attention module, a first sample fusion result is obtained;

[0016] The input data of this part of the cross-attention module and the fusion result of the first sample are input into the residual gate network to obtain the output data of this part of the cross-attention module.

[0017] In some embodiments, a partial crisscross attention module further comprises a feed-forward neural network; a residual gate network comprises a multi-layer perceptron and activation layer;

[0018] The step of inputting the input data of the part of the cross attention module and the first sample fusion result into the residual gate network to obtain the output data of the part of the cross attention module includes:

[0019] Splicing the input data of the part of the cross attention module and the first sample fusion result to obtain a first sample splicing result;

[0020] Inputting the first sample splicing result into the multi-layer perceptron, and inputting the output result of the multi-layer perceptron into the activation layer, to obtain a weight vector of each feature vector in the first sample splicing result;

[0021] Calculate the dot product results of each feature vector and the corresponding weight vector in the input data of this part of the cross attention module respectively, and obtain the first sample dot product result;

[0022] Respectively calculating the dot product results of each feature vector in the first sample fusion result and the corresponding weight vector to obtain a second sample dot product result;

[0023] Calculate a dot-add result of the dot product result of the first sample and a dot product result of the second sample;

[0024] The calculated point addition result is input into the feedforward neural network to obtain the output data of this part of the cross attention module.

[0025] In some embodiments, a partial cross attention module further includes: a first normalization layer and a second normalization layer;

[0026] The processing results of the self-attention layer and the cross-attention layer of the cross-attention module are combined to obtain a first sample fusion result, including:

[0027] Inputting the processing result of the self-attention layer of the part of the cross attention module and the point addition result of the input data of the cross attention layer of the part of the cross attention module into the first normalization layer to obtain a first sample normalization result;

[0028] Inputting the processing result of the cross attention layer of the part of the cross attention module and the point addition result of the input data of the self-attention layer of the part of the cross attention module into the second normalization layer to obtain a second sample normalization result;

[0029] splicing the first sample normalization result and the second sample normalization result to obtain a first sample fusion result;

[0030] and / or,

[0031] A partial cross attention module also includes: The third normalization layer:

[0032] The step of inputting the calculated point addition result into the feedforward neural network to obtain the output data of the cross attention module includes:

[0033] Inputting the calculated point addition result into the feedforward neural network to obtain the output result of the feedforward neural network;

[0034] The output result of the feedforward neural network is input into the third normalization layer to obtain the output data of this part of the cross attention module.

[0035] In some embodiments, the label of a sample image includes: the real box and the real category of each object in the sample image; the detection result of each object in the sample image includes: the predicted box of the object and the confidence indicating the probability that the object belongs to the corresponding real category;

[0036] The calculating the loss value based on the difference between the detection result of each object in the sample image and the label of the sample image includes:

[0037] The loss value is calculated based on the difference between the detection result of each object in the sample image and the label of the sample image using a preset loss function; wherein the preset loss function is:

[0038] ;

[0039] represents the loss value; Represents the intersection-over-union ratio between the predicted box and the true box of an object in the sample image; Indicates the confidence of the object; is a hyperparameter; Represents a logarithmic function.

[0040] In some embodiments, the first input part and the second input part correspond to the same number of channels.

[0041] A second aspect of the embodiments of the present application provides a target detection method, the method comprising:

[0042] Acquire the image to be detected;

[0043] Using an encoder in a trained target detection model to extract features from the image to be detected, obtaining a first predicted feature vector; wherein the trained target detection model further comprises a decoder; the decoder comprises at least one partial cross attention module, and a partial cross attention module comprises a self-attention layer and a cross attention layer connected in parallel;

[0044] For each cross-attention module, the input data of the cross-attention module is split in the channel dimension to obtain a third input part and a fourth input part; wherein the input data of the first cross-attention module is the first prediction feature vector;

[0045] Using the self-attention layer and the cross-attention layer of the cross-attention module, the third input part and the fourth input part are processed respectively;

[0046] Combining the processing results of the self-attention layer and the cross-attention layer of this part of the cross-attention module, the output data of this part of the cross-attention module is obtained;

[0047] The output data of the last partial cross-attention module is used for detection to obtain the detection result of each object in the image to be detected.

[0048] In some embodiments, the decoder comprises: a plurality of serially connected partial cross-attention modules.

[0049] In some embodiments, a partial crisscross attention module further comprises: a residual gate network;

[0050] The processing results of the self-attention layer and the cross-attention layer of the part of the cross-attention module are combined to obtain the output data of the part of the cross-attention module, including:

[0051] Combining the processing results of the self-attention layer and the cross-attention layer of this part of the cross-attention module, a first prediction fusion result is obtained;

[0052] The input data of this part of the cross-attention module and the first prediction fusion result are input into the residual gate network to obtain the output data of this part of the cross-attention module.

[0053] In some embodiments, a partial crisscross attention module further comprises a feed-forward neural network; a residual gate network comprises a multi-layer perceptron and activation layer;

[0054] The step of inputting the input data of the part of the cross attention module and the first prediction fusion result into the residual gate network to obtain the output data of the part of the cross attention module includes:

[0055] Splicing the input data of the part of the cross attention module and the first prediction fusion result to obtain a first prediction splicing result;

[0056] Inputting the first prediction splicing result into the multi-layer perceptron, and inputting the output result of the multi-layer perceptron into the activation layer, to obtain a weight vector of each feature vector in the first prediction splicing result;

[0057] Calculate the dot product results of each feature vector and the corresponding weight vector in the input data of the cross attention module respectively to obtain a first predicted dot product result;

[0058] Respectively calculating the dot product results of each feature vector in the first prediction fusion result and the corresponding weight vector to obtain a second prediction multiplication result;

[0059] Calculate a dot addition result of the first predicted dot product result and the second predicted dot product result;

[0060] The calculated point addition result is input into the feedforward neural network to obtain the output data of this part of the cross attention module.

[0061] In some embodiments, a partial cross attention module further includes: a first normalization layer and a second normalization layer;

[0062] The processing results of the self-attention layer and the cross-attention layer of the cross-attention module are combined to obtain a first prediction fusion result, including:

[0063] Inputting the processing result of the self-attention layer of the part of the cross attention module and the point addition result of the input data of the cross attention layer of the part of the cross attention module into the first normalization layer to obtain a first predicted normalized result;

[0064] Inputting the processing result of the cross attention layer of the part of the cross attention module and the point addition result of the input data of the self-attention layer of the part of the cross attention module into the second normalization layer to obtain a second predicted normalization result;

[0065] splicing the first prediction normalization result and the second prediction normalization result to obtain a first prediction fusion result;

[0066] and / or,

[0067] A partial cross attention module also includes: The third normalization layer:

[0068] The step of inputting the calculated point addition result into the feedforward neural network to obtain the output data of the cross attention module includes:

[0069] Inputting the calculated point addition result into the feedforward neural network to obtain the output result of the feedforward neural network;

[0070] The output result of the feedforward neural network is input into the third normalization layer to obtain the output data of this part of the cross attention module.

[0071] In some embodiments, the third input part and the fourth input part correspond to the same number of channels.

[0072] In some embodiments, the detection result of each object in the image to be detected includes: a prediction box of the object and a confidence level indicating the probability of the category to which the object belongs;

[0073] The target detection model is obtained by training based on sample images and labels of each sample image using a preset loss function; wherein the label of a sample image includes: the real frame and real category of each object in the sample image;

[0074] The default loss function is:

[0075] ;

[0076] represents the loss value; Represents the intersection-over-union ratio between the predicted box and the true box of an object in a sample image; Indicates the confidence of the object; is a hyperparameter; Represents a logarithmic function.

[0077] In a third aspect of an embodiment of the present application, a model training device is provided, the device comprising:

[0078] A sample acquisition unit, used to acquire sample images and labels of each sample image; wherein the label of a sample image is used to represent each object in the sample image;

[0079] A first feature extraction unit is used to extract features of each sample image using an encoder in the target detection model in the initial state to obtain a first sample feature vector; wherein the target detection model in the initial state further includes a decoder; the decoder includes at least one partial cross attention module, and a partial cross attention module includes a self-attention layer and a cross attention layer connected in parallel;

[0080] A first splitting unit is used for splitting the input data of each partial cross attention module in the channel dimension to obtain a first input part and a second input part; wherein the input data of the first partial cross attention module is the first sample feature vector;

[0081] A first processing unit, configured to process the first input part and the second input part respectively using a self-attention layer and a cross-attention layer of the cross-attention module;

[0082] A first fusion unit is used to combine the processing results of the self-attention layer and the cross-attention layer of the part of the cross-attention module to obtain the output data of the part of the cross-attention module;

[0083] A first detection unit, used for performing detection using the output data of the last partial cross attention module to obtain a detection result of each object in the sample image;

[0084] The loss adjustment unit is used to calculate the loss value based on the difference between the detection result of each object in the sample image and the label of the sample image, and adjust the model parameters of the target detection model in the initial state based on the obtained loss value until convergence is reached to obtain a trained target detection model.

[0085] In some embodiments, the decoder comprises: a plurality of serially connected partial cross-attention modules.

[0086] In some embodiments, a partial crisscross attention module further comprises: a residual gate network;

[0087] The first fusion unit comprises:

[0088] A first fusion subunit is used to combine the processing results of the self-attention layer and the cross-attention layer of the part of the cross-attention module to obtain a first sample fusion result;

[0089] The first residual processing subunit is used to input the input data of this part of the cross-attention module and the first sample fusion result into the residual gate network to obtain the output data of this part of the cross-attention module.

[0090] In some embodiments, a partial crisscross attention module further comprises a feed-forward neural network; a residual gate network comprises a multi-layer perceptron and activation layer;

[0091] The first residual processing subunit is specifically configured to:

[0092] Splicing the input data of the part of the cross attention module and the first sample fusion result to obtain a first sample splicing result;

[0093] Inputting the first sample splicing result into the multi-layer perceptron, and inputting the output result of the multi-layer perceptron into the activation layer, to obtain a weight vector of each feature vector in the first sample splicing result;

[0094] Calculate the dot product results of each feature vector and the corresponding weight vector in the input data of this part of the cross attention module respectively, and obtain the first sample dot product result;

[0095] Respectively calculating the dot product results of each feature vector in the first sample fusion result and the corresponding weight vector to obtain a second sample dot product result;

[0096] Calculate a dot-add result of the dot product result of the first sample and a dot product result of the second sample;

[0097] The calculated point addition result is input into the feedforward neural network to obtain the output data of this part of the cross attention module.

[0098] In some embodiments, a partial cross attention module further includes: a first normalization layer and a second normalization layer;

[0099] The first fusion subunit is specifically used for:

[0100] Inputting the processing result of the self-attention layer of the part of the cross attention module and the point addition result of the input data of the cross attention layer of the part of the cross attention module into the first normalization layer to obtain a first sample normalization result;

[0101] Inputting the processing result of the cross attention layer of the part of the cross attention module and the point addition result of the input data of the self-attention layer of the part of the cross attention module into the second normalization layer to obtain a second sample normalization result;

[0102] splicing the first sample normalization result and the second sample normalization result to obtain a first sample fusion result;

[0103] and / or,

[0104] A partial cross attention module also includes: The third normalization layer:

[0105] The first residual processing subunit is specifically configured to:

[0106] Inputting the calculated point addition result into the feedforward neural network to obtain the output result of the feedforward neural network;

[0107] The output result of the feedforward neural network is input into the third normalization layer to obtain the output data of this part of the cross attention module.

[0108] In some embodiments, the label of a sample image includes: the real box and the real category of each object in the sample image; the detection result of each object in the sample image includes: the predicted box of the object and the confidence indicating the probability that the object belongs to the corresponding real category;

[0109] The loss adjustment unit is specifically used for:

[0110] The loss value is calculated based on the difference between the detection result of each object in the sample image and the label of the sample image using a preset loss function; wherein the preset loss function is:

[0111] ;

[0112] represents the loss value; Represents the intersection-over-union ratio between the predicted box and the true box of an object in the sample image; Indicates the confidence of the object; is a hyperparameter; Represents a logarithmic function.

[0113] In some embodiments, the first input part and the second input part correspond to the same number of channels.

[0114] According to a fourth aspect of the embodiments of the present application, a target detection device is provided, the device comprising:

[0115] An image acquisition unit to be detected, used to acquire the image to be detected;

[0116] A second feature extraction unit is used to extract features of the image to be detected by using an encoder in a trained target detection model to obtain a first predicted feature vector; wherein the trained target detection model further includes a decoder; the decoder includes at least one partial cross attention module, and a partial cross attention module includes a self-attention layer and a cross attention layer connected in parallel;

[0117] A second splitting unit is used for splitting the input data of each partial cross attention module in the channel dimension to obtain a third input part and a fourth input part; wherein the input data of the first partial cross attention module is the first prediction feature vector;

[0118] A second processing unit is used to process the third input part and the fourth input part respectively by using the self-attention layer and the cross-attention layer of the part of the cross-attention module;

[0119] A second fusion unit is used to combine the processing results of the self-attention layer and the cross-attention layer of the part of the cross-attention module to obtain the output data of the part of the cross-attention module;

[0120] The second detection unit is used to use the output data of the last partial cross-attention module to perform detection to obtain the detection result of each object in the image to be detected.

[0121] In some embodiments, the decoder comprises: a plurality of serially connected partial cross-attention modules.

[0122] In some embodiments, a partial crisscross attention module further comprises: a residual gate network;

[0123] The second fusion unit comprises:

[0124] The second fusion subunit is used to combine the processing results of the self-attention layer and the cross-attention layer of the part of the cross-attention module to obtain a first prediction fusion result;

[0125] The second residual processing sub-unit is used to input the input data of this part of the cross-attention module and the first prediction fusion result into the residual gate network to obtain the output data of this part of the cross-attention module.

[0126] In some embodiments, a partial crisscross attention module further comprises a feed-forward neural network; a residual gate network comprises a multi-layer perceptron and activation layer;

[0127] The second residual processing subunit is specifically used for:

[0128] Splicing the input data of the part of the cross attention module and the first prediction fusion result to obtain a first prediction splicing result;

[0129] Inputting the first prediction splicing result into the multi-layer perceptron, and inputting the output result of the multi-layer perceptron into the activation layer, to obtain a weight vector of each feature vector in the first prediction splicing result;

[0130] Calculate the dot product results of each feature vector and the corresponding weight vector in the input data of the cross attention module respectively to obtain a first predicted dot product result;

[0131] Respectively calculating the dot product results of each feature vector in the first prediction fusion result and the corresponding weight vector to obtain a second prediction multiplication result;

[0132] Calculate a dot addition result of the first predicted dot product result and the second predicted dot product result;

[0133] The calculated point addition result is input into the feedforward neural network to obtain the output data of this part of the cross attention module.

[0134] In some embodiments, a partial cross attention module further includes: a first normalization layer and a second normalization layer;

[0135] The second fusion subunit is specifically used for:

[0136] Inputting the processing result of the self-attention layer of the part of the cross attention module and the point addition result of the input data of the cross attention layer of the part of the cross attention module into the first normalization layer to obtain a first predicted normalized result;

[0137] Inputting the processing result of the cross attention layer of the part of the cross attention module and the point addition result of the input data of the self-attention layer of the part of the cross attention module into the second normalization layer to obtain a second predicted normalization result;

[0138] splicing the first prediction normalization result and the second prediction normalization result to obtain a first prediction fusion result;

[0139] and / or,

[0140] A partial cross attention module also includes: The third normalization layer:

[0141] The second residual processing subunit is specifically used for:

[0142] Inputting the calculated point addition result into the feedforward neural network to obtain the output result of the feedforward neural network;

[0143] The output result of the feedforward neural network is input into the third normalization layer to obtain the output data of this part of the cross attention module.

[0144] In some embodiments, the third input part and the fourth input part correspond to the same number of channels.

[0145] In some embodiments, the detection result of each object in the image to be detected includes: a prediction box of the object and a confidence level indicating the probability of the category to which the object belongs;

[0146] The target detection model is obtained by training based on sample images and labels of each sample image using a preset loss function; wherein the label of a sample image includes: the real frame and real category of each object in the sample image;

[0147] The default loss function is:

[0148] ;

[0149] represents the loss value; Represents the intersection-over-union ratio between the predicted box and the true box of an object in a sample image; Indicates the confidence of the object; is a hyperparameter; Represents a logarithmic function.

[0150] According to a fifth aspect of the embodiments of the present application, an electronic device is provided, including:

[0151] Memory, used to store computer programs;

[0152] The processor is used to implement any of the above-mentioned model training methods or target detection methods when executing the program stored in the memory.

[0153] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned model training methods or target detection methods is implemented.

[0154] In another aspect of the embodiments of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned model training methods or target detection methods.

[0155] Beneficial effects of the embodiments of the present invention:

[0156] The model training method provided by the embodiment of the present invention, in the process of performing target detection on an image (i.e., a sample image or an image to be detected) through a target detection model, after obtaining the feature representation of the image extracted by the encoder in the target detection model (i.e., the first sample feature vector or the first predicted feature vector), the output data of the encoder can be input into the decoder in the target detection model for processing. In the present application, the decoder includes at least one partial cross-attention module, and a partial cross-attention module includes a self-attention layer and a cross-attention layer connected in parallel. Correspondingly, for each partial cross-attention module, the input data of the partial cross-attention module can be split in the channel dimension, and the self-attention layer and the cross-attention layer of the partial cross-attention module are used to process the two parts obtained by splitting (i.e., the first input part and the second input part, or, the third input part or the fourth input part). In other words, the partial cross-attention module in the decoder can process the output data of the encoder using the parallel structure of the self-attention layer and the cross-attention layer, and can also obtain the feature representation for the object in the image based on the self-attention mechanism and the cross-attention mechanism (i.e., the output data of the last partial cross-attention module in the decoder). Furthermore, the output data of the last partial cross-attention module can be used for detection to obtain the detection result of each object in the image. In this way, the parallel structure of the self-attention layer and the cross-attention layer in some cross-attention modules can reduce the latency of target detection and improve the efficiency of target detection.

[0157] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0158] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0159] Figure 1 A first flow chart of the model training method provided in the embodiment of the present application;

[0160] Figure 2 A data processing flow chart of the first partial cross-attention module provided in an embodiment of the present application;

[0161] Figure 3 A second flow chart of the model training method provided in the embodiment of the present application;

[0162] Figure 4A data processing flow chart of the second partial cross-attention module provided in an embodiment of the present application;

[0163] Figure 5 A data processing flow chart of the first residual gate network provided in an embodiment of the present application;

[0164] Figure 6 A first flow chart of the target detection method provided in the embodiment of the present application;

[0165] Figure 7 A structural diagram of a model training device provided in an embodiment of the present application;

[0166] Figure 8 A structural diagram of a target detection device provided in an embodiment of the present application;

[0167] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0168] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field based on this application belong to the scope of protection of the present invention.

[0169] In the field of computer vision technology, object detection models can be used to detect objects in images. In some real-time object detection scenarios, it is necessary to obtain image detection results with low latency.

[0170] The embodiment of the present application provides a model training method, which can be applied to electronic devices. For example, the electronic device is: an intelligent device (such as a server) for training and running a target detection model. Accordingly, the electronic device can run the target detection model to perform target detection on an image through its own processing unit. For example, the processing unit can be an NPU (Neural Network Processing Unit), a GPU (Graphic Processing Unit), etc.

[0171] See also Figure 1 , Figure 1 The first flow chart of the model training method provided in the embodiment of the present application comprises the following steps:

[0172] S101: Obtain sample images and labels of each sample image.

[0173] The label of a sample image is used to represent each object in the sample image.

[0174] S102: For each sample image, use the encoder in the target detection model in the initial state to extract features of the sample image to obtain a first sample feature vector.

[0175] Among them, the target detection model in the initial state also includes a decoder; the decoder includes at least one partial cross-attention module, and a partial cross-attention module includes a self-attention layer and a cross-attention layer connected in parallel.

[0176] S103: For each cross-attention module, the input data of the cross-attention module is split in the channel dimension to obtain a first input part and a second input part.

[0177] Among them, the input data of the first partial cross-attention module is the first sample feature vector.

[0178] S104: Utilize the self-attention layer and the cross-attention layer of this part of the cross-attention module to process the first input part and the second input part respectively.

[0179] S105: Combine the processing results of the self-attention layer and the cross-attention layer of this part of the cross-attention module to obtain the output data of this part of the cross-attention module.

[0180] S106: Use the output data of the last partial cross-attention module to perform detection to obtain the detection result of each object in the sample image.

[0181] S107: Based on the difference between the detection result of each object in the sample image and the label of the sample image, a loss value is calculated, and based on the obtained loss value, the model parameters of the initial state target detection model are adjusted until convergence is reached to obtain a trained target detection model.

[0182] Based on the above processing, in the process of performing target detection on an image (i.e., a sample image or an image to be detected), after obtaining the feature representation of the image extracted by the encoder in the target detection model (i.e., the first sample feature vector or the first predicted feature vector), the output data of the encoder can be input into the decoder in the target detection model for processing. In the present application, the decoder includes at least one partial cross-attention module, and a partial cross-attention module includes a self-attention layer and a cross-attention layer connected in parallel. Correspondingly, for each partial cross-attention module, the input data of the partial cross-attention module can be split in the channel dimension, and the self-attention layer and the cross-attention layer of the partial cross-attention module are used to process the two parts (i.e., the first input part and the second input part, or, the third input part or the fourth input part) obtained by splitting. In other words, the partial cross-attention module in the decoder can process the output data of the encoder using the parallel structure of the self-attention layer and the cross-attention layer, and can also obtain the feature representation for the object in the image based on the self-attention mechanism and the cross-attention mechanism (i.e., the output data of the last partial cross-attention module in the decoder). Furthermore, the output data of the last partial cross-attention module can be used for detection to obtain the detection result of each object in the image. In this way, the parallel structure of the self-attention layer and the cross-attention layer in some cross-attention modules can reduce the latency of target detection and improve the efficiency of target detection.

[0183] With respect to step S101 , during the training process of the target detection model, the electronic device may obtain sample images for training the target detection model in an initial state, and labels of each sample image.

[0184] The sample images may be pre-labeled images in a public image set, for example, the public image set may be a COCO (Common Objects in Context) image dataset.

[0185] In one implementation, a technician may select an image containing an object of a specified category as a sample image according to actual needs. For example, the category of the object may be: vehicle, human, etc.

[0186] The label of a sample image is used to represent each object in the sample image.

[0187] In one implementation, the label of a sample image includes: a real box and a real category of each object in the sample image.

[0188] That is to say, in the label of a sample image, for each object in the sample image, there are recorded: the true category of the object, and the true box of the object (that is, the position of the true box in the image).

[0189] With respect to step S102 , for each sample image, the electronic device may use an encoder (Encoder) in the target detection model in the initial state to perform feature extraction on the sample image, and the encoder may extract a feature representation of the obtained image, namely, a first sample feature vector.

[0190] In one implementation, the target detection model further includes: a feature extraction network. For example, the feature extraction network may be a convolutional neural network (CNN).

[0191] Correspondingly, the electronic device can input the sample image into the feature extraction network of the target detection model to obtain the image features of the image. Furthermore, the image features of the image are input into the encoder in the target detection model, and the encoder can encode the image features of the input image to obtain the first sample feature vector.

[0192] Accordingly, the obtained first sample feature vector can be described from multiple parameter dimensions. For example, the multiple parameter dimensions may be: channel number, length, batch size, etc.

[0193] The number of channels corresponding to the first sample feature vector (which can be denoted as C) is consistent with the number of channels of the output layer of the encoder. For example, if the number of channels of the output layer of the encoder is 64, then the number of channels corresponding to the first sample feature vector is 64.

[0194] The length of the first sample feature vector (which can be denoted as L) is determined based on the size of the sample image and the downsampling factor corresponding to the first sample feature vector. For example, when the downsampling factor is 1 and the size of the sample image is 20×20, the length of the first sample feature vector is: .

[0195] The batch size corresponding to the first sample feature vector represents the number of sample images that the target detection model can process simultaneously during one training process. The batch size is preset by the technician according to actual needs. For example, the batch size corresponding to the first sample feature vector can be 1.

[0196] The target detection model also includes a decoder. The decoder includes at least one partial cross-attention (PCA) module, and a PCA module includes a self-attention (SA) layer and a cross-attention (CA) layer connected in parallel.

[0197] In one implementation, the decoder includes only one partial cross-attention module. That is, the partial cross-attention module is both the first partial cross-attention module in the decoder and the last partial cross-attention module in the decoder. Accordingly, the input data of the partial cross-attention module is the output data of the encoder (i.e., the first sample feature vector), and the output data of the partial cross-attention module is the output data of the decoder.

[0198] In one implementation, the decoder includes: a plurality of partially cross-attention modules connected in series. That is, for each PCA module except the first PCA module, the input data of the PCA module is the output data of the previous PCA module. The input data of the first PCA module is the output data of the encoder (i.e., the first sample feature vector), and the output data of the last PCA module is the output data of the decoder.

[0199] In this way, when the decoder includes multiple PCA modules connected in series, the decoder can perform decoding in a multi-round iterative manner. Each PCA module can further refine the prediction of the object in the image based on the output of the previous PCA. Through multiple rounds of iterations, the object detection model can gradually optimize the object positioning and classification results, and improve the accuracy of detection.

[0200] In one implementation, the dimension of the input data of each PCA module is consistent with the dimension of the output data of the PCA module. That is, in the process of using the PCA module to process its own input data, the dimension of the feature vector contained in the input data will not be changed. That is, the parameter values ​​of the input data of a PCA module in each parameter dimension are consistent with the parameter values ​​of the output data of the PCA module in each parameter dimension. Among them, the specific process of using the PCA module to process its own input data will be described in the subsequent embodiments.

[0201] Accordingly, the dimension of the input data of the decoder is consistent with the dimension of the output data of the decoder. That is, in the process of processing the first sample feature vector by the decoder, the dimension of the feature vector is not changed. That is, the parameter values ​​of the first sample feature vector in each parameter dimension are consistent with the parameter values ​​of the output data of the decoder in each parameter dimension. The specific process of processing the first sample feature vector by the decoder will be described in the subsequent embodiments.

[0202] Steps S103 to S105 are the process of processing the input data of a PCA module to obtain the output data of the PCA module. When the decoder includes multiple PCA modules connected in series, the process of processing the input data of each PCA module is the same.

[0203] With respect to step S103 , for each PCA module, after obtaining the input data of the PCA module, the PCA module may split its own input data in the channel dimension to obtain a first input part and a second input part.

[0204] In some embodiments, the first input portion and the second input portion correspond to the same number of channels.

[0205] In an embodiment of the present application, a PCA module may split the input data of the PCA module into two parts in half in the channel dimension, that is, the number of channels corresponding to the first input part and the second input part is the same.

[0206] For example, for the first PCA module in the decoder, the number of channels corresponding to the input data of the PCA module (that is, the first sample feature vector) is 64, then the number of channels corresponding to the first input part obtained by splitting is 32, and the number of channels corresponding to the second input part is also 32.

[0207] In one implementation, for the input data of a partial cross attention module, the feature vectors corresponding to the first preset number of channels obtained by splitting can be used as the first input part; and the feature vectors corresponding to the second preset number of channels obtained by splitting can be used as the second input part. The preset number is: half of the number of channels corresponding to the input data of the PCA module.

[0208] Alternatively, for the input data of a partial cross-attention module, the feature vectors corresponding to the first preset number of channels obtained by splitting can be used as the second input part; and the feature vectors corresponding to the second preset number of channels obtained by splitting can be used as the first input part. In this application, the specific method of splitting the input data of each partial cross-attention module is not limited here.

[0209] It is understandable that, when the decoder includes multiple PCA modules connected in series, in order to ensure the stability of parameter adjustment of the target detection model during training, the splitting method used in the process of splitting the input data of each PCA module is consistent.

[0210] Based on the above processing, the input data of a partial cross-attention module can be split into two parts in half in the channel dimension. In this way, when the self-attention layer and the cross-attention layer of the partial cross-attention module are used to process the first input part and the second input part respectively, the data processing of the self-attention layer and the cross-attention layer can be balanced, avoiding the situation where a single module takes a long time to process, which can further reduce the delay of target detection and improve the efficiency of target detection.

[0211] For step S104, after obtaining the first input part and the second input part corresponding to the PCA module, the self-attention layer (which can be recorded as SA layer) and the cross-attention layer (which can be recorded as CA layer) of the PCA module can be used to process the first input part and the second input part respectively.

[0212] Correspondingly, for each partial cross-attention module, the first input part is input into the self-attention layer of the partial cross-attention module to obtain the output data of the self-attention layer, and the second input part is input into the cross-attention layer of the partial cross-attention module to obtain the output data of the cross-attention layer.

[0213] It is understandable that, in the case where the decoder includes multiple PCA modules connected in series, in order to ensure the stability of parameter adjustment of the target detection model during the training process, in the process of using the SA layer and CA layer of each PCA module to process the first input part and the second input part respectively, the channels corresponding to each feature vector in the data processed by each SA layer are consistent, and accordingly, the channels corresponding to each feature vector in the data processed by each CA layer are also consistent. For example, when the number of channels corresponding to the input data of each PCA module is 64, the SA layer of each PCA module can process the feature vectors corresponding to the first 32 channels in the input data of the PCA module, and the CA layer of each PCA module can process the feature vectors corresponding to the last 32 channels in the input data of the PCA module.

[0214] Among them, the dimension of the input data (i.e., the first input part) of the SA layer of each PCA module is consistent with the dimension of the output data of the SA layer of the PCA module. That is to say, in the process of using the SA layer to process its own input data, the dimension of the eigenvector contained in the input data will not be changed. That is, the parameter values ​​of the input data of the SA layer of a PCA module in each parameter dimension are consistent with the parameter values ​​of the output data of the SA layer of the PCA module in each parameter dimension. Among them, the specific process of using the SA layer of the PCA module to process its own input data will be explained in subsequent embodiments.

[0215] Correspondingly, the dimension of the input data (i.e., the second input part) of the CA layer of each PCA module is consistent with the dimension of the output data of the CA layer of the PCA module. That is to say, in the process of using the CA layer to process its own input data, the dimension of the feature vector contained in the input data will not be changed. That is, the parameter values ​​of the input data of the CA layer of a PCA module in each parameter dimension are consistent with the parameter values ​​of the output data of the CA layer of the PCA module in each parameter dimension. Among them, the specific process of using the CA layer of the PCA module to process its own input data will be described in the subsequent embodiments.

[0216] With respect to step S105, after obtaining the processing result of the SA layer in the PCA module and the processing result of the CA layer in the PCA module, the processing result of the SA layer and the processing result of the CA layer can be combined to obtain the output data of the PCA module.

[0217] In one implementation, a partial cross-attention module also includes: a first normalization layer, a second normalization layer, a feedforward neural network, and a third normalization layer.

[0218] Accordingly, step S105 includes:

[0219] Step 1: Input the processing results of the self-attention layer of this part of the cross-attention module and the point addition results of the input data of the cross-attention layer of this part of the cross-attention module into the first normalization layer to obtain the first sample normalization result.

[0220] Step 2: Input the processing results of the cross-attention layer of this part of the cross-attention module and the point addition results of the input data of the self-attention layer of this part of the cross-attention module into the second normalization layer to obtain a second sample normalization result.

[0221] Step 3: Concatenate the normalized result of the first sample and the normalized result of the second sample to obtain the first sample fusion result.

[0222] Step 4: Input the first sample fusion result into the feedforward neural network to obtain the output result of the feedforward neural network.

[0223] Step 5: Input the output result of the feedforward neural network into the third normalization layer to obtain the output data of this part of the cross-attention module.

[0224] In the embodiment of the present application, the first normalization layer, the second normalization layer, and the third normalization layer in a PCA module are normalization layers (Norm Layer) at three different positions in the PCA model.

[0225] After obtaining the output data of the SA layer of the PCA module (i.e., the processing result of the SA layer), the processing result of the SA layer and the input data of the CA layer of the PCA module can be processed by point addition, and the obtained point addition result can be input to the first normalization layer (Norm Layer). Correspondingly, the first normalization layer can process the processing result of the SA layer and the point addition result of the input data of the CA layer to obtain the output data of the first normalization layer (i.e., the first sample normalization result).

[0226] Furthermore, after obtaining the output data of the CA layer of the PCA module (i.e., the processing result of the CA layer), the processing result of the CA layer and the input data of the SA layer of the PCA module can be processed by point addition, and the obtained point addition result can be input to the second normalization layer. Correspondingly, the second normalization layer can process the processing result of the CA layer and the point addition result of the input data of the SA layer to obtain the output data of the second normalization layer (i.e., the second sample normalization result).

[0227] The number of channels corresponding to the first sample normalization result is consistent with the output data of the SA layer of the PCA module, and is consistent with the input data of the SA layer of the PCA module (i.e., the first input part). In addition, the number of channels corresponding to the second sample normalization result is consistent with the output data of the CA layer of the PCA module, and is consistent with the input data of the CA layer of the PCA module (i.e., the second input part).

[0228] Correspondingly, the PCA module can concatenate the first sample normalization result and the second sample normalization result in the channel dimension to obtain the first sample fusion result. That is, the number of channels corresponding to the first sample fusion result of the PCA module is consistent with the number of channels corresponding to the input data of the PCA module (that is, the sum of the number of channels corresponding to the first input part and the number of channels corresponding to the second input part).

[0229] The parameter values ​​of the input data of a PCA module in each parameter dimension are consistent with the parameter values ​​of the first sample fusion result of the PCA module in each parameter dimension.

[0230] Furthermore, the first sample fusion result of the PCA module can be input into the feed-forward neural network (FFN) in the PCA module. And the output result of the feed-forward neural network is input into the third normalization layer in the PCA module to obtain the output data of the third normalization layer, that is, the output data of the PCA module.

[0231] See also Figure 2 , Figure 2 A data processing flow chart of the first partial cross-attention module provided in an embodiment of the present application. Figure 2 In , Input represents: the input data of a partial cross-attention module. Accordingly, the input data of the partial cross-attention module can be divided (Split) to obtain Part1 (Part 1) and Part2 (Part 2), where Part1 represents the first input part corresponding to the partial cross-attention module; Part2 represents the second input part corresponding to the partial cross-attention module.

[0232] Furthermore, Part1 can be input into the self-attention layer (ie, SA) of the cross-attention module of this part, and Part2 can be input into the cross-attention layer (ie, CA) of the cross-attention module of this part. Furthermore, the processing result of the self-attention layer and the input data of the cross-attention layer of the cross-attention module of this part (ie, Part2) are point-added, and the obtained point-added result is input into the normalization layer (Norm) (ie, the first normalization layer) to obtain the first sample normalization result. The processing result of the cross-attention layer of the cross-attention module of this part and the input data of the self-attention layer of the cross-attention module of this part (ie, Part1) are point-added, and the obtained point-added result is input into the normalization layer (Norm) (ie, the second normalization layer) to obtain the second sample normalization result. Furthermore, the first sample normalization result and the second sample normalization result are concatenated (Concat) to obtain the first sample fusion result. The first sample fusion result is input into the feedforward neural network (ie, FFN) to obtain the output result of the feedforward neural network. The output result of the feedforward neural network is input into the normalization layer (Norm) (i.e., the third normalization layer) to obtain the output data (Output) of this part of the cross-attention module.

[0233] If the self-attention layer and the cross-attention layer of an attention module are connected in series in the decoder, that is, when the attention module processes the input data, it needs to first use the self-attention layer to process the input data of the attention module, and after obtaining the output data of the self-attention layer, it uses the cross-attention layer to process the output data of the self-attention layer to obtain the output data of the attention module.

[0234] Under the above-mentioned serial connection structure, since the self-attention layer and the cross-attention layer are sequentially connected, in the process of the above-mentioned attention module processing the input data, the cross-attention layer needs to wait for the output data of the self-attention layer to be processed. In this way, the processing speed of the attention module is slow, and then, the processing speed of the target detection model is slow. Moreover, since the self-attention layer and the cross-attention layer do not change the dimension of the feature vector corresponding to the data in the process of processing their respective input data. Therefore, in the process of the above-mentioned attention module processing the input data, the self-attention layer and the cross-attention layer need to process the data of the same dimension once respectively, which consumes a large amount of calculation and the obtained feature data is redundant.

[0235] In the present application, in a partial cross-attention module, the self-attention layer and the cross-attention layer are connected in parallel. In the process of a partial cross-attention module processing input data, the self-attention layer and the cross-attention layer do not need to wait for the output data of other network layers, therefore, the processing efficiency of the self-attention layer and the cross-attention layer can be improved. Moreover, since the self-attention layer and the cross-attention layer each process a part of the input data of the PCA module, rather than each processing all the input data of the PCA module, the amount of calculation required in the calculation process can be further reduced. Relative to the above-mentioned serially connected structure, the amount of calculation can be reduced by about 30%.

[0236] In other words, the above parallel structure can improve the inference speed of the target detection model, and is particularly suitable for high real-time target detection scenarios.

[0237] For step S106, the output data of the last partial cross attention module in the decoder, that is, the output data of the decoder in the object detection model. Accordingly, the electronic device can use the output data of the decoder to perform detection and obtain the detection result of each object in the sample image.

[0238] In one implementation, the target detection model may further include: a detection head, wherein the detection head includes: a regression head and a classification head.

[0239] Correspondingly, for each object in the sample image, the regression head can predict the position of the prediction box of the object; the classification head can predict the probability that the object belongs to the corresponding true category (that is, the confidence of the object).

[0240] That is to say, the output data of the decoder can be input into the detection head in the target detection model. For each object in the sample image, the detection head can output the detection result of the object, that is, the predicted box of the object and the confidence level representing the probability that the object belongs to the corresponding true category.

[0241] With respect to step S107 , for each object in the sample image, a loss value may be calculated based on the real box and real category of the object in the label of the sample image and the predicted detection result of the object.

[0242] In one implementation, a FL (Focal Loss) loss function may be used to calculate a loss value based on the difference between a detection result of each object in the sample image and a label of the sample image.

[0243] Among them, the FL loss function is expressed as:

[0244] ;

[0245] In the embodiments of the present application, Represents the loss value calculated using the FL loss function, (Balance factor) is a hyperparameter of the target detection model, which is used to balance the importance of positive and negative samples. (Adjustment factor) is a hyperparameter of the target detection model, which is used to balance the attention of difficult and easy samples. For a predicted object, Represents: the confidence of the object (i.e., the probability that the object belongs to the corresponding true category). Represents a positive sample, that is, the sample image contains an object of the true category. Represents a negative sample, that is, the sample image does not contain an object of the true category.

[0246] in, The value of is usually between [0,5]. For example, It can be 2.

[0247] In this way, when calculating the loss value based on the FL loss function, based on the binary classification method, it is strictly defined whether the object in the predicted prediction box corresponds to the object of the real type, that is, when the object in the predicted box is an object of the real type, ; When the object in the prediction box is not an object of the real type, In this way, the accuracy of the target detection model can be improved, positive detection can be increased, and false detection can be suppressed. However, since the obtained prediction box may be offset, that is, part of the area of ​​the prediction box is the object in the image, and the other part is other image content (such as other types of objects, or background, etc.), the recall rate of the trained target detection model may be low based on the binary classification method.

[0248] In one implementation, a VFL (Varifocal Loss) loss function may be used to calculate a loss value based on the difference between a detection result of each object in the sample image and a label of the sample image.

[0249] Among them, the VFL loss function is expressed as:

[0250] ;

[0251] In the embodiments of the present application, Indicates the use of The loss value calculated by the loss function, (Balance factor) is a hyperparameter of the target detection model, which is used to balance the importance of positive and negative samples. (Adjustment factor) is a hyperparameter of the target detection model, which is used to balance the attention of difficult and easy samples. For a predicted object, Represents: the confidence of the object (i.e., the probability that the object belongs to the corresponding true category), Represents the intersection-over-union ratio between the predicted box and the true box of the object.

[0252] In this way, the VFL loss function can introduce the intersection-and-union ratio between the predicted box and the true box for an object on the basis of the FL loss function, replacing the above two-classification method to calculate the loss value. At this time, for an object, the loss function can combine the intersection-and-union ratio between the predicted box and the true box of the object, as well as the confidence of the object, to jointly calculate the loss value. And because the intersection-and-union ratio between the predicted box and the true box of the object can represent a continuous value between [0,1], the intersection-and-union ratio between the predicted box and the true box of the object can more accurately represent the positional relationship between the predicted box and the true box of the object, thereby improving the recall rate of the target detection model. However, in the process of calculating the loss value based on the above VFL loss function, the accuracy requirement for the predicted box is relatively low. Accordingly, compared with the method of calculating the loss value using the above FL loss function, the accuracy of the target detection model prediction is lower.

[0253] In some embodiments, the step of calculating the loss value based on the difference between the detection result of each object in the sample image and the label of the sample image includes:

[0254] The loss value is calculated based on the difference between the detection result of each object in the sample image and the label of the sample image using a preset loss function; wherein the preset loss function is:

[0255] ;

[0256] represents the loss value; Represents the intersection-over-union ratio between the predicted box and the true box of an object in the sample image; Indicates the confidence of the object; is a hyperparameter; Represents a logarithmic function.

[0257] In the embodiments of the present application, (Adjustment factor) is a hyperparameter of the target detection model, which is used to balance the attention paid to difficult and easy samples. The preset loss function can be called: RFL loss function (Robust Focal Loss).

[0258] Correspondingly, for an object in the predicted sample image, the sample image is a positive sample (the predicted object exists in the sample image), that is, When , calculate the loss value. For an object in the predicted sample image, the sample image is a negative sample (the predicted object does not exist in the sample image), that is, When , and the loss value is calculated. Among them, Represents the loss part when a sample image of an object belongs to a negative sample.

[0259] exist When calculating the loss value based on the above VFL loss function, The corresponding weights are expressed as: In the process of calculating the loss value based on the RFL loss function, The corresponding weights are expressed as: .

[0260] It is understandable that in hour, The larger the value is (i.e., the larger the intersection-over-union ratio between the predicted box and the true box of the object), the closer the predicted box of the object is to the true box of the object. Correspondingly, the possibility that the sample image is a negative sample for the object is smaller. The corresponding weight should also be smaller. In other words, The larger (and hour), Relative to The smaller it is, the RFL loss function can be further reduced relative to the VFL loss function. The corresponding weight.

[0261] exist hour, The smaller it is (i.e., the smaller the intersection-over-union ratio between the predicted box and the true box of the object), the greater the difference between the predicted box and the true box of the object. Correspondingly, the greater the possibility that the sample image is a negative sample for the object. The corresponding weight should also be larger. In other words, The smaller (and hour), Relative to The larger the value, the RFL loss function can be further improved relative to the VFL loss function. In this way, the penalty of false detection by the target detection model can be improved during the training process.

[0262] exist When , in the above VFL loss function, The corresponding weights are expressed as: In the RFL loss function, The corresponding weights are expressed as: That is to say, compared with the VFL loss function, the RFL loss function can further reduce the parameters required to calculate the loss value and reduce the amount of calculation for calculating the loss value.

[0263] Based on the above processing, the preset loss function (i.e., RFL loss function) can combine the intersection-and-union ratio between the predicted box and the true box of an object, as well as the confidence of the object, to jointly calculate the loss value. And in the process of calculating the loss value, the weights corresponding to some losses in the loss function can be adjusted according to actual needs based on the difference between the predicted box and the true box of the object. In this way, the target detection model trained based on the above RFL loss function can improve the accuracy of target detection and reduce false positives while ensuring the recall rate.

[0264] Furthermore, after obtaining the loss value, the electronic device can adjust the model parameters of the target detection model in the initial state based on the obtained loss value until convergence is reached to obtain a trained target detection model.

[0265] In some embodiments, a partial crisscross attention module further comprises: a residual gate network. Figure 3 , Figure 3 The second flow chart of the model training method provided in the embodiment of the present application. Figure 1On the basis of, step S105 comprises:

[0266] S1051: Combine the processing results of the self-attention layer and the cross-attention layer of this part of the cross-attention module to obtain the first sample fusion result.

[0267] S1052: Input the input data of this part of the cross-attention module and the fusion result of the first sample into the residual gate network to obtain the output data of this part of the cross-attention module.

[0268] In an embodiment of the present application, for each cross-attention module (i.e., PCA module) in the decoder, after obtaining the processing results of the self-attention (SA) layer and the cross-attention (CA) layer of the PCA module, the processing results of the SA layer and the CA layer can be fused to obtain a first sample fusion result. The specific process of obtaining the first sample fusion result can refer to the relevant descriptions of steps 1 to 3 above, which will not be repeated here.

[0269] Furthermore, after obtaining the first sample fusion result of the PCA module, the input data of the PCA module and the first sample fusion result may be input into the residual-gating layer (RGL) of the PCA module for fusion to obtain the output data of the PCA.

[0270] For example, the residual gate network can perform point addition processing on the input data of the part of the cross attention module and the first sample fusion result to obtain the output data of the residual gate network. Subsequently, the electronic device can obtain the output data of the part of the cross attention module according to the output data of the residual gate network.

[0271] See also Figure 4 , Figure 4 A data processing flow chart of the second partial cross-attention module provided in an embodiment of the present application.

[0272] Figure 4 In the example, a PCA module includes a residual gate network (RGL). Input represents the input data of a partial cross-attention module. Accordingly, the input data of the partial cross-attention module can be divided (Split) to obtain Part1 and Part2, where Part1 represents the first input part corresponding to the partial cross-attention module; Part2 represents the second input part corresponding to the partial cross-attention module.

[0273] Furthermore, Part1 can be input into the self-attention layer (ie, SA) of the cross-attention module of this part, and Part2 can be input into the cross-attention layer (ie, CA) of the cross-attention module of this part. Furthermore, the processing result of the self-attention layer and the input data of the cross-attention layer of the cross-attention module of this part (ie, Part2) are point-added, and the obtained point-added result is input into the normalization layer (Norm) (ie, the first normalization layer) to obtain the first sample normalization result. The processing result of the cross-attention layer of this part of the cross-attention module and the input data of the self-attention layer of the cross-attention module of this part (ie, Part1) are point-added, and the obtained point-added result is input into the normalization layer (Norm) (ie, the second normalization layer) to obtain the second sample normalization result. Furthermore, the first sample normalization result and the second sample normalization result are concatenated (Concat) to obtain the first sample fusion result.

[0274] Then, the input data of the cross attention module and the first sample fusion result can be input into the residual gate network (RGL) to obtain the output data of the residual gate network. Furthermore, the output data of the residual gate network is input into the feedforward neural network (FFN) to obtain the output result of the feedforward neural network. The output result of the feedforward neural network is input into the normalization layer (Norm) (i.e., the third normalization layer) to obtain the output data (Output) of the cross attention module.

[0275] Based on the above processing, for a residual gate network of a PCA module, the residual gate network can obtain the input data of the PCA module and the first sample fusion result obtained by the attention mechanism for fusion. In other words, the residual gate network has the ability to fuse the original input and the two sets of features processed by the attention mechanism. Correspondingly, the output data of the residual gate network can represent the image features in combination with the above two sets of features. In this way, the representation ability of feature data can be improved and the accuracy of the target detection model can be improved.

[0276] In some embodiments, a residual gate network includes a multi-layer perceptron and activation layers.

[0277] Step S1052 includes:

[0278] Step 1: Splice the input data of this part of the cross-attention module and the first sample fusion result to obtain the first sample splicing result.

[0279] Step 2: Input the first sample splicing result into a multi-layer perceptron, and input the output result of the multi-layer perceptron into the activation layer to obtain a weight vector of each feature vector in the first sample splicing result.

[0280] Step 3: Calculate the dot product of each feature vector and the corresponding weight vector in the input data of this part of the cross-attention module to obtain the first sample dot product result.

[0281] Step 4: Calculate the dot product of each feature vector in the first sample fusion result and the corresponding weight vector to obtain the second sample dot product result.

[0282] Step 5: Calculate the dot addition result of the dot product result of the first sample and the dot product result of the second sample.

[0283] Step 6: Input the calculated point addition result into the feedforward neural network to obtain the output data of this part of the cross-attention module.

[0284] In an embodiment of the present application, the input data of the cross-attention module and the first sample fusion result can be spliced ​​in the channel dimension to obtain the first sample splicing result. In other words, the number of channels corresponding to the first sample splicing result of a PCA module is: the sum of the number of channels corresponding to the input data of the PCA module and the number of channels corresponding to the first sample fusion result of the PCA module.

[0285] For example, the input data of a PCA module is represented as a 64×400 feature vector, where 64 is the number of channels corresponding to the input data of the PCA module; and 400 is the length of each feature vector in the input data of the PCA module. And since the parameter values ​​of the input data of a PCA module in each parameter dimension are consistent with the parameter values ​​of the first sample fusion result of the PCA module in each parameter dimension, that is, the first sample fusion result of the PCA module is also represented as a 64×400 feature vector, where 64 is the number of channels corresponding to the first sample fusion result of the PCA module; and 400 is the length of each feature vector in the first sample fusion result of the PCA module. Correspondingly, the input data of this part of the cross attention module and the first sample fusion result are spliced ​​in the channel dimension, and the first sample splicing result is represented as a 128×400 feature vector, where 128 is the number of channels corresponding to the first sample splicing result of the PCA module; and 400 is the length of each feature vector in the first sample splicing result of the PCA module.

[0286] Then, the first sample concatenation result is input into a multilayer perceptron (MLP) to obtain output data of the multilayer perceptron, wherein the parameter values ​​of the output data of the multilayer perceptron in each parameter dimension are consistent with the parameter values ​​of the input data of the multilayer perceptron in each parameter dimension.

[0287] Continuing with the above example, after obtaining the first sample concatenation result of the PCA module, the first sample concatenation result can be input into the MLP in the RGL of the PCA module, and the output data of the MLP can also be expressed as: a 128×400 feature vector.

[0288] Accordingly, the output result of the multilayer perceptron is input to the activation layer, and the activation layer can obtain the weight vector corresponding to each feature vector in the first sample splicing result based on the preset activation function (i.e., sigmoid function). The length of the weight vector is consistent with the length of each feature vector in the first sample splicing result, and the value range of each position in a weight vector is [0,1].

[0289] That is, for each feature vector in the first sample splicing result, the value at each position in the weight vector corresponding to the feature vector represents: the weight corresponding to the value at the same position of the feature vector.

[0290] Continuing with the above example, when the output data of the MLP is represented as a 128×400 feature vector, the weight vector output by the activation layer is also represented as a 128×400 feature vector, where 128 is the number of weight vectors output by the activation layer, and 400 is the length of the weight vector output by the activation layer.

[0291] Then, for the input data of the PCA module, the dot product results of each eigenvector in the input data of the PCA module and the corresponding weight vector are calculated respectively to obtain the first sample dot product result (i.e., the first sample dot product result of the PCA module). The value at each position in the dot product result of a eigenvector and the weight vector corresponding to the eigenvector is: the product of the value at the position in the eigenvector and the value at the position in the weight vector corresponding to the eigenvector.

[0292] That is to say, the parameter values ​​of the first sample dot product result in each parameter dimension are consistent with the parameter values ​​of the input data of the PCA module in each parameter dimension.

[0293] Continuing with the above example, the input data of a PCA module is represented as a 64×400 feature vector. Correspondingly, the first sample dot product result of the PCA module is also represented as a 64×400 feature vector, where 64 is the number of channels corresponding to the first sample dot product result of the PCA module; and 400 is the length of each feature vector in the first sample dot product result of the PCA module.

[0294] For the first sample fusion result of the PCA module, the dot product results of each feature vector in the first sample fusion result and the corresponding weight vector are calculated respectively to obtain the second sample dot product result (that is, the second sample dot product result of the PCA module). The specific process of calculating the dot product result can refer to the above embodiment, which will not be repeated here.

[0295] Similarly, the parameter values ​​of the second sample dot product result in each parameter dimension are consistent with the parameter values ​​of the first sample fusion result of the PCA module in each parameter dimension.

[0296] Continuing with the above example, the first sample fusion result of a PCA module is expressed as a 64×400 feature vector. Correspondingly, the second sample dot product result of the PCA module is also expressed as a 64×400 feature vector, where 64 is the number of channels corresponding to the second sample dot product result of the PCA module; 400 is the length of each feature vector in the second sample dot product result of the PCA module.

[0297] That is, the number of channels corresponding to the first sample dot product result is consistent with the number of channels corresponding to the second sample dot product result, and accordingly, the dot addition result of the first sample dot product result and the second sample dot product result can be calculated. That is, for the same channel, the dot addition result of the feature vector corresponding to the channel in the first sample dot product result and the feature vector corresponding to the channel in the second sample dot product result is calculated.

[0298] The value at each position in the point addition result of one eigenvector and another eigenvector is: the sum of the value at that position in the eigenvector and the value at that position in the weight vector corresponding to the eigenvector.

[0299] That is to say, the parameter values ​​of the point addition results of the first sample dot product result and the second sample dot product result in each parameter dimension are consistent with the parameter values ​​of the first sample dot product result in each parameter dimension, and are consistent with the parameter values ​​of the second sample dot product result in each parameter dimension.

[0300] In the above example, the first sample dot product result of a PCA module is expressed as a 64×400 feature vector, and the second sample dot product result of the PCA module is also expressed as a 64×400 feature vector. Correspondingly, the dot addition result of the first sample dot product result and the second sample dot product result is also expressed as a 64×400 feature vector, where 64 is the number of channels corresponding to the dot addition result of the PCA module; 400 is the length of each feature vector in the dot addition result of the PCA module.

[0301] See also Figure 5 , Figure 5 A data processing flow chart of the first residual gate network provided in an embodiment of the present application.

[0302] Figure 5In the example, Input1 represents the input data of a partial cross attention module. Input2 represents the first sample fusion result of the partial cross attention module. Accordingly, the input data of the partial cross attention module and the first sample fusion result can be concatenated to obtain the first sample concatenation result. The first sample concatenation result of the PCA module is input into a multi-layer perceptron (MLP), and the output result of the multi-layer perceptron is input into the activation layer, and the Sigmoid activation function is used to obtain the weight vector of each feature vector in the first sample concatenation result.

[0303] Then, the dot product results of each eigenvector and the corresponding weight vector in the input data of the PCA module are calculated respectively to obtain the first sample dot product result (Mul1, dot product result 1); the dot product results of each eigenvector and the corresponding weight vector in the first sample fusion result of the PCA module are calculated respectively to obtain the second sample dot product result (Mul2, dot product result 2). The dot addition result (Sum) of the first sample dot product result and the second sample dot product result of the PCA module is calculated to obtain the output data of the RGL of the PCA module.

[0304] Based on the above processing, a residual gate network of a partial cross attention module can splice the input data of the PCA module and the first sample fusion result according to the channel, and use the multi-layer perceptron and the activation function to obtain the weight vector corresponding to each feature vector in the input data and the first sample fusion result. In this way, the weight of each feature vector can be dynamically adjusted according to the actual input data of the PCA module and the first sample fusion result. Correspondingly, the model can also learn appropriate weights to perform weighted processing on each feature vector. That is, the two sets of weight vectors calculated are respectively multiplied by the input data of the PCA module and the first sample fusion result, and the two sets of dot product results are fused by dot addition to obtain the output data of the residual gate network. In this way, the output data of the residual gate network can have the characteristics of two sets of data (i.e., the input data of the PCA module and the first sample fusion result), so that the output data of the residual gate network can better reflect the image characteristics of the image and have stronger representation ability.

[0305] After obtaining the output data of the residual gate network of the PCA (i.e., the point addition result), the calculated point addition result can be input into the feedforward neural network to obtain the output data of this part of the cross attention module.

[0306] In one implementation, a partial cross attention module further includes: a third normalization layer:

[0307] The above step six includes:

[0308] The calculated point addition result is input into the feedforward neural network to obtain the output result of the feedforward neural network; the output result of the feedforward neural network is input into the third normalization layer to obtain the output data of this part of the cross-attention module.

[0309] In an embodiment of the present application, a feedforward neural network of a PCA module can process the output data of the residual gate network of the PCA module. Furthermore, the output data of the feedforward neural network of the PCA module is input into the third normalization layer in the PCA module to obtain the output data of the third normalization layer, that is, the output data of the PCA module.

[0310] In one implementation, after obtaining a trained target detection model based on the model training method provided in this application, the trained target detection model can be tested.

[0311] In an embodiment of the present application, a technician can select a preset test set, input the images in the test set into the trained target detection model, and obtain the detection results of the images. Furthermore, based on the detection results of each image in the test set, the test results of the trained target detection model can be obtained. For example, the test set can be: COCO (Common Objects in Context) image dataset, and / or, COCO-FP (Common Objects in Context- False Positive) image dataset.

[0312] See Table 1, which shows the test results corresponding to the target detection models trained using different schemes.

[0313] Table 1

[0314]

[0315] Among them, the scheme represents: the training scheme of the target detection model. The parameter amount represents the number of model parameters of the target detection model. FLOPs represents: the number of floating-point operations performed by the target detection model per second. Latency represents: the time required for the target detection model to detect the target in an image. AP (Average Precision) represents: the area under the precision-recall curve corresponding to the target detection model. AP50 represents: the area under the precision-recall curve corresponding to the target detection model when the intersection-over-union threshold is 0.5. AP75 represents: the area under the precision-recall curve corresponding to the target detection model when the intersection-over-union threshold is 0.75. Among them, for the detection result of an object in an image, if the intersection-over-union ratio of the predicted box and the true box of the object is greater than the intersection-over-union ratio threshold, it means that the image is a true positive sample for the object; if the intersection-over-union ratio of the predicted box and the true box of the object is not greater than the intersection-over-union ratio threshold, it means that the image is not a true positive sample for the object.

[0316] Correspondingly, DETR (Detection Transformer) means: the target detection model obtained by training based on the Transformer network (for the sake of convenience, it can be called model A), in which the self-attention layer and the cross-attention layer in the decoder of the target detection model are connected in series. For model A, the number of parameters of model A is 32.96 (M, megabytes), the FLOPs of model A is 107.2 (G, gigabytes), the latency of model A is: 9.02 (ms, milliseconds), the AP corresponding to model A is: 53.9, the AP50 corresponding to model A is: 72.0, and the AP75 corresponding to model A is: 58.4.

[0317] PCA means: the target detection model trained according to the model training method provided in this application (for the sake of convenience, it can be referred to as model B). That is, the decoder in model B contains at least one PCA module, and a PCA module contains a self-attention layer and a cross-attention layer connected in parallel. And the PCA module of model B does not contain an RGL network, that is, the above Figure 2 The PCA structure shown in the figure. For model B, the number of parameters of model B is 29.71 (M, megabytes), the FLOPs of model B is 95.1 (G, gigabytes), the latency of model B is: 8.21 (ms, milliseconds), the AP corresponding to model B is: 54.1, the AP50 corresponding to model B is: 72.3, and the AP75 corresponding to model B is: 58.7.

[0318] RGL means: the target detection model trained according to the model training method provided in this application (for the sake of convenience, it can be referred to as model C). That is, the decoder in model C contains at least one PCA module, and a PCA module contains a self-attention layer and a cross-attention layer connected in parallel. And the PCA module of model C contains the RGL network, that is, the above Figure 4 The PCA structure shown in the figure. For model C, the number of parameters of model C is 29.71 (M, mega), the FLOPs of model C is 95.4 (G, giga), the latency of model C is: 8.29 (ms, milliseconds), the AP corresponding to model C is: 54.3, the AP50 corresponding to model C is: 72.5, and the AP75 corresponding to model C is: 58.8.

[0319] See Table 2, which shows the test results of the target detection model trained using different loss functions.

[0320] Table 2

[0321]

[0322] Among them, the scheme represents: the loss function used in the training process of the target detection model. AP (AveragePrecision) represents: the area under the precision-recall curve corresponding to the target detection model. AP50 represents: the area under the precision-recall curve corresponding to the target detection model when the intersection-over-union threshold is 0.5. AP75 represents: the area under the precision-recall curve corresponding to the target detection model when the intersection-over-union threshold is 0.75. Among them, for the detection result of an object in the image, if the intersection-over-union ratio of the predicted box and the true box of the object is greater than the intersection-over-union ratio threshold, it means that the image is a true positive sample for the object; if the intersection-over-union ratio of the predicted box and the true box of the object is not greater than the intersection-over-union ratio threshold, it means that the image is not a true positive sample for the object.

[0323] COCO means: The test set used in the testing process of the target detection model is the COCO image dataset.

[0324] Correspondingly, the target detection model trained according to the model training method provided in this application, and the target detection model (which can be referred to as model D) is trained based on the above FL loss function. For model D, the AP corresponding to model D is: 52.1, the AP50 corresponding to model D is: 70.2, and the AP75 corresponding to model D is: 56.4.

[0325] The target detection model trained according to the model training method provided in this application, and the target detection model (which can be referred to as model E) is trained based on the above VFL loss function. For model E, the AP corresponding to model E is: 54.4, the AP50 corresponding to model E is: 72.6, and the AP75 corresponding to model E is: 58.8.

[0326] The target detection model trained according to the model training method provided in this application, and the target detection model (which can be referred to as model E) is trained based on the above VFL loss function. For model E, the AP corresponding to model E is: 54.4, the AP50 corresponding to model E is: 72.6, and the AP75 corresponding to model E is: 58.8.

[0327] COCO-FP means: The test set used in the testing process of the target detection model is the COCO-FP image dataset.

[0328] Correspondingly, for model D, the AP corresponding to model D is: 48.9, the AP50 corresponding to model D is: 67.3, and the AP75 corresponding to model D is: 52.8.

[0329] For model E, the AP corresponding to model E is: 48.8, the AP50 corresponding to model E is: 67.2, and the AP75 corresponding to model E is: 52.5.

[0330] For model E, the AP corresponding to model E is: 50.8, the AP50 corresponding to model E is: 68.9, and the AP75 corresponding to model E is: 54.9.

[0331] Based on the same inventive concept, an embodiment of the present application provides a target detection method. The method can be applied to electronic devices. For example, the electronic device is: an intelligent device (such as a server) for running a target detection model. Accordingly, the electronic device can perform target detection on an image by running a target detection model through its own processing unit. For example, the processing unit can be an NPU (Neural Network Processing Unit), a GPU (Graphic Processing Unit), etc.

[0332] See also Figure 6 , Figure 6 The first flow chart of the target detection method provided in the embodiment of the present application. The method comprises the following steps:

[0333] S601: Acquire an image to be detected.

[0334] S602: Use the encoder in the trained object detection model to extract features from the image to be detected to obtain a first predicted feature vector.

[0335] The trained target detection model also includes a decoder; the decoder includes at least one partial cross-attention module, and a partial cross-attention module includes a self-attention layer and a cross-attention layer connected in parallel.

[0336] S603: For each cross-attention module, the input data of the cross-attention module is split in the channel dimension to obtain a third input part and a fourth input part.

[0337] Among them, the input data of the first partial cross-attention module is the first predicted feature vector.

[0338] S604: Use the self-attention layer and the cross-attention layer of this part of the cross-attention module to process the third input part and the fourth input part respectively.

[0339] S605: Combine the processing results of the self-attention layer and the cross-attention layer of this part of the cross-attention module to obtain the output data of this part of the cross-attention module.

[0340] S606: Use the output data of the last partial cross-attention module to perform detection to obtain the detection results of each object in the image to be detected.

[0341] Based on the above processing, in the process of performing target detection on an image (i.e., a sample image or an image to be detected), after obtaining the feature representation of the image extracted by the encoder in the target detection model (i.e., the first sample feature vector or the first predicted feature vector), the output data of the encoder can be input into the decoder in the target detection model for processing. In the present application, the decoder includes at least one partial cross-attention module, and a partial cross-attention module includes a self-attention layer and a cross-attention layer connected in parallel. Correspondingly, for each partial cross-attention module, the input data of the partial cross-attention module can be split in the channel dimension, and the self-attention layer and the cross-attention layer of the partial cross-attention module are used to process the two parts (i.e., the first input part and the second input part, or, the third input part or the fourth input part) obtained by splitting. In other words, the partial cross-attention module in the decoder can process the output data of the encoder using the parallel structure of the self-attention layer and the cross-attention layer, and can also obtain the feature representation for the object in the image based on the self-attention mechanism and the cross-attention mechanism (i.e., the output data of the last partial cross-attention module in the decoder). Furthermore, the output data of the last partial cross-attention module can be used for detection to obtain the detection result of each object in the image. In this way, the parallel structure of the self-attention layer and the cross-attention layer in some cross-attention modules can reduce the latency of target detection and improve the efficiency of target detection.

[0342] With respect to step S601 , the image to be detected represents an image that currently needs to perform target detection.

[0343] For example, a user may upload multiple videos to be detected to the electronic device, and accordingly, the target detection model may obtain each video frame in each video to be detected as an image to be detected.

[0344] Among them, the target detection model is trained based on any model training method provided in this application. The target detection model supports processing multiple video streams at the same time and can process more than 100 images in one second.

[0345] For step S602, the process of extracting features from the image to be detected using the encoder in the trained target detection model to obtain the first predicted feature vector can refer to the relevant description of step S102, and the process of obtaining the first sample feature vector of a sample image is not repeated here.

[0346] For step S603, for each part of the cross-attention module, the input data of the part of the cross-attention module is split in the channel dimension to obtain the third input part and the fourth input part. The process of obtaining the first input part and the second input part can be referred to the relevant description of step S103, and will not be repeated here.

[0347] For step S604, the process of using the self-attention layer and the cross-attention layer of this part of the cross-attention module to process the third input part and the fourth input part respectively can refer to the relevant description of step S104, and the process of processing the first input part and the second input part respectively will not be repeated here.

[0348] For step S605, the process of obtaining the output data of this part of the cross-attention module by combining the processing results of the self-attention layer and the cross-attention layer of this part of the cross-attention module can refer to the relevant description of step S105 and will not be repeated here.

[0349] For step S606, the process of using the output data of the last partial cross-attention module to perform detection to obtain the detection results of each object in the image to be detected can be referred to the relevant description of step S106, and the process of obtaining the detection results of each object in the sample image will not be repeated here.

[0350] In some embodiments, the decoder comprises: a plurality of serially connected partial cross-attention modules.

[0351] In the embodiment of the present application, for each PCA module except the first PCA module, the input data of the PCA module is the output data of the previous PCA module. The input data of the first PCA module is the output data of the encoder (i.e., the first sample feature vector), and the output data of the last PCA module is the output data of the decoder.

[0352] In this way, when the decoder includes multiple PCA modules connected in series, the decoder can perform decoding in a multi-round iterative manner. Each PCA module can further refine the prediction of the object in the image based on the output of the previous PCA. Through multiple rounds of iterations, the object detection model can gradually optimize the object positioning and classification results, and improve the accuracy of detection.

[0353] In some embodiments, a partial criss-cross attention module further comprises: a residual gate network.

[0354] Step S605 includes:

[0355] Step S6051: Combine the processing results of the self-attention layer and the cross-attention layer of this part of the cross-attention module to obtain the first prediction fusion result.

[0356] Step S6052: Input the input data of this part of the cross-attention module and the first prediction fusion result into the residual gate network to obtain the output data of this part of the cross-attention module.

[0357] In an embodiment of the present application, the process of obtaining the first predicted fusion result by combining the processing results of the self-attention layer and the cross-attention layer of this part of the cross-attention module can refer to the relevant description of step S1051, and the process of obtaining the first sample fusion result will not be repeated here.

[0358] The input data of this partial cross-attention module and the first prediction fusion result are input into the residual gate network to obtain the output data of this partial cross-attention module. The process of obtaining the output data of a partial cross-attention module can be referred to the relevant description of step S1052, which will not be repeated here.

[0359] Based on the above processing, for a residual gate network of a PCA module, the residual gate network can obtain the input data of the PCA module and the first sample fusion result obtained by the attention mechanism for fusion. In other words, the residual gate network has the ability to fuse the original input and the two sets of features processed by the attention mechanism. Correspondingly, the output data of the residual gate network can represent the image features in combination with the above two sets of features. In this way, the representation ability of feature data can be improved and the accuracy of the target detection model can be improved.

[0360] In some embodiments, a partial cross attention module further includes a feedforward neural network; a residual gate network includes a multi-layer perceptron and an activation layer. Step S6052 includes:

[0361] Step S60521: splice the input data of this part of the cross-attention module and the first prediction fusion result to obtain a first prediction splicing result.

[0362] Step S60522: Input the first prediction splicing result into a multi-layer perceptron, and input the output result of the multi-layer perceptron into the activation layer to obtain a weight vector of each feature vector in the first prediction splicing result.

[0363] Step S60523: Calculate the dot product results of each feature vector and the corresponding weight vector in the input data of this part of the cross-attention module respectively to obtain the first predicted dot product result.

[0364] Step S60524: Calculate the dot product results of each feature vector in the first prediction fusion result and the corresponding weight vector respectively to obtain a second prediction multiplication result.

[0365] Step S60525: Calculate the dot addition result of the first predicted dot product result and the second predicted dot product result.

[0366] Step S60526: Input the calculated point addition result into the feedforward neural network to obtain the output data of this part of the cross-attention module.

[0367] In an embodiment of the present application, the input data of this part of the cross-attention module and the first predicted fusion result are spliced ​​to obtain the first predicted splicing result. The process of obtaining the first sample splicing result can be referred to the relevant description of the above step one, and will not be repeated here.

[0368] The process of inputting the first prediction splicing result into a multi-layer perceptron, and inputting the output result of the multi-layer perceptron into the activation layer to obtain the weight vector of each feature vector in the first prediction splicing result can refer to the relevant description of the above step 2 and will not be repeated here.

[0369] The process of respectively calculating the dot product results of each feature vector in the input data of this part of the cross-attention module and the corresponding weight vector to obtain the first predicted dot product result can refer to the relevant description of the above step three, and the process of obtaining the first sample dot product result will not be repeated here.

[0370] The process of respectively calculating the dot product results of each feature vector in the first prediction fusion result and the corresponding weight vector to obtain the second prediction multiplication result can refer to the relevant description of the above step four, and the process of obtaining the second sample dot product result will not be repeated here.

[0371] The process of calculating the dot addition result of the first predicted dot product result and the second predicted dot product result can refer to the relevant description of the above step five, and the process of calculating the dot addition result of the first sample dot product result and the second sample dot product result, which will not be repeated here.

[0372] The process of inputting the calculated point addition result into the feedforward neural network to obtain the output data of this part of the cross-attention module can refer to the relevant description of step 6 above and will not be repeated here.

[0373] Based on the above processing, a residual gate network of a partial cross attention module can splice the input data of the PCA module and the first sample fusion result according to the channel, and use the multi-layer perceptron and the activation function to obtain the weight vector corresponding to each feature vector in the input data and the first sample fusion result. In this way, the weight of each feature vector can be dynamically adjusted according to the actual input data of the PCA module and the first sample fusion result. Correspondingly, the model can also learn appropriate weights to perform weighted processing on each feature vector. That is, the two sets of weight vectors calculated are respectively multiplied by the input data of the PCA module and the first sample fusion result, and the two sets of dot product results are fused by dot addition to obtain the output data of the residual gate network. In this way, the output data of the residual gate network can have the characteristics of two sets of data (i.e., the input data of the PCA module and the first sample fusion result), so that the output data of the residual gate network can better reflect the image characteristics of the image and have stronger representation ability.

[0374] In some embodiments, a partial cross attention module further includes: a first normalization layer and a second normalization layer. Step S6051 includes:

[0375] The processing result of the self-attention layer of this part of the cross-attention module and the dot-add result of the input data of the cross-attention layer of this part of the cross-attention module are input to the first normalization layer to obtain a first predicted normalized result; the processing result of the cross-attention layer of this part of the cross-attention module and the dot-add result of the input data of the self-attention layer of this part of the cross-attention module are input to the second normalization layer to obtain a second predicted normalized result; the first predicted normalized result and the second predicted normalized result are spliced ​​to obtain a first predicted fusion result.

[0376] and / or,

[0377] A partial cross attention module also includes: a third normalization layer. Step S60526 includes:

[0378] The calculated point addition result is input into the feedforward neural network to obtain the output result of the feedforward neural network; the output result of the feedforward neural network is input into the third normalization layer to obtain the output data of this part of the cross-attention module.

[0379] In the embodiment of the present application, with regard to the first normalization layer, the second normalization layer and the third normalization layer, reference may be made to the relevant description in the above embodiment, which will not be elaborated here.

[0380] In some embodiments, the third input portion and the fourth input portion correspond to the same number of channels.

[0381] Based on the above processing, the input data of a partial cross-attention module can be split into two parts in half in the channel dimension. In this way, when the self-attention layer and the cross-attention layer of the partial cross-attention module are used to process the third input part and the fourth input part respectively, the data processing of the self-attention layer and the cross-attention layer can be balanced, avoiding the situation where a single module takes a long time to process, which can further reduce the delay of target detection and improve the efficiency of target detection.

[0382] In some embodiments, the detection result of each object in the image to be detected includes: the predicted box of the object and the confidence level indicating the probability of the category to which the object belongs. The object detection model is trained based on sample images and labels of each sample image using a preset loss function; wherein the label of a sample image includes: the real box and real category of each object in the sample image; the preset loss function is:

[0383] ;

[0384] represents the loss value; Represents the intersection-over-union ratio between the predicted box and the true box of an object in a sample image; Indicates the confidence of the object; is a hyperparameter; Represents a logarithmic function.

[0385] Based on the same inventive concept, the present application embodiment provides a model training device. Figure 7 , Figure 7 A structural diagram of a model training device provided in an embodiment of the present application. The device comprises:

[0386] The sample acquisition unit 701 is used to acquire sample images and labels of each sample image; wherein the label of a sample image is used to represent each object in the sample image;

[0387] A first feature extraction unit 702 is used to extract features of each sample image using an encoder in the target detection model in the initial state to obtain a first sample feature vector; wherein the target detection model in the initial state further includes a decoder; the decoder includes at least one partial cross attention module, and a partial cross attention module includes a self-attention layer and a cross attention layer connected in parallel;

[0388] A first splitting unit 703 is used to split the input data of each cross-attention module in the channel dimension to obtain a first input part and a second input part; wherein the input data of the first cross-attention module is the first sample feature vector;

[0389] A first processing unit 704 is used to process the first input part and the second input part respectively by using the self-attention layer and the cross-attention layer of the cross-attention module;

[0390] A first fusion unit 705 is used to combine the processing results of the self-attention layer and the cross-attention layer of the part of the cross-attention module to obtain the output data of the part of the cross-attention module;

[0391] A first detection unit 706, configured to perform detection using the output data of the last partial cross attention module to obtain a detection result of each object in the sample image;

[0392] The loss adjustment unit 707 is used to calculate the loss value based on the difference between the detection result of each object in the sample image and the label of the sample image, and adjust the model parameters of the initial state target detection model based on the obtained loss value until convergence is reached to obtain a trained target detection model.

[0393] In some embodiments, the decoder comprises: a plurality of serially connected partial cross-attention modules.

[0394] In some embodiments, a partial crisscross attention module further comprises: a residual gate network;

[0395] The first fusion unit 705 includes:

[0396] A first fusion subunit is used to combine the processing results of the self-attention layer and the cross-attention layer of the part of the cross-attention module to obtain a first sample fusion result;

[0397] The first residual processing subunit is used to input the input data of this part of the cross-attention module and the first sample fusion result into the residual gate network to obtain the output data of this part of the cross-attention module.

[0398] In some embodiments, a partial crisscross attention module further comprises a feed-forward neural network; a residual gate network comprises a multi-layer perceptron and activation layer;

[0399] The first residual processing subunit is specifically configured to:

[0400] Splicing the input data of the part of the cross attention module and the first sample fusion result to obtain a first sample splicing result;

[0401] Inputting the first sample splicing result into the multi-layer perceptron, and inputting the output result of the multi-layer perceptron into the activation layer, to obtain a weight vector of each feature vector in the first sample splicing result;

[0402] Calculate the dot product results of each feature vector and the corresponding weight vector in the input data of this part of the cross attention module respectively, and obtain the first sample dot product result;

[0403] Respectively calculating the dot product results of each feature vector in the first sample fusion result and the corresponding weight vector to obtain a second sample dot product result;

[0404] Calculate a dot-add result of the dot product result of the first sample and a dot product result of the second sample;

[0405] The calculated point addition result is input into the feedforward neural network to obtain the output data of this part of the cross attention module.

[0406] In some embodiments, a partial cross attention module further includes: a first normalization layer and a second normalization layer;

[0407] The first fusion subunit is specifically used for:

[0408] Inputting the processing result of the self-attention layer of the part of the cross attention module and the point addition result of the input data of the cross attention layer of the part of the cross attention module into the first normalization layer to obtain a first sample normalization result;

[0409] Inputting the processing result of the cross attention layer of the part of the cross attention module and the point addition result of the input data of the self-attention layer of the part of the cross attention module into the second normalization layer to obtain a second sample normalization result;

[0410] splicing the first sample normalization result and the second sample normalization result to obtain a first sample fusion result;

[0411] and / or,

[0412] A partial cross attention module also includes: The third normalization layer:

[0413] The first residual processing subunit is specifically configured to:

[0414] Inputting the calculated point addition result into the feedforward neural network to obtain the output result of the feedforward neural network;

[0415] The output result of the feedforward neural network is input into the third normalization layer to obtain the output data of this part of the cross attention module.

[0416] In some embodiments, the label of a sample image includes: the real box and the real category of each object in the sample image; the detection result of each object in the sample image includes: the predicted box of the object and the confidence indicating the probability that the object belongs to the corresponding real category;

[0417] The loss adjustment unit 707 is specifically used to:

[0418] The loss value is calculated based on the difference between the detection result of each object in the sample image and the label of the sample image using a preset loss function; wherein the preset loss function is:

[0419] ;

[0420] represents the loss value; Represents the intersection-over-union ratio between the predicted box and the true box of an object in the sample image; Indicates the confidence of the object; is a hyperparameter; Represents a logarithmic function.

[0421] In some embodiments, the first input part and the second input part correspond to the same number of channels.

[0422] Based on the same inventive concept, the present application embodiment provides a target detection device. Figure 8 , Figure 8 This is a structural diagram of a target detection device provided in an embodiment of the present application. The device includes:

[0423] The image acquisition unit 801 is used to acquire the image to be detected;

[0424] The second feature extraction unit 802 is used to extract features of the image to be detected by using the encoder in the trained target detection model to obtain a first predicted feature vector; wherein the trained target detection model further includes a decoder; the decoder includes at least one partial cross attention module, and a partial cross attention module includes a self-attention layer and a cross attention layer connected in parallel;

[0425] The second splitting unit 803 is used to split the input data of each partial cross attention module in the channel dimension to obtain a third input part and a fourth input part; wherein the input data of the first partial cross attention module is the first prediction feature vector;

[0426] A second processing unit 804 is used to process the third input part and the fourth input part respectively by using the self-attention layer and the cross-attention layer of the cross-attention module;

[0427] The second fusion unit 805 is used to combine the processing results of the self-attention layer and the cross-attention layer of the part of the cross-attention module to obtain the output data of the part of the cross-attention module;

[0428] The second detection unit 806 is used to use the output data of the last partial cross-attention module to perform detection and obtain the detection result of each object in the image to be detected.

[0429] In some embodiments, the decoder comprises: a plurality of serially connected partial cross-attention modules.

[0430] In some embodiments, a partial crisscross attention module further comprises: a residual gate network;

[0431] The second fusion unit 805 includes:

[0432] The second fusion subunit is used to combine the processing results of the self-attention layer and the cross-attention layer of the part of the cross-attention module to obtain a first prediction fusion result;

[0433] The second residual processing sub-unit is used to input the input data of this part of the cross-attention module and the first prediction fusion result into the residual gate network to obtain the output data of this part of the cross-attention module.

[0434] In some embodiments, a partial crisscross attention module further comprises a feed-forward neural network; a residual gate network comprises a multi-layer perceptron and activation layer;

[0435] The second residual processing subunit is specifically used for:

[0436] Splicing the input data of the part of the cross attention module and the first prediction fusion result to obtain a first prediction splicing result;

[0437] Inputting the first prediction splicing result into the multi-layer perceptron, and inputting the output result of the multi-layer perceptron into the activation layer, to obtain a weight vector of each feature vector in the first prediction splicing result;

[0438] Calculate the dot product results of each feature vector and the corresponding weight vector in the input data of the cross attention module respectively to obtain a first predicted dot product result;

[0439] Respectively calculating the dot product results of each feature vector in the first prediction fusion result and the corresponding weight vector to obtain a second prediction multiplication result;

[0440] Calculate a dot addition result of the first predicted dot product result and the second predicted dot product result;

[0441] The calculated point addition result is input into the feedforward neural network to obtain the output data of this part of the cross attention module.

[0442] In some embodiments, a partial cross attention module further includes: a first normalization layer and a second normalization layer;

[0443] The second fusion subunit is specifically used for:

[0444] Inputting the processing result of the self-attention layer of the part of the cross attention module and the point addition result of the input data of the cross attention layer of the part of the cross attention module into the first normalization layer to obtain a first predicted normalized result;

[0445] Inputting the processing result of the cross attention layer of the part of the cross attention module and the point addition result of the input data of the self-attention layer of the part of the cross attention module into the second normalization layer to obtain a second predicted normalization result;

[0446] splicing the first prediction normalization result and the second prediction normalization result to obtain a first prediction fusion result;

[0447] and / or,

[0448] A partial cross attention module also includes: The third normalization layer:

[0449] The second residual processing subunit is specifically used for:

[0450] Inputting the calculated point addition result into the feedforward neural network to obtain the output result of the feedforward neural network;

[0451] The output result of the feedforward neural network is input into the third normalization layer to obtain the output data of this part of the cross attention module.

[0452] In some embodiments, the third input part and the fourth input part correspond to the same number of channels.

[0453] In some embodiments, the detection result of each object in the image to be detected includes: a prediction box of the object and a confidence level indicating the probability of the category to which the object belongs;

[0454] The target detection model is obtained by training based on sample images and labels of each sample image using a preset loss function; wherein the label of a sample image includes: the real frame and real category of each object in the sample image;

[0455] The default loss function is:

[0456] ;

[0457] represents the loss value; Represents the intersection-over-union ratio between the predicted box and the true box of an object in a sample image; Indicates the confidence of the object; is a hyperparameter; Represents a logarithmic function.

[0458] The embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 901 , a communication interface 902 , a memory 903 and a communication bus 904 , wherein the processor 901 , the communication interface 902 , and the memory 903 communicate with each other via the communication bus 904 .

[0459] Memory 903, used for storing computer programs;

[0460] The processor 901 is used to implement any of the above-mentioned model training methods or steps of the target detection method when executing the program stored in the memory 903.

[0461] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0462] The communication interface is used for communication between the above electronic device and other devices.

[0463] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0464] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0465] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned model training methods or target detection methods are implemented.

[0466] In another embodiment provided by the present invention, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any model training method or target detection method in the above-mentioned embodiments.

[0467] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0468] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0469] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0470] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A model training method, characterized in that: The method comprises: Obtain sample images and labels of each sample image; wherein the label of a sample image is used to represent each object in the sample image; For each sample image, extract features of the sample image using an encoder in the target detection model in the initial state to obtain a first sample feature vector; wherein the target detection model in the initial state further includes a decoder; the decoder includes at least one partial cross attention module, and a partial cross attention module includes a self-attention layer and a cross attention layer connected in parallel; For each cross-attention module, the input data of the cross-attention module is split in the channel dimension to obtain a first input part and a second input part; wherein the input data of the first cross-attention module is the first sample feature vector; Using the self-attention layer and the cross-attention layer of the cross-attention module, the first input part and the second input part are processed respectively; Combining the processing results of the self-attention layer and the cross-attention layer of this part of the cross-attention module, the output data of this part of the cross-attention module is obtained; Use the output data of the last partial cross attention module to perform detection and obtain the detection results of each object in the sample image; Based on the difference between the detection result of each object in the sample image and the label of the sample image, a loss value is calculated, and based on the obtained loss value, the model parameters of the initial state of the target detection model are adjusted until convergence is reached to obtain a trained target detection model.

2. The method according to claim 1, characterized in that The decoder comprises: a plurality of serially connected partial cross attention modules.

3. The method according to claim 1, characterized in that A partial criss-cross attention module also includes: a residual gate network; The processing results of the self-attention layer and the cross-attention layer of the part of the cross-attention module are combined to obtain the output data of the part of the cross-attention module, including: Combining the processing results of the self-attention layer and the cross-attention layer of this part of the cross-attention module, a first sample fusion result is obtained; The input data of this part of the cross-attention module and the fusion result of the first sample are input into the residual gate network to obtain the output data of this part of the cross-attention module.

4. The method according to claim 3, characterized in that A partial criss-cross attention module also includes a feed-forward neural network; a residual gate network includes a multi-layer perceptron and activation layer; The step of inputting the input data of the part of the cross attention module and the first sample fusion result into the residual gate network to obtain the output data of the part of the cross attention module includes: Splicing the input data of the part of the cross attention module and the first sample fusion result to obtain a first sample splicing result; Inputting the first sample splicing result into the multi-layer perceptron, and inputting the output result of the multi-layer perceptron into the activation layer, to obtain a weight vector of each feature vector in the first sample splicing result; Calculate the dot product results of each feature vector and the corresponding weight vector in the input data of this part of the cross attention module respectively, and obtain the first sample dot product result; Respectively calculating the dot product results of each feature vector in the first sample fusion result and the corresponding weight vector to obtain a second sample dot product result; Calculate a dot-add result of the dot product result of the first sample and a dot product result of the second sample; The calculated point addition result is input into the feedforward neural network to obtain the output data of this part of the cross attention module.

5. The method according to claim 4, characterized in that A partial cross attention module also includes: a first normalization layer and a second normalization layer; The processing results of the self-attention layer and the cross-attention layer of the cross-attention module are combined to obtain a first sample fusion result, including: Inputting the processing result of the self-attention layer of the part of the cross attention module and the point addition result of the input data of the cross attention layer of the part of the cross attention module into the first normalization layer to obtain a first sample normalization result; Inputting the processing result of the cross attention layer of the part of the cross attention module and the point addition result of the input data of the self-attention layer of the part of the cross attention module into the second normalization layer to obtain a second sample normalization result; splicing the first sample normalization result and the second sample normalization result to obtain a first sample fusion result; and / or, A partial cross attention module also includes: The third normalization layer: The step of inputting the calculated point addition result into the feedforward neural network to obtain the output data of the cross attention module includes: Inputting the calculated point addition result into the feedforward neural network to obtain the output result of the feedforward neural network; The output result of the feedforward neural network is input into the third normalization layer to obtain the output data of this part of the cross attention module.

6. The method according to claim 1, characterized in that The label of a sample image includes: the real box and real category of each object in the sample image; the detection result of each object in the sample image includes: the predicted box of the object and the confidence indicating the probability that the object belongs to the corresponding real category; The calculating the loss value based on the difference between the detection result of each object in the sample image and the label of the sample image includes: The loss value is calculated based on the difference between the detection result of each object in the sample image and the label of the sample image using a preset loss function; wherein the preset loss function is: ； represents the loss value; Represents the intersection-over-union ratio between the predicted box and the true box of an object in the sample image; Indicates the confidence of the object; is a hyperparameter; Represents a logarithmic function.

7. The method according to any one of claims 1 to 6, characterized in that: The first input part and the second input part have the same number of channels.

8. A target detection method, characterized in that: The method comprises: Acquire the image to be detected; Using an encoder in a trained target detection model to extract features from the image to be detected, obtaining a first predicted feature vector; wherein the trained target detection model further comprises a decoder; the decoder comprises at least one partial cross attention module, and a partial cross attention module comprises a self-attention layer and a cross attention layer connected in parallel; For each cross-attention module, the input data of the cross-attention module is split in the channel dimension to obtain a third input part and a fourth input part; wherein the input data of the first cross-attention module is the first prediction feature vector; Using the self-attention layer and the cross-attention layer of the cross-attention module, the third input part and the fourth input part are processed respectively; Combining the processing results of the self-attention layer and the cross-attention layer of this part of the cross-attention module, the output data of this part of the cross-attention module is obtained; The output data of the last partial cross-attention module is used for detection to obtain the detection result of each object in the image to be detected.

9. The method according to claim 8, characterized in that The decoder comprises: a plurality of serially connected partial cross attention modules.

10. The method according to claim 8, characterized in that A partial criss-cross attention module also includes: a residual gate network; The processing results of the self-attention layer and the cross-attention layer of the part of the cross-attention module are combined to obtain the output data of the part of the cross-attention module, including: Combining the processing results of the self-attention layer and the cross-attention layer of this part of the cross-attention module, a first prediction fusion result is obtained; The input data of this part of the cross-attention module and the first prediction fusion result are input into the residual gate network to obtain the output data of this part of the cross-attention module.

11. The method according to claim 10, characterized in that A partial criss-cross attention module also includes a feed-forward neural network; a residual gate network includes a multi-layer perceptron and activation layer; The step of inputting the input data of the part of the cross attention module and the first prediction fusion result into the residual gate network to obtain the output data of the part of the cross attention module includes: Splicing the input data of the part of the cross attention module and the first prediction fusion result to obtain a first prediction splicing result; Inputting the first prediction splicing result into the multi-layer perceptron, and inputting the output result of the multi-layer perceptron into the activation layer, to obtain a weight vector of each feature vector in the first prediction splicing result; Calculate the dot product results of each feature vector and the corresponding weight vector in the input data of the cross attention module respectively to obtain a first predicted dot product result; Respectively calculating the dot product results of each feature vector in the first prediction fusion result and the corresponding weight vector to obtain a second prediction multiplication result; Calculate a dot addition result of the first predicted dot product result and the second predicted dot product result; The calculated point addition result is input into the feedforward neural network to obtain the output data of this part of the cross attention module.

12. The method according to claim 11, characterized in that A partial cross attention module also includes: a first normalization layer and a second normalization layer; The processing results of the self-attention layer and the cross-attention layer of the cross-attention module are combined to obtain a first prediction fusion result, including: Inputting the processing result of the self-attention layer of the part of the cross attention module and the point addition result of the input data of the cross attention layer of the part of the cross attention module into the first normalization layer to obtain a first predicted normalized result; Inputting the processing result of the cross attention layer of the part of the cross attention module and the point addition result of the input data of the self-attention layer of the part of the cross attention module into the second normalization layer to obtain a second predicted normalized result; splicing the first prediction normalization result and the second prediction normalization result to obtain a first prediction fusion result; and / or, A partial cross attention module also includes: The third normalization layer: The step of inputting the calculated point addition result into the feedforward neural network to obtain the output data of the cross attention module includes: Inputting the calculated point addition result into the feedforward neural network to obtain the output result of the feedforward neural network; The output result of the feedforward neural network is input into the third normalization layer to obtain the output data of this part of the cross attention module.

13. The method according to any one of claims 8 to 12, characterized in that: The third input part and the fourth input part correspond to the same number of channels.

14. The method according to any one of claims 8 to 12, characterized in that: The detection result of each object in the image to be detected includes: a prediction box of the object and a confidence level indicating the probability of the category to which the object belongs; The target detection model is obtained by training based on sample images and labels of each sample image using a preset loss function; wherein the label of a sample image includes: the real frame and real category of each object in the sample image; The default loss function is: ； represents the loss value; Represents the intersection-over-union ratio between the predicted box and the true box of an object in a sample image; Indicates the confidence of the object; is a hyperparameter; Represents a logarithmic function.

15. A model training device, characterized in that: The device comprises: A sample acquisition unit, used to acquire sample images and labels of each sample image; wherein the label of a sample image is used to represent each object in the sample image; A first feature extraction unit is used to extract features of each sample image using an encoder in the target detection model in the initial state to obtain a first sample feature vector; wherein the target detection model in the initial state further includes a decoder; the decoder includes at least one partial cross attention module, and a partial cross attention module includes a self-attention layer and a cross attention layer connected in parallel; A first splitting unit is used for splitting the input data of each partial cross attention module in the channel dimension to obtain a first input part and a second input part; wherein the input data of the first partial cross attention module is the first sample feature vector; A first processing unit, configured to process the first input part and the second input part respectively using a self-attention layer and a cross-attention layer of the cross-attention module; A first fusion unit is used to combine the processing results of the self-attention layer and the cross-attention layer of the part of the cross-attention module to obtain the output data of the part of the cross-attention module; A first detection unit, used for performing detection using the output data of the last partial cross attention module to obtain a detection result of each object in the sample image; The loss adjustment unit is used to calculate the loss value based on the difference between the detection result of each object in the sample image and the label of the sample image, and adjust the model parameters of the target detection model in the initial state based on the obtained loss value until convergence is reached to obtain a trained target detection model.

16. A target detection device, characterized in that: The device comprises: An image acquisition unit to be detected, used to acquire the image to be detected; A second feature extraction unit is used to extract features of the image to be detected by using an encoder in a trained target detection model to obtain a first predicted feature vector; wherein the trained target detection model further includes a decoder; the decoder includes at least one partial cross attention module, and a partial cross attention module includes a self-attention layer and a cross attention layer connected in parallel; A second splitting unit is used for splitting the input data of each partial cross attention module in the channel dimension to obtain a third input part and a fourth input part; wherein the input data of the first partial cross attention module is the first prediction feature vector; A second processing unit is used to process the third input part and the fourth input part respectively by using the self-attention layer and the cross-attention layer of the part of the cross-attention module; A second fusion unit is used to combine the processing results of the self-attention layer and the cross-attention layer of the part of the cross-attention module to obtain the output data of the part of the cross-attention module; The second detection unit is used to use the output data of the last partial cross-attention module to perform detection to obtain the detection result of each object in the image to be detected.

17. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-7, or the method steps described in any one of claims 8-14, when executing a program stored in a memory.

Images