Model training and image processing method, device, equipment, and storage medium

Abstract

The embodiments of the present application disclose a model training and image processing method, apparatus, device, storage medium and computer program product, wherein the method includes: obtaining a first augmented image and a second augmented image; performing target detection on the first augmented image using a first model to obtain at least one first detection result including a first prediction object sequence, and performing target detection on the second augmented image using a second model to obtain at least one second detection result including a second prediction object sequence; matching each first prediction object sequence and each second prediction object sequence to obtain at least one pair of first prediction object sequences and second prediction object sequences having a target matching relationship; updating the model parameters of the first model at least once based on each pair of first prediction object sequences and second prediction object sequences having a target matching relationship to obtain a trained first model.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of artificial intelligence, and in particular to a model training and image processing method, apparatus, device, storage medium and computer program product. Background Art

[0002] Object detection is a critical problem in fields such as computer vision and industrial inspection. It uses algorithms to determine the location and corresponding classification of objects of interest within an image. Compared to image classification, object detection is a more prediction-intensive computer vision task. Training object detection models requires high annotation requirements, resulting in high annotation costs. Summary of the Invention

[0003] In view of this, embodiments of the present application provide a model training and image processing method, apparatus, device, storage medium and computer program product.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In one aspect, an embodiment of the present application provides a model training method, the method comprising:

[0006] Obtaining a first augmented image and a second augmented image obtained by respectively performing augmentation processing on the first image sample;

[0007] Performing object detection on the first augmented image using a first model to be trained to obtain at least one first detection result including a first predicted object sequence, and performing object detection on the second augmented image using a second model to obtain at least one second detection result including a second predicted object sequence;

[0008] Matching each of the first prediction object sequences and each of the second prediction object sequences to obtain at least one pair of first prediction object sequences and second prediction object sequences having a target matching relationship;

[0009] Based on each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship, the model parameters of the first model are updated at least once to obtain the trained first model.

[0010] On the other hand, an embodiment of the present application provides an image processing method, including:

[0011] Get the image to be processed;

[0012] Using the trained fourth model, target detection is performed on the image to be processed to obtain a third detection result; wherein, the third model includes at least one of the following: the first model obtained by the above-mentioned model training method, and the third model obtained by the above-mentioned model training method.

[0013] On the other hand, an embodiment of the present application provides a model training device, the device comprising:

[0014] A first acquisition module is used to acquire a first augmented image and a second augmented image obtained by respectively performing augmentation processing on the first image sample;

[0015] a first detection module, configured to perform object detection on the first augmented image using a first model to be trained, to obtain at least one first detection result including a first predicted object sequence, and to perform object detection on the second augmented image using a second model, to obtain at least one second detection result including a second predicted object sequence;

[0016] a first matching module, configured to match each of the first prediction object sequences with each of the second prediction object sequences to obtain at least one pair of first prediction object sequences and second prediction object sequences having a target matching relationship;

[0017] The first updating module is configured to update the model parameters of the first model at least once based on each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship, so as to obtain the trained first model.

[0018] In another aspect, an embodiment of the present application provides an image processing device, comprising:

[0019] A third acquisition module is used to acquire the image to be processed;

[0020] The second detection module is used to use the trained fourth model to perform target detection on the image to be processed to obtain a third detection result; wherein, the third model includes at least one of the following: the first model obtained by using the above-mentioned model training method, and the third model obtained by using the above-mentioned model training method.

[0021] On the other hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, it implements some or all of the steps in the above method.

[0022] On the other hand, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements some or all of the steps in the above method when executed by a processor.

[0023] On the other hand, an embodiment of the present application provides a computer program, including computer-readable code. When the computer-readable code is executed in a computer device, the processor in the display device executes some or all of the steps for implementing the above method.

[0024] On the other hand, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, implements some or all of the steps in the above method.

[0025] In an embodiment of the present application, a first augmented image and a second augmented image are obtained after augmenting a first image sample respectively; a first model to be trained is used to perform target detection on the first augmented image to obtain at least one first detection result including a first prediction object sequence, and a second model is used to perform target detection on the second augmented image to obtain at least one second detection result including a second prediction object sequence; each first prediction object sequence and each second prediction object sequence are matched to obtain at least one pair of first prediction object sequences and second prediction object sequences having a target matching relationship; based on each pair of first prediction object sequences and second prediction object sequences having a target matching relationship, the model parameters of the first model are updated at least once to obtain the trained first model. In this way, the sequence-level self-supervised training process of the target detection model can be realized by maintaining the consistency between the first prediction object sequence and the second prediction object sequence obtained after the first model and the second model process the first augmented image and the second augmented image of the same image sample respectively, and the overall network structure of the target detection model can be trained, thereby effectively improving the performance of the entire target detection model and reducing the labeling cost during the target detection model training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0032] Figure 7A A schematic diagram of a process for implementing model training based on a pre-training method provided in an embodiment of the present application;

[0033] Figure 7B A schematic diagram of an implementation architecture of a model training method provided in an embodiment of the present application;

[0034] Figure 8 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0035] Figure 9 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0036] Figure 10 A hardware entity diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions of this application are further elaborated in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0038] In the following description, references to "some embodiments" describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or sequence of "first / second / third" may be interchanged where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing this application only and are not intended to limit this application.

[0040] To address the high labeling costs associated with training target detection models in related technologies, self-supervised training algorithms can be used to leverage unlabeled data to improve target detection model performance. However, these algorithms are primarily used for image classification tasks, treating the entire image as a whole. They are not suitable for prediction-intensive tasks like target detection. Furthermore, these algorithms typically only pre-train parameters for a portion of the target detection model's network, such as the backbone network. Consequently, they offer limited performance improvements to the overall target detection model.

[0041] The present invention provides a model training method that can be executed by a processor of a computer device. The computer device may include a server, laptop, tablet, desktop computer, smart TV, set-top box, mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or other device with data processing capabilities. Figure 1 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes the following steps S101 to S104:

[0042] Step S101 : Acquire a first augmented image and a second augmented image obtained by respectively performing augmentation processing on a first image sample.

[0043] Here, the first image sample may be any suitable image containing at least one object. The object contained in the first image sample may be determined based on the actual application scenario, and may include, for example, but not limited to, at least one of a person, a human body part, an animal, an animal limb, a plant, a flower, a leaf, a stone, a cloud, a fence, and the like.

[0044] The augmentation processing performed on the first image sample may include, but is not limited to, at least one of random scaling, random cropping, random flipping, random resizing, color dithering, grayscale processing, Gaussian blurring, and random erasing. The first augmented image and the second augmented image may be obtained by performing different augmentation processing on the same first image sample, or may be obtained by performing the same augmentation processing on the same first image sample. During implementation, those skilled in the art may perform appropriate augmentation processing on the first image sample according to actual circumstances to obtain the first augmented image and the second augmented image, and this embodiment of the present application is not limiting.

[0045] In step S102, target detection is performed on the first augmented image using the first model to be trained to obtain at least one first detection result including a first predicted object sequence, and target detection is performed on the second augmented image using the second model to obtain at least one second detection result including a second predicted object sequence.

[0046] Here, the first model can be any suitable model for target detection based on sequence characteristics, such as a vision transformer (ViT), a transformer-based target detection model (DETR), a deformable DETR, etc. The first model can convert the target detection problem into a prediction problem of a feature sequence set, so that it can output at least one first detection result including a first prediction object sequence. The first prediction object sequence can be obtained after the first model performs sequence encoding and sequence decoding on the first augmented image. Each first prediction object sequence can represent a predicted object in the first image sample. During implementation, those skilled in the art can use any suitable sequence encoding method and sequence decoding method to process the first augmented image according to actual conditions to obtain at least one first prediction object sequence, which is not limited in the embodiments of the present application.

[0047] In some embodiments, the first model may be a deformable DETR. The first predicted object sequence in the first detection result may be a predicted object sequence output by a decoder in a transformer, or may be a mapped predicted object sequence obtained by performing a mapping process such as dimensionality transformation on the predicted object sequence output by the decoder in the transformer.

[0048] In some embodiments, the first detection result may include a first predicted object sequence, a first object region corresponding to the first predicted object sequence, and a first object category. The first predicted object sequence may represent a predicted object, and the first object region and the first object category corresponding to the first predicted object sequence may represent a predicted location region and a predicted category of the predicted object, respectively.

[0049] The second model can have the same network structure as the first model, or a different network structure, which is not limited here. The process of performing target detection on the second augmented image using the second model corresponds to the process of performing target detection on the first augmented image using the first model. During implementation, reference can be made to the process of performing target detection on the first augmented image using the first model. The second predicted object sequence can be obtained by sequentially encoding and decoding the second augmented image using the second model. Each second predicted object sequence can represent a predicted object in the first image sample.

[0050] In some embodiments, when the third model is a Transformer-based target detection model, the second predicted object sequence in the second detection result can be a predicted object sequence output by the decoder in the Transformer, or it can be a mapped predicted object sequence obtained after performing mapping processing such as dimensionality transformation on the predicted object sequence output by the decoder in the Transformer.

[0051] In some embodiments, the second detection result may include a second predicted object sequence, a second object region corresponding to the second predicted object sequence, and a second object category. The second predicted object sequence may represent a predicted object, and the second object region and second object category corresponding to the second predicted object sequence may represent a predicted location region and a predicted category of the predicted object, respectively.

[0052] Step S103 : Match each of the first prediction object sequences and each of the second prediction object sequences to obtain at least one pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship.

[0053] Here, the first predicted object sequence and the second predicted object sequence having a target matching relationship may represent the same predicted object in the first image sample. During implementation, those skilled in the art may use any suitable matching method to match each first predicted object sequence with each second predicted object sequence based on actual circumstances, and this is not limited here.

[0054] In some embodiments, the output timing of each first prediction object sequence and the output timing of each second prediction object sequence can be determined, and the first prediction object sequence and the second prediction object sequence with the same output timing are determined as the first prediction object sequence and the second prediction object sequence with a target matching relationship, so that at least one pair of first prediction object sequences and second prediction object sequences with a target matching relationship can be obtained.

[0055] In some embodiments, a bipartite graph matching method can be used to match each first prediction object sequence and each second prediction object sequence to obtain at least one pair of first prediction object sequences and second prediction object sequences having a target matching relationship. During implementation, the matching loss adopted in the bipartite graph matching process can be calculated in any suitable manner, which is not limited here. For example, the matching loss adopted in the bipartite graph matching process can be determined based on at least one of the following: the similarity between each pair of first prediction object sequences and each second prediction object sequence that match each other, the intersection-over-union ratio between the first object region and the second object region that each pair of first prediction object sequences and each second prediction object sequence that match each other, the focus loss between the first object category and the second object category that each pair of first prediction object sequences and each second prediction object sequence that match each other, etc.

[0056] Step S104 : Based on each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship, the model parameters of the first model are updated at least once to obtain the trained first model.

[0057] Here, in some embodiments, whether the model parameters of the first model need to be updated can be determined based on each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship. If the model parameters of the first model need to be updated, the model parameters of the first model are updated using a suitable parameter updating algorithm, and after the update, each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship is re-determined to determine whether the model parameters of the first model need to be further updated based on each re-determined pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship. If it is determined that the model parameters of the first model do not need to be further updated, the finally updated first model is determined as the trained first model.

[0058] For example, a target loss value can be determined based on each pair of a first prediction object sequence and a second prediction object sequence having a target matching relationship, and when the target loss value does not meet the preset conditions, the model parameters of the first model are updated; when the target loss value meets the preset conditions or the number of times the model parameters of the first model are updated reaches a set threshold, the updating of the model parameters of the first model is stopped, and the final updated first model is determined as the trained first model.

[0059] In an embodiment of the present application, a first augmented image and a second augmented image are obtained after augmenting a first image sample respectively; a first model to be trained is used to perform target detection on the first augmented image to obtain at least one first detection result including a first prediction object sequence, and a second model is used to perform target detection on the second augmented image to obtain at least one second detection result including a second prediction object sequence; each first prediction object sequence and each second prediction object sequence are matched to obtain at least one pair of first prediction object sequences and second prediction object sequences having a target matching relationship; based on each pair of first prediction object sequences and second prediction object sequences having a target matching relationship, the model parameters of the first model are updated at least once to obtain the trained first model. In this way, by maintaining the consistency between the first prediction object sequence and the second prediction object sequence obtained after the first model and the second model process the first augmented image and the second augmented image of the same image sample respectively, a sequence-level self-supervised training process of the target detection model can be realized, and the overall network structure of the target detection model can be trained, thereby effectively improving the performance of the entire target detection model and reducing the labeling cost during the target detection model training process.

[0060] In some embodiments, the first model includes a feature extraction network and a converter network; the method of performing object detection on the first augmented image using the first model to be trained in step S102 to obtain at least one first detection result including a first predicted object sequence includes the following steps S111 to S112:

[0061] Step S111: Using the feature extraction network of the first model, perform feature extraction on the first augmented image to obtain image feature information.

[0062] Here, the feature extraction network can be any suitable network capable of extracting image features, such as a convolutional neural network, a recurrent neural network, a transformer-based feature extraction network, etc. During implementation, those skilled in the art can adopt a suitable feature extraction network in the first model to obtain image feature information based on actual conditions, and this is not limited here.

[0063] Step S112: Using the transformer network of the first model, perform prediction processing on the image feature information to obtain at least one first prediction object sequence.

[0064] Here, the transformer network may include an encoder network and a decoder network. During implementation, those skilled in the art may use an appropriate transformer network in the first model to perform prediction processing on the image feature information according to actual conditions, and this is not limited here.

[0065] In some embodiments, the image feature information can be position-encoded and then input into an encoder network to obtain at least one encoded feature sequence after the encoder network performs feature encoding processing on the position-encoded image feature information; using the decoder network, each encoded feature sequence is identified to obtain context identification information corresponding to at least one predicted object, and each encoded feature sequence is feature decoded according to each context identification information to obtain at least one first predicted object sequence.

[0066] In the above embodiment, the first model includes a feature extraction network and a converter network. In this way, based on the sequence characteristics of the converter network, a sequence-level self-supervised training process of the target detection model based on the converter network can be realized, and the overall network structure of the target detection model based on the converter network can be trained, thereby effectively improving the performance of the entire target detection model and reducing the labeling cost during the target detection model training process.

[0067] In some embodiments, the first model further includes a first feedforward neural network; the above step S112 may include the following steps S121 to S122:

[0068] Step S121, using the transformer network of the first model to perform prediction processing on the image feature information to obtain at least one feature sequence;

[0069] Step S122: Map each of the feature sequences to a target dimension using the first feedforward neural network to obtain at least one first prediction object sequence.

[0070] Here, the first feedforward neural network can be any suitable feedforward neural network that can map the feature sequence to the target dimension, and is not limited here.

[0071] The target dimension may be pre-set. During implementation, those skilled in the art may set an appropriate target dimension based on the actual business scenario.

[0072] For example, the feature sequence output by the converter network is a 256-dimensional feature, and the 256-dimensional feature sequence can be mapped into a 512-dimensional first prediction object sequence through the first feedforward neural network.

[0073] In the above embodiment, the feature sequence output by the transformer network is mapped to a target dimension via a first feedforward neural network to obtain a second prediction target sequence. This allows the detection performance of the first model to be improved by presetting an appropriate target dimension. For example, a higher target dimension can improve the detection accuracy of the first model, while a lower target dimension can improve the detection efficiency of the first model.

[0074] In some embodiments, the first detection result further includes a first object region and a first object category, and the first model further includes a second feedforward neural network and a third feedforward neural network; and the step S102 of performing object detection on the first augmented image using the first model to be trained to obtain at least one first detection result including a first predicted object sequence further includes:

[0075] Step S131 : For each of the feature sequences, use the second feedforward neural network to perform region prediction on the feature sequence to obtain a first object region, and use the third feedforward neural network to perform category prediction on the feature sequence to obtain a first object category.

[0076] Here, the second feedforward neural network can be any suitable feedforward neural network capable of performing region prediction, and is not limited here. In some embodiments, the second feedforward neural network can be used to predict the location region of the predicted object represented by the feature sequence in the first augmented image, and the obtained first object region can be a detection box of the predicted object.

[0077] The third feedforward neural network can be any suitable feedforward neural network capable of performing category prediction, and is not limited herein. In some embodiments, the third feedforward neural network can be used to predict the object category of the prediction object represented by the feature sequence to obtain the first object category. In implementation, the number of outputs of the third feedforward neural network can be determined based on the number of object categories required to be detected in an actual business scenario, and is not limited herein.

[0078] In some embodiments, the second model and the first model have the same network structure. In implementation, the process of performing target detection on the second augmented image using the second model can refer to the process of performing target detection on the first augmented image using the first model.

[0079] In some embodiments, the above step S101 may include the following steps S141 to S142:

[0080] Step S141, performing a first image augmentation process on the first image sample to obtain a first augmented image;

[0081] Step S142: performing a second image augmentation process on the first image sample to obtain a second augmented image.

[0082] During implementation, the first image augmentation processing and the second image augmentation processing may adopt the same augmentation processing method or different augmentation processing methods, which is not limited here.

[0083] In some embodiments, the first image augmentation processing includes at least one of the following: color dithering, grayscale processing, Gaussian blur, and random erasing; the second image augmentation processing includes at least one of the following: random scaling, random cropping, random flipping, and random resizing.

[0084] In the above embodiment, the first image sample and the second image sample are respectively subjected to the first image augmentation processing and the second image augmentation processing to obtain the first augmented image and the second augmented image. Compared with the image disturbance caused by the random scaling, random cropping, random flipping and random resizing included in the second image augmentation processing, the image disturbance caused by the color jittering, grayscale processing, Gaussian blur and random erasing included in the first augmentation processing is stronger, so that the target detection difficulty of the first model can be higher than that of the second model, thereby improving the learning ability of the trained first model and reducing the situation where the model collapse is caused by the same learning ability of the first model and the second model.

[0085] An embodiment of the present application provides a model training method, which can be executed by a processor of a computer device. Figure 2 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes the following steps S201 to S206:

[0086] Step S201 : Acquire a first augmented image and a second augmented image obtained by respectively performing augmentation processing on a first image sample.

[0087] In step S202, target detection is performed on the first augmented image using the first model to be trained to obtain at least one first detection result including a first predicted object sequence, and target detection is performed on the second augmented image using the second model to obtain at least one second detection result including a second predicted object sequence.

[0088] Step S203 : Match each of the first prediction object sequences and each of the second prediction object sequences to obtain at least one pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship.

[0089] Here, the above steps S201 to S203 correspond to the above steps S101 to S103 respectively, and the implementation of the above steps S101 to S103 can be referred to.

[0090] Step S204 : determining a target loss value based on the similarity between each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship.

[0091] Here, any suitable similarity loss function can be used to determine the similarity loss between each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship, and a target loss value can be determined based on each similarity loss. The similarity loss function may include, but is not limited to, at least one of an absolute value loss function, a minimum square error loss function, a cosine loss function, a BYOL (Bootstrap Your Own Latent) algorithm, a Momentum Contrastive (MOCO) algorithm, and the like.

[0092] Step S205: When the target loss value does not meet the preset conditions, the model parameters of the first model are updated to obtain an updated first model.

[0093] Here, the preset conditions may include but are not limited to the target loss value being less than a set loss value threshold, the target loss value changing and converging, etc. During implementation, the preset conditions may be set according to actual conditions and are not limited here.

[0094] The method for updating the model parameters of the first model may be determined according to actual conditions, and may include but is not limited to at least one of the gradient descent method, the momentum update method, the Newton momentum method, etc., which is not limited here.

[0095] Step S206: Determine the trained first model based on the updated first model.

[0096] Here, in some implementations, the updated first model may be determined as the trained first model.

[0097] In some implementations, the updated first model may be further updated, and the final updated first model may be determined as the trained first model.

[0098] In an embodiment of the present application, a target loss value is determined based on the similarity between each pair of first prediction object sequences and second prediction object sequences having a target matching relationship. If the target loss value does not meet a preset condition, the model parameters of the first model are updated to obtain an updated first model. The trained first model is determined based on the updated first model. In this way, the model parameters of the first model can be updated at least once if the target loss value does not meet the preset condition. Since the target loss value is determined based on the similarity between each pair of first prediction object sequences and second prediction object sequences having a target matching relationship, the consistency of the prediction object sequences obtained after the trained first model and the second model process different augmented images of the same image sample can be improved, thereby further improving the performance of the trained target detection model.

[0099] In some embodiments, the above step S205 may include the following step S211:

[0100] Step S211: When the target loss value does not meet the preset conditions, the model parameters of the first model and the model parameters of the second model are updated respectively to obtain an updated first model and an updated second model.

[0101] Here, when the target loss value does not meet the preset conditions, the model parameters of the first model and the model parameters of the second model can be updated to achieve comparative learning of the first model and the second model.

[0102] The method for updating the model parameters of the second model can be determined based on actual conditions and may include, but is not limited to, at least one of a gradient descent method, a momentum update method, and a Newton momentum method, etc., and is not limited here. In implementation, the model parameter updating methods for the first model and the second model can be the same or different, and are not limited here.

[0103] The above step S206 may include the following step S212:

[0104] Step S212: Determine the trained first model based on the updated first model and the updated second model.

[0105] In some embodiments, a new target loss value may be determined based on the updated first model and the updated second model, and whether the updated first model should be further updated is determined by determining whether the new target loss value satisfies a preset condition. If the new target loss value satisfies the preset condition, it may be determined that the updated first model should not be further updated, and the updated first model may be determined as the trained first model. If the new target loss value does not satisfy the preset condition, the updated first model may be further updated, and the final updated first model may be determined as the trained first model.

[0106] In the above embodiment, in the process of updating the model parameters of the first model, the model parameters of the second model are also updated, so that the learning capabilities of the first model and the second model can enhance each other, thereby further improving the performance of the trained target detection model.

[0107] In some embodiments, the above step S211 may include the following steps S221 to S222:

[0108] Step S221 : Based on the current model parameters of the first model, momentum update is performed on the model parameters of the second model to obtain an updated second model.

[0109] Here, those skilled in the art may adopt any appropriate momentum update method to perform momentum update on the model parameters of the second model based on the current model parameters of the first model according to actual conditions during implementation, and the embodiments of the present application are not limited thereto.

[0110] In some embodiments, the current model parameters of the first model and the current model parameters of the second model can be weighted and summed based on the set weights to obtain the updated second model. For example, the momentum update of the model parameters of the second model can be performed using the following formula 1:

[0111] Θ m+1 =k*Θ m +(1-k)*Θ o (1);

[0112] Among them, Θ m and Θ o are the current model parameters of the second model and the current model parameters of the first model, Θ m+1 is the updated second model, and k is the set momentum coefficient. In some embodiments, k can be a value greater than or equal to 0.9 and less than 1, for example, k is 0.995.

[0113] Step S222: Update the current model parameters of the first model using a gradient update method to obtain an updated first model.

[0114] Here, any suitable gradient update algorithm can be used to update the current model parameters of the first model, and the embodiments of the present application are not limited thereto. For example, the gradient update algorithm may include but is not limited to at least one of batch gradient descent, stochastic gradient descent, mini-batch gradient descent, etc.

[0115] In the above embodiment, based on the current model parameters of the first model, the model parameters of the second model are updated with momentum to obtain an updated second model. Furthermore, the current model parameters of the first model are updated using a gradient update to obtain an updated first model. This allows the first and second models to be updated at different rates, reducing model collapse and further improving the performance of the trained object detection model.

[0116] In some embodiments, the above step S212 may include the following steps S231 to S235:

[0117] In step S231 , a first augmented image and a second augmented image obtained by performing augmentation processing on the next first image sample are determined as a current first augmented image and a current second augmented image respectively.

[0118] Here, the next first image sample may be the same image as the current first image sample, or may be an image different from the current first image sample.

[0119] Step S232: Use the currently updated first model to perform target detection on the current first augmented image to obtain at least one first detection result including a first predicted object sequence, and use the currently updated second model to perform target detection on the current second augmented image to obtain at least one second detection result including a second predicted object sequence.

[0120] Step S233 : Match each of the first prediction object sequences and each of the second prediction object sequences to obtain at least one pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship.

[0121] Step S234 : determining a current target loss value based on the similarity between each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship.

[0122] Here, the above steps S231 to S234 correspond to the above steps S201 to S204 respectively, and the implementation of the above steps S201 to S204 can be referred to.

[0123] Step S235: When the current target loss value satisfies the preset condition or the number of times the model parameters of the first model are updated reaches a threshold number, the currently updated first model is determined as the trained first model.

[0124] Here, the number threshold may be set in advance by the user according to actual conditions, or may be a default value.

[0125] In some embodiments, the above step S212 may further include the following steps S241 to S242:

[0126] Step S241, when the current target loss value does not meet the preset conditions, the model parameters of the first model and the model parameters of the second model are updated for the next time to obtain the first model after the next update and the second model after the next update.

[0127] Step S242: Determine the first model after training based on the first model after the next update and the second model after the next update.

[0128] In the above embodiment, when the target loss value does not meet the preset conditions, the model parameters of the first model and the model parameters of the second model can be updated for the next time, and the trained first model can be determined based on the first model after the next update and the second model after the next update, so that the performance of the trained first model can be further improved through continuous iterative updates.

[0129] An embodiment of the present application provides a model training method, which can be executed by a processor of a computer device. Figure 3 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the method includes the following steps S301 to S310:

[0130] Step S301 : Acquire a first augmented image and a second augmented image obtained by respectively performing augmentation processing on a first image sample.

[0131] Step S302: Using the first model to be trained, perform target detection on the first augmented image to obtain at least one first detection result, and using the second model, perform target detection on the second augmented image to obtain at least one second detection result including a second predicted object sequence; the first detection result includes a first predicted object sequence and a first object region and a first object category corresponding to the first predicted object sequence.

[0132] Step S303 : Match each of the first prediction object sequences and each of the second prediction object sequences to obtain at least one pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship.

[0133] Here, the above steps S301 to S303 correspond to the above steps S101 to S103 respectively, and the implementation of the above steps S101 to S103 may be referred to.

[0134] Step S304: Acquire at least one candidate object in the first image sample, each candidate object having a candidate object region and a candidate object category.

[0135] Here, the at least one candidate object in the first image sample may be randomly determined or obtained by performing object detection on the first image sample using any suitable unsupervised algorithm, without limitation. For example, the unsupervised detection algorithm may include, but is not limited to, at least one of a sliding window method, a candidate region algorithm, a selective search algorithm, and the like.

[0136] The candidate object region of the candidate object is the predicted position region of the candidate object in the first image sample, and the candidate object category of the candidate object is the predicted type of the candidate object. The candidate object category of the candidate object can be used as a pseudo label for the candidate object region of the candidate object.

[0137] In some embodiments, step S304 may include: performing target detection on the first image sample in an unsupervised manner to obtain at least one predicted object region and a pseudo-label for each predicted object region; the pseudo-label for each predicted object region is used to characterize the predicted object category of the predicted object region; and for each predicted object region, using the predicted object region as a candidate object region and the pseudo-label for the predicted object region as a candidate object category to obtain a candidate object. Any suitable unsupervised algorithm may be used to implement unsupervised target detection on the first image sample. This further reduces the labeling cost during target detection model training.

[0138] Step S305, based on the first object area and the first object category corresponding to each first predicted object sequence, and the candidate object area and the candidate object category of each candidate object, each first predicted object sequence and each candidate object are matched to obtain at least one pair of first predicted object sequence and candidate object having a target matching relationship.

[0139] Here, the first predicted object sequence and the candidate object having a target matching relationship may represent the same predicted object in the first image sample. During implementation, those skilled in the art may use any suitable matching method to match each first predicted object sequence with each candidate object according to actual circumstances, and this is not limited here.

[0140] In some embodiments, a bipartite graph matching method can be used to match each first prediction object sequence and each candidate object to obtain at least one pair of first prediction object sequences and candidate objects having a target matching relationship. During implementation, the matching loss used in the bipartite graph matching process can be calculated in any suitable manner, which is not limited here. For example, the matching loss used in the bipartite graph matching process can be determined based on at least one of the following: the intersection-over-union ratio between the first object region and the candidate object region corresponding to each pair of mutually matching first prediction object sequences and candidate objects, the focus loss between the first object category and the candidate object category corresponding to each pair of mutually matching first prediction object sequences and candidate objects, etc.

[0141] Step S306 : determining a first loss value based on the similarity between each pair of the first predicted object sequence and the second predicted object sequence having a target matching relationship.

[0142] Here, any appropriate similarity loss function may be used to determine the first loss value between each pair of the first predicted object sequence and the second predicted object sequence having a target matching relationship, and this embodiment of the present application is not limited thereto.

[0143] In some embodiments, the similarity loss between each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship can be determined, and each similarity loss can be accumulated to obtain a first loss value. For example, the first loss value can be determined using the following formula 2:

[0144]

[0145] Where N is the number of pairs of the first prediction object sequence and the second prediction object sequence that have a target matching relationship, N is a positive integer, s i is the first prediction object sequence, is with s i A second prediction object sequence having a target matching relationship. is the similarity loss algorithm, is the first loss value determined.

[0146] Step S307 : determining a second loss value based on each pair of the first predicted object sequence and the candidate object having a target matching relationship.

[0147] Here, any suitable loss function can be used to determine the second loss value between each pair of the first predicted object sequence and the candidate object having a target matching relationship, and this embodiment of the application is not limited to this. The loss function may include but is not limited to at least one of a similarity loss function, a focal loss function, an intersection-over-union loss function, a generalized intersection-over-union loss function, and the like.

[0148] Step S308: Determine a target loss value based on the first loss value and the second loss value.

[0149] Here, the target loss value can be determined based on the first loss value and the second loss value in an appropriate manner according to the actual situation, and the embodiments of the present application are not limited thereto. For example, the sum of the first loss value and the second loss value can be determined as the target loss value, or the average of the first loss value and the second loss value can be determined as the target loss value, or the target loss value can be obtained by weighted summing the first loss value and the second loss value using different weights.

[0150] Step S309: When the target loss value does not meet the preset conditions, the model parameters of the first model are updated to obtain an updated first model.

[0151] Step S310: Determine the trained first model based on the updated first model.

[0152] Here, the above steps S309 to S310 correspond to the above steps S205 to S206 respectively, and the implementation of the above steps S205 to S206 can be referred to.

[0153] In an embodiment of the present application, a first loss value is determined based on the similarity between each pair of the first predicted object sequence and the second predicted object sequence having a target matching relationship, a second loss value is determined based on each pair of the first predicted object sequence and the candidate object having a target matching relationship, and a target loss value is determined based on the first loss value and the second loss value. Since the candidate object category of each candidate object can be used as a pseudo-label for the candidate object area of ​​the candidate object, the second loss value determined based on each pair of the first predicted object sequence and the candidate object having a target matching relationship can provide objective supervision of the prediction object positioning capability of the first model, thereby improving the object positioning capability of the trained first model, and further improving the detection accuracy of the trained first model.

[0154] In some embodiments, the above step S307 may include the following steps S321 to S322:

[0155] In step S321, for each pair of a first predicted object sequence and a candidate object having a target matching relationship, a first sub-loss value is determined based on the first object area corresponding to the first predicted object sequence and the candidate object area of ​​the candidate object, and a second sub-loss value is determined based on the first object category corresponding to the first predicted object sequence and the candidate object category of the candidate object.

[0156] Here, any suitable loss function may be used to determine the first sub-loss value between the first object region and the candidate object region, and the second sub-loss value between the first object category and the candidate object category, and the embodiments of the present application are not limited thereto. For example, an intersection-over-union loss function, a generalized intersection-over-union loss function, etc. may be used to determine the first sub-loss value between the first object region and the candidate object region, and a focal loss function may be used to determine the second sub-loss value between the first object category and the candidate object category.

[0157] Step S322: Determine a second loss value based on each of the first sub-loss values ​​and each of the second sub-loss values.

[0158] Here, the second loss value can be determined based on the first sub-loss value and the second sub-loss value in an appropriate manner according to the actual situation, and the embodiments of the present application are not limited thereto. For example, the sum of the first sub-loss value and the second sub-loss value can be determined as the second loss value, or the average of the first sub-loss value and the second sub-loss value can be determined as the second loss value, or the first sub-loss value and the second sub-loss value can be weighted and summed using different weights to obtain the second loss value.

[0159] In some embodiments, a target loss value may be obtained by weighted summing each of the first sub-loss values, each of the second sub-loss values, and each similarity loss between the first prediction object sequence and the second prediction object sequence having a target matching relationship. For example, the target loss value may be determined using the following formula 3:

[0160]

[0161] Wherein, N is the number of pairs of first prediction object sequence and second prediction object sequence with target matching relationship, and N is a positive integer; i is the first prediction object sequence, is with s i A second prediction object sequence with a target matching relationship, is the first prediction object sequence s i With the second prediction object sequence Similarity loss between i is the first prediction object sequence s i The corresponding first object category, c σ(i) is with s i candidate object categories of candidate objects with target matching relationships, is the first object category c calculated using the intersection loss function i and candidate object categories The first sub-loss value between ; Indicates c i When it is empty, it is 0, c i If not empty, take 1; b i is the first prediction object sequence s i The corresponding first object area, is with s i Candidate object regions of candidate objects with target matching relationships, is the first object region b calculated using the generalized intersection-over-union loss function i and candidate object areas The second sub-loss value between λ f ,λ b and λ eare the weights of the first sub-loss value, the second sub-loss value, and the similarity loss respectively; The first prediction object sequence y and the second prediction object sequence The target loss value between .

[0162] In the above embodiment, for each pair of a first predicted object sequence and a candidate object having a target matching relationship, a first sub-loss value is determined based on the first object region corresponding to the first predicted object sequence and the candidate object region of the candidate object, and a second sub-loss value is determined based on the first object category corresponding to the first predicted object sequence and the candidate object category of the candidate object; and a second loss value is determined based on each first sub-loss value and each second sub-loss value. In this way, the object region regression in the first model detection and the self-supervised representation learning process of the object category can be simultaneously implemented, thereby further improving the detection accuracy of the trained first model.

[0163] An embodiment of the present application provides a model training method, which can be executed by a processor of a computer device. Figure 4 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method includes the following steps S401 to S404:

[0164] Step S401 : Acquire a first augmented image and a second augmented image obtained by respectively performing augmentation processing on a first image sample.

[0165] Step S402: Using the first model to be trained, perform target detection on the first augmented image to obtain at least one first detection result, and using the second model, perform target detection on the second augmented image to obtain at least one second detection result; the first detection result includes a first predicted object sequence and a first object area and a first object category corresponding to the first predicted object sequence, and the second detection result includes a second predicted object sequence and a second object area and a second object category corresponding to the second predicted object sequence.

[0166] Here, the above steps S401 to S402 correspond to the above steps S101 to S102 respectively, and the implementation of the above steps S101 to S102 may be referred to.

[0167] The second object region may be obtained by predicting a location region of the predicted object represented by the second predicted object sequence in the second augmented image, and may be a detection box of the predicted object. The second object category may be obtained by predicting an object category of the predicted object represented by the second predicted object sequence.

[0168] Step S403, based on the first object area and the first object category corresponding to each first predicted object sequence, and the second object area and the second object category corresponding to each second predicted object sequence, each first predicted object sequence and each second predicted object sequence are subjected to bipartite graph matching to obtain at least one pair of first predicted object sequence and second predicted object sequence having a target matching relationship.

[0169] Here, any suitable bipartite graph matching algorithm can be used to match each first prediction object sequence and each second prediction object sequence to obtain at least one pair of first prediction object sequences and second prediction object sequences having a target matching relationship. For example, the bipartite graph matching algorithm used may include but is not limited to at least one of the Hungarian matching algorithm, the maximum flow matching algorithm, etc. During implementation, the matching loss used in the bipartite graph matching process can be calculated in any suitable manner, which is not limited here. For example, the matching loss used in the bipartite graph matching process can be determined based on at least one of the following: the similarity between each pair of first prediction object sequences and second prediction object sequences that match each other, the intersection-over-union ratio between the first object region and the second object region corresponding to each pair of first prediction object sequences and second prediction object sequences that match each other, the focus loss between the first object category and the second object category corresponding to each pair of first prediction object sequences and second prediction object sequences that match each other, etc.

[0170] Step S404 : Based on each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship, the model parameters of the first model are updated at least once to obtain the trained first model.

[0171] Here, the above step S404 corresponds to the above step S104, and the implementation method of the above step S104 can be referred to during implementation.

[0172] In some embodiments, the above step S403 may include the following steps S411 to S413:

[0173] Step S411: determining at least one candidate sequence pair set based on each first prediction object sequence and each second prediction object sequence; each candidate sequence pair set includes at least one pair of a first prediction object sequence and a second prediction object sequence having a candidate matching relationship.

[0174] Here, any suitable method can be used to perform a one-to-one match between each first prediction target sequence and each second prediction target sequence to obtain at least one candidate sequence pair set, which is not limited in the present embodiment. For example, each first prediction target sequence and each second prediction target sequence can be randomly matched at least once to obtain at least one candidate sequence pair set.

[0175] Step S412: For each candidate sequence pair set, the matching loss of the candidate sequence pair set is determined based on the first object region and the first object category corresponding to the first prediction object sequence in each pair of the first prediction object sequence and the second prediction object sequence having a candidate matching relationship in the candidate sequence pair set, and the second object region and the second object category corresponding to the second prediction object sequence.

[0176] Here, the matching loss of the candidate sequence pair set may be calculated in any appropriate manner.

[0177] In some embodiments, the matching loss of the candidate sequence pair set can be determined based on the intersection-over-union ratio between the first object region and the second object region respectively corresponding to each pair of first prediction object sequences and second prediction object sequences that match each other in the candidate sequence pair set, and the focal loss between the first object category and the second object category respectively corresponding to each pair of first prediction object sequences and each second prediction object sequence that match each other.

[0178] For example, the matching loss of the candidate sequence pair set can be calculated as shown in the following formula 4:

[0179]

[0180] Wherein, N is the number of pairs of the first prediction object sequence and the second prediction object sequence having a target matching relationship, and N is a positive integer; represents the Hungarian matching loss, Indicates at least one pair of a first prediction target sequence and a second prediction target sequence that match each other in the candidate sequence pair set; is the second object category corresponding to the second predicted object sequence in the i-th pair of the first predicted object sequence and the second predicted object sequence having a target matching relationship, The first object category of the first prediction object sequence having a target matching relationship with the second prediction object sequence is confidence level; exist If it is empty, it is 0. If not empty, take 1; b i is the first object region corresponding to the first prediction object sequence in the i-th pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship, is a second target region of a second prediction target sequence having a target matching relationship with the first prediction target sequence, is the first object region b calculated using the generalized intersection-over-union loss function i and the second object area The loss value between .

[0181] Step S413 , determining each pair of the first prediction object sequence and the second prediction object sequence having a candidate matching relationship in the candidate sequence pair set with the smallest matching loss in the at least one candidate sequence pair set as one less pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship.

[0182] In an embodiment of the present application, a bipartite graph matching method is used to match each first prediction object sequence and each second prediction object sequence, which can improve the accuracy of the target matching relationship between at least one pair of first prediction object sequences and second prediction object sequences with a target matching relationship, thereby further improving the detection accuracy of the trained first model.

[0183] An embodiment of the present application provides a model training method, which can be executed by a processor of a computer device. Figure 5 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the method includes the following steps S501 to S506:

[0184] Step S501 : Acquire a first augmented image and a second augmented image obtained by respectively performing augmentation processing on a first image sample.

[0185] In step S502, target detection is performed on the first augmented image using the first model to be trained to obtain at least one first detection result including a first predicted object sequence, and target detection is performed on the second augmented image using the second model to obtain at least one second detection result including a second predicted object sequence.

[0186] Step S503 : Match each of the first prediction object sequences and each of the second prediction object sequences to obtain at least one pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship.

[0187] Step S504 : Based on each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship, the model parameters of the first model are updated at least once to obtain the trained first model.

[0188] Here, the above steps S501 to S504 correspond to the above steps S101 to S104 respectively, and the implementation of the above steps S101 to S104 may be referred to.

[0189] Step S505: Determine an initial third model based on the trained first model.

[0190] Here, in some embodiments, the feedforward neural network in the trained first model can be adjusted according to the actual target detection scenario, and the adjusted first model can be determined as the initial third model.

[0191] In some embodiments, the first model includes a feature extraction network, a converter network, and a first feedforward neural network, a second feedforward neural network, and a third feedforward neural network connected to the converter network; the first feedforward neural network, the second feedforward neural network, and the third feedforward neural network are used to output a first predicted object sequence, a first object area corresponding to the first predicted object sequence, and a first object category corresponding to the first predicted object sequence, respectively; the first feedforward neural network in the trained first model can be removed, and the third feedforward neural network in the first model can be adjusted according to the actual target detection scenario, and the adjusted first model is determined as the initial third model.

[0192] Step S506: Based on at least one second image sample, update the model parameters of the third model to obtain the trained third model.

[0193] Here, the second image sample may have labeling information or may not have labeling information. During implementation, those skilled in the art may determine a suitable second image sample according to an actual target detection scenario, which is not limited here.

[0194] In some embodiments, the model parameters of the third model may be fine-tuned based on at least one second image sample to obtain the trained third model.

[0195] In the embodiment of the present application, an initial third model is determined based on the trained first model, and the model parameters of the third model are updated based on at least one second image sample to obtain a trained third model. In this way, the model parameters of the trained first model can be transferred to other object detection models for application in various object detection scenarios, thereby improving the training efficiency of the third model and the detection accuracy of the trained third model.

[0196] An embodiment of the present application provides an image processing method, which can be executed by a processor of a computer device. Figure 6 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the method includes the following steps S601 to S602:

[0197] Step S601, obtaining an image to be processed;

[0198] Step S602: Use the trained fourth model to perform target detection on the image to be processed to obtain a third detection result; wherein, the third model includes at least one of the following: the first model obtained by the model training method described in the above embodiment, and the third model obtained by the model training method described in the above embodiment.

[0199] Here, the image to be processed can be any suitable image to be subjected to target detection. During implementation, those skilled in the art can select a suitable image to be processed according to the actual application scenario, and the embodiments of the present application are not limited thereto.

[0200] In an embodiment of the present application, since the model training method described in the above embodiment can realize the sequence-level self-supervised training process of the target detection model by maintaining the consistency between the first predicted object sequence and the second predicted object sequence obtained after the first model and the second model process the first augmented image and the second augmented image of the same image sample respectively, and the overall network structure of the target detection model can be trained, thereby effectively improving the performance of the entire target detection model, therefore, based on the first model obtained by the model training method described in the above embodiment and / or the third model for target detection on the processed image, the accuracy of target detection can be improved.

[0201] An embodiment of the present application provides a pre-training method for a self-supervised target detection model based on Transformer sequence consistency. This method can use unlabeled data to train the overall network structure of the target detection model, and based on the sequence characteristics of the Transformer, can simultaneously realize the object region regression and object category self-supervised representation learning process in the target detection model. Figure 7A A schematic diagram of a process for implementing model training based on the pre-training method provided in an embodiment of the present application is shown in FIG. Figure 7A As shown, the method may include the following steps S701 to S703:

[0202] Step S701 : obtaining at least one candidate object in a first image sample in an unsupervised manner, wherein each candidate object has a candidate object region and a candidate object category.

[0203] During implementation, any suitable unsupervised detection algorithm may be used to detect the target object in the first image sample to obtain at least one candidate object. For example, a selective search algorithm may be used to unsupervisedly obtain at least one candidate object with a high recall rate from the first image sample.

[0204] Step S702: Pre-train the first model using a pre-training method for a self-supervised object detection model based on Transformer sequence consistency.

[0205] In some embodiments, the Figure 7B The model training architecture shown implements the pre-training method of the self-supervised target detection model based on Transformer sequence consistency, such as Figure 7BAs shown, the model training architecture includes a first model 10 and a second model 20, wherein the network structures of the first model 10 and the second model 20 are the same, both of which include a convolutional neural network (CNN) 11 or 21, a Transformer encoder 12 or 22, a Transformer decoder 13 or 23, and a feed-forward neural network (FFN) 14 or 24, and the feed-forward neural network may include a first feed-forward neural network, a second feed-forward neural network and a third feed-forward neural network; during the model training process, the inputs of the first model 10 and the second model 20 are respectively the first augmented image and the second augmented image obtained after the first image sample 30 is augmented, wherein the perturbation of the first augmented image input in the first model 10 includes more color layer perturbations. The processes of the first model 10 and the second model 20 performing target detection on the first augmented image and the second augmented image respectively are the same. Taking the process of the first model 10 performing target detection on the first augmented image as an example, after the first augmented image is feature extracted by the convolutional neural network 11, a position code 40 is added to the extracted feature, and the feature with the added position code is processed by the Transformer encoder 12 and the Transformer decoder 13. After being processed by the Transformer encoder 12 and the Transformer decoder 13, at least one feature sequence 31 representing the predicted object can be obtained. The second feedforward neural network and the third feedforward neural network process each feature sequence 31. For each feature sequence 31, the first predicted object sequence Prj1 output by the first feedforward neural network, the first object area Box1 corresponding to the first predicted object sequence output by the second feedforward neural network, and the first object category Cls1 corresponding to the first predicted object sequence output by the third feedforward neural network can be obtained. Accordingly, after processing the second augmented image using the second model 20, the feature sequence 32, the second predicted object sequence Prj2, the second object area Box2 corresponding to the second predicted object sequence, and the second object category Cls2 corresponding to the second predicted object sequence can be obtained.For the output results of the first model 10 and the second model 20, a bipartite graph matching algorithm can be used to match at least one first prediction object sequence Prj1 and at least one second prediction object sequence Prj2 to obtain at least one pair of first prediction object sequences and second prediction object sequences with a target matching relationship (such as the first prediction object sequence corresponding to the first object area Box1-1 and the second prediction object sequence of the second object area Box2-1, the first prediction object sequence corresponding to the first object area Box1-4 and the second prediction object sequence of the second object area Box2-2, the first prediction object sequence corresponding to the first object area Box1-4 and the second prediction object sequence of the second object area Box2-3, the first prediction object sequence corresponding to the first object area Box1-4 and the second prediction object sequence of the second object area Box2-4). Predicted object sequence and second predicted object sequence of second object area Box2-4), then based on at least one pair of first predicted object sequence and second predicted object sequence with target matching relationship, similarity loss can be calculated using absolute value loss function, based on the similarity loss, target loss value can be determined, based on the target loss value, network parameters of the first model 10 and the second model 20 can be updated to ensure the consistency of Transformer feature sequence of augmented image after different augmentation processing of the same image sample; wherein, the network parameters of the first model 10 can be updated by gradient update, and the update of the network parameters of the second model 20 can adopt the design of stopping gradient, and momentum update is performed based on the current network parameters of the first model 10. Among them, the bipartite graph matching algorithm is a set-based matching method, and the input of the bipartite graph matching algorithm is at least one first predicted object sequence and at least one second predicted object sequence output by the first model 10 and the second model 20 respectively, as well as the confidence of the first object region and the first object category corresponding to each first predicted object sequence, and the confidence of the second object region and the second object category corresponding to each second predicted object sequence. Compared with one-to-one sequence matching based on time series, the bipartite graph matching algorithm can find better sequence matching pairs (i.e., the first prediction object sequence and the second prediction object sequence with a target matching relationship), and bring more useful information to the self-supervised learning of the first model, ultimately improving the efficiency and accuracy of self-supervised learning.

[0206] In some embodiments, the target loss value considered in the process of updating the network parameters of the first model 10 and the network parameters of the second model 20 may also include the difference between the first object region corresponding to at least one first predicted object sequence output by the first object detection network and the candidate object region of at least one candidate object, and the difference between the first object category corresponding to each first predicted object sequence and the candidate object category of each candidate object. In implementation, a bipartite graph matching algorithm can be used to match the first object region and the first object category corresponding to each first predicted object sequence, and the candidate object region and the candidate object category of each candidate object. Then, a generalized intersection-over-union function is used to determine the first sub-loss value between the first object region and the candidate object region corresponding to each pair of the first predicted object sequence and the candidate object having a target matching relationship, and a focal loss function is used to determine the second sub-loss value between the first object category and the candidate object category corresponding to each pair of the first predicted object sequence and the candidate object having a target matching relationship. Based on each first sub-loss value, each second sub-loss value, and the similarity loss between each pair of the first predicted object sequence and the second predicted object sequence having a target matching relationship, the target loss value can be determined.

[0207] Step S603: Migrate the pre-trained first model to the target detection task.

[0208] Here, according to the target detection tasks in different target detection scenarios (such as industrial quality inspection, industrial inspection, medical scene detection, autonomous driving, etc.), the first feedforward neural network in the trained first model can be removed, and the number of output categories of the third feedforward neural network in the first model can be adjusted according to the actual target detection task. The adjusted first model is determined as the initial third model, and then the model parameters of the third model are fine-tuned to obtain a third model that can be used for target detection tasks.

[0209] Figure 8 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the model training device 800 includes: a first acquisition module 810, a first detection module 820, a first matching module 830 and a first update module 840, wherein:

[0210] A first acquisition module 810 is configured to acquire a first augmented image and a second augmented image obtained by respectively performing augmentation processing on the first image sample;

[0211] a first detection module 820 configured to perform object detection on the first augmented image using a first model to be trained, obtaining at least one first detection result including a first predicted object sequence, and to perform object detection on the second augmented image using a second model, obtaining at least one second detection result including a second predicted object sequence;

[0212] A first matching module 830 is configured to match each of the first prediction object sequences with each of the second prediction object sequences to obtain at least one pair of first prediction object sequences and second prediction object sequences having a target matching relationship;

[0213] The first updating module 840 is configured to update the model parameters of the first model at least once based on each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship, to obtain the trained first model.

[0214] In some embodiments, the first update module is further used to: determine a target loss value based on the similarity between each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship; when the target loss value does not meet the preset conditions, update the model parameters of the first model to obtain an updated first model; and determine the trained first model based on the updated first model.

[0215] In some embodiments, the first update module is further used to: when the target loss value does not meet the preset conditions, update the model parameters of the first model and the model parameters of the second model respectively to obtain an updated first model and an updated second model; based on the updated first model and the updated second model, determine the trained first model.

[0216] In some embodiments, the first update module is further used to: perform momentum update on the model parameters of the second model based on the current model parameters of the first model to obtain an updated second model; and update the current model parameters of the first model using a gradient update method to obtain an updated first model.

[0217] In some embodiments, the first update module is further used to: determine the first augmented image and the second augmented image obtained after augmenting the next first image sample as the current first augmented image and the current second augmented image respectively; use the currently updated first model to perform target detection on the current first augmented image to obtain at least one first detection result including a first prediction object sequence, and use the currently updated second model to perform target detection on the current second augmented image to obtain at least one second detection result including a second prediction object sequence; match each first prediction object sequence and each second prediction object sequence to obtain at least one pair of first prediction object sequence and second prediction object sequence with a target matching relationship; determine the current target loss value based on the similarity between each pair of first prediction object sequence and second prediction object sequence with a target matching relationship; when the current target loss value meets the preset condition or the number of times the model parameters of the first model are updated reaches a threshold number, determine the currently updated first model as the trained first model.

[0218] In some embodiments, the first update module is further used to: when the current target loss value does not meet the preset conditions, perform the next update on the model parameters of the first model and the model parameters of the second model respectively, to obtain the first model after the next update and the second model after the next update; based on the first model after the next update and the second model after the next update, determine the first model after training.

[0219] In some embodiments, the first detection result also includes a first object area and a first object category corresponding to the first predicted object sequence in the first detection result; the device also includes: a second acquisition module, used to acquire at least one candidate object in the first image sample, each candidate object having a candidate object area and a candidate object category; a second matching module, used to match each first predicted object sequence and each candidate object based on the first object area and first object category corresponding to each first predicted object sequence, and the candidate object area and candidate object category of each candidate object, to obtain at least one pair of first predicted object sequence and candidate object with a target matching relationship; the first update module is also used to: determine a first loss value based on the similarity between each pair of first predicted object sequence and second predicted object sequence with a target matching relationship; determine a second loss value based on each pair of first predicted object sequence and candidate object with a target matching relationship; determine a target loss value based on the first loss value and the second loss value.

[0220] In some embodiments, the first update module is further used to: for each pair of a first predicted object sequence and a candidate object having a target matching relationship, determine a first sub-loss value based on the first object area corresponding to the first predicted object sequence and the candidate object area of ​​the candidate object, and determine a second sub-loss value based on the first object category corresponding to the first predicted object sequence and the candidate object category of the candidate object; determine a second loss value based on each of the first sub-loss values ​​and each of the second sub-loss values.

[0221] In some embodiments, the second acquisition module is further used to: perform target detection on the first image sample in an unsupervised manner to obtain at least one predicted object area and a pseudo-label for each predicted object area; the pseudo-label of each predicted object area is used to characterize the predicted object category of the predicted object area; for each predicted object area, the predicted object area is used as a candidate object area, and the pseudo-label of the predicted object area is used as a candidate object category to obtain a candidate object.

[0222] In some embodiments, the first detection result also includes a first object area and a first object category corresponding to the first predicted object sequence in the first detection result, and the second detection result also includes a second object area and a second object category corresponding to the second predicted object sequence in the second detection result; the first matching module is also used to: based on the first object area and the first object category corresponding to each first predicted object sequence, and the second object area and the second object category corresponding to each second predicted object sequence, perform bipartite graph matching on each first predicted object sequence and each second predicted object sequence to obtain at least one pair of first predicted object sequences and second predicted object sequences having a target matching relationship.

[0223] In some embodiments, the first matching module is further used to: determine at least one candidate sequence pair set based on each first prediction object sequence and each second prediction object sequence; each candidate sequence pair set includes at least one pair of first prediction object sequence and second prediction object sequence with a candidate matching relationship; for each candidate sequence pair set, based on the first object area and first object category corresponding to the first prediction object sequence in each pair of first prediction object sequence and second prediction object sequence with a candidate matching relationship in the candidate sequence pair set, and the second object area and second object category corresponding to the second prediction object sequence, determine the matching loss of the candidate sequence pair set; determine each pair of first prediction object sequence and second prediction object sequence with a candidate matching relationship in the candidate sequence pair set with the smallest matching loss in the at least one candidate sequence pair set as one less pair of first prediction object sequence and second prediction object sequence with a target matching relationship.

[0224] In some embodiments, the first model includes a feature extraction network and a converter network; the first detection module is further used to: use the feature extraction network of the first model to extract features from the first augmented image to obtain image feature information; use the converter network of the first model to perform prediction processing on the image feature information to obtain at least one first prediction object sequence.

[0225] In some embodiments, the first model also includes a first feedforward neural network; the first detection module is further used to: use the converter network of the first model to perform prediction processing on the image feature information to obtain at least one feature sequence; use the first feedforward neural network to map each of the feature sequences to a target dimension to obtain at least one first prediction object sequence.

[0226] In some embodiments, the first detection result also includes a first object region and a first object category, and the first model also includes a second feedforward neural network and a third feedforward neural network; the first detection module is further used to: for each of the feature sequences, use the second feedforward neural network to perform region prediction on the feature sequence to obtain the first object region, and use the third feedforward neural network to perform category prediction on the feature sequence to obtain the first object category.

[0227] In some embodiments, the second model has the same network structure as the first model.

[0228] In some embodiments, the first acquisition module is further configured to: perform a first image augmentation process on the first image sample to obtain a first augmented image; and perform a second image augmentation process on the first image sample to obtain a second augmented image.

[0229] In some embodiments, the first image augmentation processing includes at least one of the following: color dithering, grayscale processing, Gaussian blur, and random erasing; the second image augmentation processing includes at least one of the following: random scaling, random cropping, random flipping, and random resizing.

[0230] In some embodiments, the device also includes: a determination module for determining an initial third model based on the trained first model; and a second update module for updating the model parameters of the third model based on at least one second image sample to obtain the trained third model.

[0231] Figure 9 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Figure 9 As shown, the image processing device 900 includes: a third acquisition module 910 and a second detection module 920, wherein:

[0232] The third acquisition module 910 is used to acquire the image to be processed;

[0233] The second detection module 920 is used to use the trained fourth model to perform target detection on the image to be processed to obtain a third detection result; wherein, the third model includes at least one of the following: the first model obtained by the model training method described in the above embodiment, and the third model obtained by the model training method described in the above embodiment.

[0234] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0235] It should be noted that, in the embodiment of the present application, if the above-mentioned model training method or image processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0236] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps in the above method when executing the program.

[0237] The embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor. The computer-readable storage medium may be transient or non-transient.

[0238] An embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, implements some or all of the steps in the above method. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0239] It should be noted that the descriptions of the above storage medium, computer program product, and device embodiments are similar to the descriptions of the above method embodiments and have similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, computer program product, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0240] It should be noted that Figure 10 A schematic diagram of a hardware entity of a computer device in an embodiment of the present application is shown in FIG. Figure 10 As shown, the hardware entity of the computer device 1000 includes: a processor 1001, a communication interface 1002 and a memory 1003, wherein:

[0241] Processor 1001 generally controls the overall operation of computer device 1000 .

[0242] The communication interface 1002 enables the computer device to communicate with other terminals or servers through a network.

[0243] The memory 1003 is configured to store instructions and applications executable by the processor 1001, and can also cache data to be processed or processed by the processor 1001 and various modules in the computer device 1000 (for example, image data, audio data, voice communication data, and video communication data). This can be implemented using flash memory (FLASH) or random access memory (RAM). Data can be transmitted between the processor 1001, the communication interface 1002, and the memory 1003 via the bus 1004.

[0244] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0245] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0246] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0247] The units described above as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, the functional units in the various embodiments of the present application may all be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0248] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0249] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0250] The above is only an implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Obtaining a first augmented image and a second augmented image obtained by respectively performing augmentation processing on the first image sample; Performing object detection on the first augmented image using a first model to be trained to obtain at least one first detection result including a first predicted object sequence, and performing object detection on the second augmented image using a second model to obtain at least one second detection result including a second predicted object sequence; Matching each of the first prediction object sequences and each of the second prediction object sequences to obtain at least one pair of first prediction object sequences and second prediction object sequences having a target matching relationship; Based on each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship, updating the model parameters of the first model at least once to obtain the trained first model; The first detection result includes a first object region and a first object category corresponding to a first predicted object sequence in the first detection result, and the second detection result includes a second object region and a second object category corresponding to a second predicted object sequence in the second detection result; The matching of each first prediction object sequence and each second prediction object sequence to obtain at least one pair of first prediction object sequence and second prediction object sequence having a target matching relationship includes: Based on the first object area and the first object category corresponding to each first predicted object sequence, and the second object area and the second object category corresponding to each second predicted object sequence, bipartite graph matching is performed on each first predicted object sequence and each second predicted object sequence to obtain at least one pair of first predicted object sequences and second predicted object sequences having a target matching relationship.

2. The method according to claim 1, characterized in that The updating of the model parameters of the first model at least once based on each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship to obtain the trained first model includes: Determining a target loss value based on the similarity between each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship; When the target loss value does not meet the preset conditions, updating the model parameters of the first model to obtain an updated first model; Based on the updated first model, the trained first model is determined.

3. The method according to claim 2, characterized in that When the target loss value does not meet the preset conditions, updating the model parameters of the first model to obtain an updated first model includes: When the target loss value does not meet the preset conditions, updating the model parameters of the first model and the model parameters of the second model respectively to obtain an updated first model and an updated second model; The determining the trained first model based on the updated first model includes: Based on the updated first model and the updated second model, the trained first model is determined.

4. The method according to claim 3, characterized in that The updating of the model parameters of the first model and the model parameters of the second model respectively to obtain an updated first model and an updated second model includes: Based on the current model parameters of the first model, momentum update the model parameters of the second model to obtain an updated second model; The current model parameters of the first model are updated using a gradient update method to obtain an updated first model.

5. The method according to claim 3 or 4, characterized in that The step of determining the trained first model based on the updated first model and the updated second model includes: Determining a first augmented image and a second augmented image obtained by performing augmentation processing on the next first image sample as the current first augmented image and the current second augmented image respectively; Performing object detection on the current first augmented image using the currently updated first model to obtain at least one first detection result including a first predicted object sequence, and performing object detection on the current second augmented image using the currently updated second model to obtain at least one second detection result including a second predicted object sequence; Matching each of the first prediction object sequences and each of the second prediction object sequences to obtain at least one pair of first prediction object sequences and second prediction object sequences having a target matching relationship; Determining a current target loss value based on a similarity between each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship; When the current target loss value meets the preset condition or the number of times the model parameters of the first model are updated reaches a threshold number, the currently updated first model is determined as the trained first model.

6. The method according to claim 5, characterized in that The step of determining the trained first model based on the updated first model and the updated second model further includes: When the current target loss value does not meet the preset conditions, the model parameters of the first model and the model parameters of the second model are updated for the next time to obtain the first model after the next update and the second model after the next update; Based on the first model after the next update and the second model after the next update, the first model after training is determined.

7. The method according to any one of claims 2 to 4, characterized in that The method further comprises: Acquire at least one candidate object in the first image sample, each candidate object having a candidate object region and a candidate object category; Based on the first object region and the first object category corresponding to each first predicted object sequence, and the candidate object region and the candidate object category of each candidate object, matching each first predicted object sequence with each candidate object to obtain at least one pair of first predicted object sequence and candidate object having a target matching relationship; The determining of the target loss value based on the similarity between each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship includes: Determining a first loss value based on a similarity between each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship; Determining a second loss value based on each pair of the first predicted object sequence and the candidate object having a target matching relationship; A target loss value is determined based on the first loss value and the second loss value.

8. The method according to claim 7, characterized in that The determining of the second loss value based on each pair of the first prediction object sequence and the candidate object having a target matching relationship includes: For each pair of a first predicted object sequence and a candidate object having a target matching relationship, determining a first sub-loss value based on a first object region corresponding to the first predicted object sequence and a candidate object region of the candidate object, and determining a second sub-loss value based on a first object category corresponding to the first predicted object sequence and a candidate object category of the candidate object; A second loss value is determined based on each of the first sub-loss values ​​and each of the second sub-loss values.

9. The method according to claim 7, characterized in that The acquiring of at least one candidate object in the first image sample, each candidate object having a candidate object region and a candidate object category, comprises: Performing target detection on the first image sample in an unsupervised manner to obtain at least one predicted object region and a pseudo label for each predicted object region; the pseudo label for each predicted object region is used to characterize a predicted object category of the predicted object region; For each of the prediction object regions, the prediction object region is used as a candidate object region, and the pseudo label of the prediction object region is used as a candidate object category to obtain a candidate object.

10. The method according to claim 1, characterized in that The method further comprises: performing bipartite graph matching on each first predicted object sequence and each second predicted object sequence based on the first object region and the first object category corresponding to each first predicted object sequence, and the second object region and the second object category corresponding to each second predicted object sequence, to obtain at least one pair of first predicted object sequences and second predicted object sequences having a target matching relationship, including: Determining at least one candidate sequence pair set based on each of the first prediction object sequences and each of the second prediction object sequences; each of the candidate sequence pair sets includes at least one pair of the first prediction object sequence and the second prediction object sequence having a candidate matching relationship; For each candidate sequence pair set, determining a matching loss for the candidate sequence pair set based on a first object region and a first object category corresponding to the first predicted object sequence, and a second object region and a second object category corresponding to the second predicted object sequence, in each pair of first predicted object sequences and second predicted object sequences having a candidate matching relationship in the candidate sequence pair set; For each pair of first prediction object sequence and second prediction object sequence with a candidate matching relationship in the candidate sequence pair set with the smallest matching loss in the at least one candidate sequence pair set, at least one pair of first prediction object sequence and second prediction object sequence with a target matching relationship is determined.

11. The method according to any one of claims 1 to 4, characterized in that The first model includes a feature extraction network and a converter network; The method of performing target detection on the first augmented image using the first model to be trained to obtain at least one first detection result including a first predicted object sequence includes: Using the feature extraction network of the first model, extract features from the first augmented image to obtain image feature information; The image feature information is predicted using the transformer network of the first model to obtain at least one first prediction object sequence.

12. The method according to claim 11, characterized in that The first model also includes a first feedforward neural network; The step of using the transformer network of the first model to perform prediction processing on the image feature information to obtain at least one first prediction object sequence includes: Using the transformer network of the first model, predicting the image feature information to obtain at least one feature sequence; Utilizing the first feedforward neural network, each of the feature sequences is mapped to a target dimension to obtain at least one first prediction object sequence.

13. The method according to claim 12, characterized in that The first detection result further includes a first object region and a first object category, and the first model further includes a second feedforward neural network and a third feedforward neural network; The step of performing target detection on the first augmented image using the first model to be trained to obtain at least one first detection result including a first predicted object sequence further includes: For each of the feature sequences, the second feedforward neural network is used to perform region prediction on the feature sequence to obtain a first object region, and the third feedforward neural network is used to perform category prediction on the feature sequence to obtain a first object category.

14. The method according to claim 11, characterized in that The second model has the same network structure as the first model.

15. The method according to any one of claims 1 to 4, characterized in that The acquiring of the first augmented image and the second augmented image obtained by respectively performing augmentation processing on the first image sample includes: Performing a first image augmentation process on the first image sample to obtain a first augmented image; Perform a second image augmentation process on the first image sample to obtain a second augmented image.

16. The method according to claim 15, characterized in that The first image augmentation processing includes at least one of the following: color dithering, grayscale processing, Gaussian blur, and random erasure; The second image augmentation process includes at least one of the following: random scaling, random cropping, random flipping, and random resizing.

17. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Determining an initial third model based on the trained first model; Based on at least one second image sample, the model parameters of the third model are updated to obtain the trained third model.

18. An image processing method, characterized in that: include: Get the image to be processed; Using the trained fourth model, target detection is performed on the image to be processed to obtain a third detection result; wherein the third model includes at least one of the following: a first model obtained by the model training method according to any one of claims 1 to 16, and a third model obtained by the model training method according to claim 17.

19. A model training device, characterized in that: include: A first acquisition module is used to acquire a first augmented image and a second augmented image obtained by respectively performing augmentation processing on the first image sample; a first detection module, configured to perform object detection on the first augmented image using a first model to be trained, to obtain at least one first detection result including a first predicted object sequence, and to perform object detection on the second augmented image using a second model, to obtain at least one second detection result including a second predicted object sequence; a first matching module, configured to match each of the first prediction object sequences with each of the second prediction object sequences to obtain at least one pair of first prediction object sequences and second prediction object sequences having a target matching relationship; A first updating module is configured to update the model parameters of the first model at least once based on each pair of the first prediction object sequence and the second prediction object sequence having a target matching relationship, to obtain the trained first model; The first detection result includes a first object region and a first object category corresponding to a first predicted object sequence in the first detection result, and the second detection result includes a second object region and a second object category corresponding to a second predicted object sequence in the second detection result; The first matching module is also used to: perform bipartite graph matching on each first predicted object sequence and each second predicted object sequence based on the first object area and first object category corresponding to each first predicted object sequence, and the second object area and second object category corresponding to each second predicted object sequence, to obtain at least one pair of first predicted object sequences and second predicted object sequences having a target matching relationship.

20. An image processing device, characterized in that: include: A third acquisition module is used to acquire the image to be processed; The second detection module is used to use the trained fourth model to perform target detection on the image to be processed to obtain a third detection result; wherein the third model includes at least one of the following: a first model obtained by the model training method according to any one of claims 1 to 16, and a third model obtained by the model training method according to claim 17.

21. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 18 are implemented.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 18 are implemented.

23. A computer program product, comprising a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, the steps of the method according to any one of claims 1 to 18 are implemented.

Images