Model training and image processing method, device, equipment, and storage medium

Abstract

The embodiments of the present application disclose a model training and image processing method, apparatus, device, storage medium and computer program product, wherein the method comprises: determining a first scene image and a second scene image based on a first image sample; there is an overlapping area between the first scene image and the second scene image; using a first model to be trained, respectively extracting features from the first scene image and the second scene image to obtain a first scene feature of the first scene image and a second scene feature of the second scene image, and using a second model, respectively extracting features from the first scene image and the second scene image to obtain a third scene feature of the first scene image and a fourth scene feature of the second scene image; determining a target loss value based on the first scene feature and the second scene feature, the third scene feature and the fourth scene feature; based on the target loss value, updating the model parameters of the first model at least once to obtain the trained first model.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of artificial intelligence, and in particular to a model training and image processing method, device, equipment, storage medium and computer program product. Background Art

[0002] With the continuous development of computer vision technology, it is becoming easier and easier to obtain unlabeled data. However, for the massive unlabeled data sets in the field of computer vision, manual labeling will result in missing labels and consume a lot of labor costs. In related technologies, self-supervised training algorithms can be used to train neural network models. Self-supervised training algorithms can train models without providing labeled data, and provide pre-trained models for various tasks in the field of computer vision. Compared with supervised training algorithms, self-supervised training algorithms have obvious advantages in reducing supervised bias, long-tail problems, and fitting massive business data sets.

[0003] However, the self-supervised training algorithm in the related art can only perform model training on a single target image, but cannot be applied to model training on multiple target images, and has poor versatility. Summary of the invention

[0004] In view of this, embodiments of the present application provide a model training and image processing method, apparatus, device, storage medium and computer program product.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] On the one hand, an embodiment of the present application provides a model training method, the method comprising:

[0007] Determine a first scene image and a second scene image based on the first image sample; the first scene image and the second scene image have an overlapping area;

[0008] Using the first model to be trained, respectively extracting features from the first scene image and the second scene image to obtain a first scene feature of the first scene image and a second scene feature of the second scene image, and using the second model, respectively extracting features from the first scene image and the second scene image to obtain a third scene feature of the first scene image and a fourth scene feature of the second scene image;

[0009] Determining a target loss value based on the first scene feature and the second scene feature, and the third scene feature and the fourth scene feature;

[0010] Based on the target loss value, the model parameters of the first model are updated at least once to obtain the trained first model.

[0011] On the other hand, an embodiment of the present application provides an image processing method, the method comprising:

[0012] Get the image to be processed;

[0013] The image to be processed is processed using the trained fourth model to obtain a processing result; wherein the fourth model includes at least one of the following: a first model obtained using the above-mentioned model training method, and a third model obtained using the above-mentioned model training method.

[0014] On the other hand, the embodiment of the present application provides a model training device, the device comprising:

[0015] A first determining module, configured to determine a first scene image and a second scene image based on a first image sample; the first scene image and the second scene image have an overlapping area;

[0016] a first extraction module, configured to perform feature extraction on the first scene image and the second scene image respectively using the first model to be trained to obtain a first scene feature of the first scene image and a second scene feature of the second scene image, and to perform feature extraction on the first scene image and the second scene image respectively using the second model to obtain a third scene feature of the first scene image and a fourth scene feature of the second scene image;

[0017] A second determination module, configured to determine a target loss value based on the first scene feature and the second scene feature, and the third scene feature and the fourth scene feature;

[0018] The first updating module is used to update the model parameters of the first model at least once based on the target loss value to obtain the trained first model.

[0019] On the other hand, an embodiment of the present application provides an image processing device, the device comprising:

[0020] A third acquisition module is used to acquire the image to be processed;

[0021] The second detection module is used to use the trained fourth model to perform target detection on the image to be processed to obtain a detection result; wherein the fourth model includes at least one of the following: a first model obtained by the above-mentioned model training method, and a third model obtained by the above-mentioned model training method.

[0022] On the other hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0023] On the other hand, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements part or all of the steps in the above method when executed by a processor.

[0024] On the other hand, an embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is executed in a computer device, a processor in the display device executes some or all of the steps for implementing the above method.

[0025] On the other hand, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, it implements some or all of the steps in the above method.

[0026] In an embodiment of the present application, based on a first image sample, a first scene image and a second scene image are determined; since there is an overlapping area between the first scene image and the second scene image; using the first model to be trained, feature extraction is performed on the first scene image and the second scene image respectively to obtain a first scene feature of the first scene image and a second scene feature of the second scene image, and using the second model, feature extraction is performed on the first scene image and the second scene image respectively to obtain a third scene feature of the first scene image and a fourth scene feature of the second scene image; based on the first scene feature and the second scene feature, as well as the third scene feature and the fourth scene feature, a target loss value is determined; based on the target loss value, the model parameters of the first model are updated at least once to obtain the trained first model. In this way, since there is an overlapping area between the first scene image and the second scene image, the global similarity between the first scene image and the second scene image can be improved, and then by maintaining the consistency between the first scene features and the second scene features obtained after the first model and the second model process the first scene image and the second scene image respectively, and the third scene features and the fourth scene features, a self-supervised training process on the first image sample can be implemented, and the performance of the trained first model can be improved. It is applicable to model training on both single-target images and multi-target images, which can greatly reduce the high requirements and high dependence on sample data quality in the training process, and has better versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of an implementation flow of a model training method provided in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of an implementation flow of a model training method provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of an implementation flow of a model training method provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of an implementation flow of a model training method provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0032] Fig. 6A A schematic diagram of an implementation flow of a self-supervised pre-training method provided in an embodiment of the present application;

[0033] Figure 6B A schematic diagram of determining a first scene image and a second scene image provided in an embodiment of the present application;

[0034] Figure 6C A schematic diagram of an implementation architecture of a self-supervised pre-training method provided in an embodiment of the present application;

[0035] Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0036] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0037] Fig. 9 A hardware entity diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0039] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0041] In the related art, on the one hand, at the level of training data, the basic assumption of the current self-supervised training algorithm is that any perspective of a single-target image is the same object. However, this assumption cannot be extended to natural images with multiple targets (i.e., multi-target images). Therefore, the self-supervised training algorithm in the related art can only be trained on single-target images, but cannot be applied to training on multi-target images which are the easiest to obtain, resulting in poor versatility. On the other hand, at the level of applicable tasks, the current self-supervised training algorithm only considers representation learning at a single granularity, and is usually overfitted to classification tasks (such as classification between images, classification between scenes, etc.). As a result, the migrated model obtained after migration to various downstream tasks usually performs worse than the supervised training model, resulting in poor generalization ability.

[0042] The embodiment of the present application provides a model training method, which can be executed by a processor of a computer device. The computer device may refer to a server, a laptop, a tablet computer, a desktop computer, a smart TV, a set-top box, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device), or other device with data processing capabilities. Figure 1 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes the following steps S101 to S104:

[0043] Step S101: determining a first scene image and a second scene image based on a first image sample; the first scene image and the second scene image have an overlapping area.

[0044] Here, the first image sample may be any suitable image, including but not limited to at least one of a natural image of multiple targets, a single target image, etc. In some embodiments, the first sample image may include one or more than two instances. The instances included in the first image sample may be determined according to the actual application scenario, for example, may include but not limited to at least one of instances such as a person, a human body part, an animal, an animal limb, a plant, a flower, a leaf, a stone, a cloud, a fence, a television, a refrigerator, etc.

[0045] The first image sample may include at least one scene area, the first scene image and the second scene image may respectively correspond to a scene area in the first image sample, and the scene areas respectively corresponding to the first scene image and the second scene image may overlap, that is, there is an overlapping area between the first scene image and the second scene image. During implementation, the first scene image and the second scene image may be directly extracted from the first image sample, or extracted from the first image sample after image augmentation processing is performed on the first image sample, or obtained by performing image augmentation processing on the images corresponding to the two scene areas extracted from the first image sample, which is not limited here.

[0046] In some implementations, two scene regions that occupy more than half of the area of ​​the first image sample may be randomly determined in the first image sample, thereby obtaining a first scene image and a second scene image that respectively correspond to the two scene regions.

[0047] Step S102: Using the first model to be trained, feature extraction is performed on the first scene image and the second scene image respectively to obtain a first scene feature of the first scene image and a second scene feature of the second scene image; and using the second model, feature extraction is performed on the first scene image and the second scene image respectively to obtain a third scene feature of the first scene image and a fourth scene feature of the second scene image.

[0048] Here, the first model can be any suitable model that can extract image features, for example, it can include but is not limited to at least one of a target detection model based on feature engineering, an image classification model, an image segmentation model, etc., or it can be a specific feature extraction network model, which is not limited here. During implementation, the feature extraction algorithm used in the first model may include but is not limited to at least one of feature extraction based on a convolutional neural network, feature extraction based on a transformer, etc. Those skilled in the art can adopt a suitable feature extraction algorithm in the first model according to the actual business scenario, which is not limited here.

[0049] The second model can also be any suitable model that can extract image features. During implementation, the second model can have the same network structure as the first model, or a different network structure from the first model. The feature extraction algorithm used in the second model can be the same as that of the first model, or different, which is not limited here.

[0050] Using the first model, feature extraction can be performed on the first scene image and the second scene image respectively, wherein feature extraction on the first scene image can obtain a first scene feature, and feature extraction on the second scene image can obtain a second scene feature. Using the second model, feature extraction can also be performed on the first scene image and the second scene image respectively, wherein feature extraction on the first scene image can obtain a third scene feature, and feature extraction on the second scene image can obtain a fourth scene feature.

[0051] In some embodiments, the network structure of the first model may include a backbone network, a projection head, and a prediction head, and the network structure of the second model may include a backbone network and a projection head. In this way, the output difference between the first model and the second model can be increased, thereby reducing the collapse of model training.

[0052] Step S103: determining a target loss value based on the first scene feature, the second scene feature, the third scene feature, and the fourth scene feature.

[0053] Here, the first scene feature and the second scene feature are respectively features extracted by the first model from the first scene image and the second scene image, and are the representation of the first scene image and the representation of the second scene image learned by the first model; the third scene feature and the fourth scene feature are respectively features extracted by the second model from the first scene image and the second scene image, and are the representation of the first scene image and the representation of the second scene image learned by the second model.

[0054] In some embodiments, the target loss value may include the loss between the representation of the first scene image and the representation of the second scene image. The loss between the representation of the first scene image and the representation of the second scene image may include, but is not limited to, at least one of the loss between the first scene feature and the second scene feature, the loss between the first scene feature and the fourth scene feature, the loss between the second scene feature and the third scene feature, the loss between the third scene feature and the fourth scene feature, the loss between the combination of the first scene feature and the third scene feature and the combination of the second scene feature and the fourth scene feature. During implementation, those skilled in the art may determine the target loss value based on the first scene feature, the second scene feature, the third scene feature, and the fourth scene feature in an appropriate manner according to actual conditions, which is not limited here. For example, the target loss value may be determined based on the similarity between the first scene feature and the third scene feature, and the similarity between the second scene feature and the fourth scene feature; the target loss value may also be determined based on the similarity between the first scene feature and the fourth scene feature, and the similarity between the second scene feature and the third scene feature.

[0055] In some embodiments, the target loss value may include at least one of the loss between the representation of the first scene image and the representation of the second scene image, the loss between the representation of at least one instance in the first scene image and the representation of at least one instance in the second scene image, the loss between the representation of at least one instance in the first scene image or the second scene image and the representation of the first scene image, the loss between the representation of at least one instance in the first scene image or the second scene image and the representation of the second scene image, and the like.

[0056] Step S104: Based on the target loss value, update the model parameters of the first model at least once to obtain the trained first model.

[0057] Here, in some embodiments, it can be determined whether the model parameters of the first model need to be updated based on the target loss value. In the case where the model parameters of the first model need to be updated, the model parameters of the first model are updated using a suitable parameter update algorithm, and the target loss value is re-determined after the update, so as to determine whether the model parameters of the first model need to be continued to be updated based on the re-determined target loss value. In the case where it is determined that the model parameters of the first model do not need to be continued to be updated, the first model after the final update is determined as the first model after training. In implementation, after the model parameters of the first model are updated once, the updated first scene image and the updated second scene image can be determined based on the next first image sample; the updated first model is used to extract features from the updated first scene image and the updated second scene image, respectively, to obtain the updated first scene feature and the updated second scene feature, and the second model is used to extract features from the updated first scene image and the updated second scene image, respectively, to obtain the updated third scene feature and the updated fourth scene feature; based on the updated first scene feature and the updated second scene feature, and the updated third scene feature and the updated fourth scene feature, the updated target loss value is determined. The first image samples used in each updating process may be the same or different, which is not limited here.

[0058] For example, if the target loss value does not meet the preset conditions, the model parameters of the first model may be updated; if the target loss value meets the preset conditions or the number of updates to the model parameters of the first model reaches a set threshold, the model parameters of the first model may be stopped from being updated, and the finally updated first model may be determined as the trained first model. The preset conditions may include, but are not limited to, at least one of the target loss value being less than a set loss threshold, the target loss value changing and converging, and the like.

[0059] In an embodiment of the present application, based on a first image sample, a first scene image and a second scene image are determined; since there is an overlapping area between the first scene image and the second scene image; using the first model to be trained, feature extraction is performed on the first scene image and the second scene image respectively to obtain a first scene feature of the first scene image and a second scene feature of the second scene image, and using the second model, feature extraction is performed on the first scene image and the second scene image respectively to obtain a third scene feature of the first scene image and a fourth scene feature of the second scene image; based on the first scene feature and the second scene feature, as well as the third scene feature and the fourth scene feature, a target loss value is determined; based on the target loss value, the model parameters of the first model are updated at least once to obtain the trained first model. In this way, since there is an overlapping area between the first scene image and the second scene image, the global similarity between the first scene image and the second scene image can be improved, and then by maintaining the consistency between the first scene features and the second scene features obtained after the first model and the second model process the first scene image and the second scene image respectively, and the third scene features and the fourth scene features, a self-supervised training process on the first image sample can be implemented, and the performance of the trained first model can be improved. It is applicable to model training on both single-target images and multi-target images, which can greatly reduce the high requirements and high dependence on sample data quality in the training process, and has better versatility.

[0060] In some embodiments, the above step S103 may include the following steps S111 to S112:

[0061] Step S111, determining a target scene loss value based on the first scene feature and the second scene feature, and the third scene feature and the fourth scene feature.

[0062] Here, the target scene loss value may represent the loss between the representation of the first scene image and the representation of the second scene image. During implementation, those skilled in the art may determine the target scene loss value based on the first scene feature, the second scene feature, the third scene feature, and the fourth scene feature in an appropriate manner according to the actual situation, which is not limited here. For example, the target scene loss value may be determined based on the similarity between the first scene feature and the third scene feature, and the similarity between the second scene feature and the fourth scene feature; the target scene loss value may also be determined based on the similarity between the first scene feature and the fourth scene feature, and the similarity between the second scene feature and the third scene feature; the first scene feature and the third scene feature may be combined to obtain a first combined feature, the second scene feature and the fourth scene feature may be combined to obtain a second combined feature, and the target scene loss value may be determined based on the similarity between the first combined feature and the second combined feature.

[0063] Step S112, determining a target loss value based on the target scene loss value.

[0064] Here, in some implementations, a target scene loss value may be determined as the target loss value.

[0065] In other embodiments, the target scene loss value and other loss values ​​may be comprehensively considered to determine the target loss value. Other loss values ​​may include, but are not limited to, at least one of the loss between the representation of at least one instance in the first scene image and the representation of at least one instance in the second scene image, the loss between the representation of at least one instance in the first scene image or the second scene image and the representation of the first scene image, the loss between the representation of at least one instance in the first scene image or the second scene image and the representation of the second scene image, etc.

[0066] During implementation, those skilled in the art may determine the target loss value based on the target scene loss value in an appropriate manner according to actual conditions, which is not limited here.

[0067] In some embodiments, the above step S111 may include the following steps S121 to S123:

[0068] Step S121: determining a first scene loss value based on the similarity between the first scene feature and the fourth scene feature.

[0069] Step S122: determining a second scene loss value based on the similarity between the second scene feature and the third scene feature.

[0070] Here, any suitable similarity loss function can be used to determine the first scene loss value based on the similarity between the first scene feature and the fourth scene feature, and to determine the second scene loss value based on the similarity between the second scene feature and the third scene feature, and there is no limitation here. The similarity loss function may include but is not limited to at least one of an absolute value loss function, a minimum square error loss function, a cosine loss function, etc. In implementation, the same similarity loss function may be used in the process of determining the first scene loss value and the second scene loss value, or different similarity loss functions may be used.

[0071] Step S123: determining the target scene loss value based on the first scene loss value and the second scene loss value.

[0072] Here, the target scene loss value can be determined based on the first scene loss value and the second scene loss value in an appropriate manner according to the actual situation, and the embodiments of the present application are not limited. For example, the sum of the first scene loss value and the second scene loss value can be determined as the target scene loss value, or the average of the first scene loss value and the second scene loss value can be determined as the target scene loss value, and different weights can be used to perform weighted summation of the first scene loss value and the second scene loss value to obtain the target scene loss value.

[0073] In the above embodiment, the first scene loss value is determined based on the similarity between the first scene feature and the fourth scene feature; the second scene loss value is determined based on the similarity between the second scene feature and the third scene feature; and the target scene loss value is determined based on the first scene loss value and the second scene loss value. In this way, the consistency between the representation of the first scene image learned by the first model and the representation of the second scene image learned by the second model, as well as the consistency between the representation of the second scene image learned by the first model and the representation of the first scene image learned by the second model can be maintained, thereby improving the ability of the trained first model to extract common scene representations, and further improving the performance of the trained first model.

[0074] In some embodiments, the above step S121 may include: determining a first scene loss value based on a negative cosine similarity between the first scene feature and the fourth scene feature;

[0075] The above step S122 may include: determining a second scene loss value based on the negative cosine similarity between the second scene feature and the third scene feature.

[0076] In some implementations, the method shown in the following formula 1 can be used to determine the first scene loss value based on the negative cosine similarity between the first scene feature and the fourth scene feature:

[0077]

[0078] Among them, x1 represents the first scene image, x2 represents the second scene image, and f θ (x1) represents the first scene feature of the first scene image obtained using the first model, g x (x2) represents the fourth scene feature of the second scene image obtained using the second model, f θ (x1),g x (x2)> represents f θ (x1) and g x(x2), ‖I‖2 represents the length of the vector. In addition, the method for determining the second scene loss value based on the negative cosine similarity between the second scene feature and the third scene feature can refer to the method for determining the first scene loss value based on the negative cosine similarity between the first scene feature and the fourth scene feature.

[0079] In some embodiments, the overlapping area includes at least one candidate instance; the above step S112 may include:

[0080] Step S131, determining a target loss value based on the target scene loss value and the target instance loss value; wherein the target instance loss value includes at least one of the following: a first instance loss value and a second instance loss value.

[0081] Here, the target instance loss value is a loss value determined by considering the characterization of the instance granularity in the first sample image, and may include at least one of the first instance loss value and the second instance loss value.

[0082] The first instance loss value may represent the loss between the representation of the scene and the representation of the instance in the first sample image, and may include but is not limited to at least one of the loss between the representation of at least one instance in the first scene image or the second scene image and the representation of the first scene image, the loss between the representation of at least one instance in the first scene image or the second scene image and the representation of the second scene image, etc.

[0083] The second instance loss value may represent the loss between representations of each instance in the two scenes in an overlapping region of the two scenes in the first sample image, for example, the loss between a representation of at least one instance in the first scene image and a representation of at least one instance in the second scene image.

[0084] During implementation, the target loss value can be determined based on the target scene loss value and the target instance loss value in an appropriate manner according to the actual situation, and the embodiments of the present application are not limited thereto. For example, the sum of the target scene loss value and the target instance loss value can be determined as the target loss value, or the average of the target scene loss value and the target instance loss value can be determined as the target loss value, or different weights can be used to perform weighted summation of the target scene loss value and the target instance loss value to obtain the target loss value.

[0085] In some embodiments, the target loss value can be determined based on the target scenario loss value, the first instance loss value, and the second instance loss value. During implementation, the target loss value can be determined based on the target scenario loss value, the first instance loss value, and the second instance loss value in a suitable manner according to the actual situation, and the embodiments of the present application are not limited. For example, the sum of the target scenario loss value, the first instance loss value, and the second instance loss value can be determined as the target loss value, or the average of the target scenario loss value, the first instance loss value, and the second instance loss value can be determined as the target loss value, and different weights can be used to perform weighted summation on the target scenario loss value, the first instance loss value, and the second instance loss value to obtain the target loss value.

[0086] The above step S103 may further include the following steps S132 to S133:

[0087] Step S132, when the target instance loss value includes the first instance loss value, determine the first instance loss value based on the third scene feature and the fourth scene feature, and the first instance feature of each of the candidate instances in the first scene image; each of the first instance features is extracted using the first model.

[0088] Here, at least one candidate instance may be included in the overlapping area of ​​the first scene image and the second scene image. The first instance feature of each candidate instance in the first scene image may be extracted in advance using the first model, or may be extracted in real time during the process of the first instance loss value, which is not limited here. During implementation, the first model may be used in any suitable manner to extract the first instance feature of each candidate instance in the first scene image. For example, the position information of each candidate instance in the first scene image may be determined, and the first model may be used to extract features from the first scene image based on each position information, so as to obtain the first instance feature of each candidate instance in the first scene image. For another example, the first instance image corresponding to each candidate instance may be cropped out from the first scene image, and the first model may be used to extract features from each first instance image, so as to obtain the first instance feature of each candidate instance in the first scene image.

[0089] The first instance loss value can represent the loss between the representation of the scene in the first sample image and the representation of the instance. Therefore, based on the third scene feature and the fourth scene feature, and the first instance feature of each candidate instance in the first scene image, the first instance loss value can be determined. In implementation, the first instance loss value can be determined based on the third scene feature and the fourth scene feature, and the first instance feature of each candidate instance in the first scene image in an appropriate manner according to the actual situation, which is not limited here. For example, each first instance feature can be mapped to the dimension of the third scene feature to obtain the mapped first instance feature, and the first instance loss value is determined based on the similarity between each mapped first instance feature and the third scene feature, and the similarity between each mapped first instance feature and the fourth scene feature. For another example, each first instance feature can be connected and mapped to the dimension of the third scene feature to obtain a mapping feature, and the first instance loss value is determined based on the similarity between the mapping feature and the third scene feature, and the similarity between the mapping feature and the fourth scene feature.

[0090] Step S133, when the target instance loss value includes the second instance loss value, determine the second instance loss value based on each of the first instance features and the second instance features of each of the candidate instances in the second scene image; each of the second instance features is extracted using the second model.

[0091] Here, the second instance features of each candidate instance in the second scene image may be extracted in advance using the second model, or may be extracted in real time during the process of determining the second instance loss value, which is not limited here. During implementation, the second model may be used in any suitable manner to extract the second instance features of each candidate instance in the second scene image. For example, the position information of each candidate instance in the second scene image may be determined, and the second model may be used to perform feature extraction on the second scene image based on each position information to obtain the second instance features of each candidate instance in the second scene image. For another example, the second instance image corresponding to each candidate instance may be cropped from the second scene image, and the second model may be used to perform feature extraction on each second instance image to obtain the second instance features of each candidate instance in the second scene image.

[0092] The second instance loss value can represent the loss between the representations of each instance in the two scenes in the overlapping area of ​​the two scenes in the first sample image. Therefore, based on the first instance feature of each candidate instance in the second scene image and the second instance feature of each candidate instance in the second scene image, the second instance loss value can be determined. During implementation, the second instance loss value can be determined based on each first instance feature and each second instance feature in a suitable manner according to the actual situation, which is not limited here. For example, the second instance loss value can be determined based on the transmission cost between each first instance feature and each second instance feature, or based on the similarity between each first instance feature and each second instance feature.

[0093] In the above embodiment, when determining the target loss value, in addition to considering the target scene loss value, the loss between the representation of the scene in the first sample image and the representation of the instance and / or the loss between the representations of each instance in the overlapping area of ​​the two scenes in the first sample image are also considered. In this way, the learning ability of the trained first model for multi-granularity representations can be improved, thereby improving the generalization ability of the trained first model, so that the trained first model can be generalized to multiple categories of downstream tasks.

[0094] In some embodiments, the third scene feature and the fourth scene feature have the same dimension; the step S132 described above, determining the first instance loss value based on the third scene feature and the fourth scene feature, and the first instance feature of each candidate instance in the first scene image, may include the following steps S141 to S143:

[0095] Step S141, connecting each of the first instance features to obtain a connection feature.

[0096] Here, each first instance feature can be connected together in any appropriate manner to obtain a connected connection feature. In implementation, the connection order of the first instance features corresponding to each candidate instance can be determined according to the position of each candidate instance in the first scene image, and each first instance feature can be connected according to the connection order to obtain a connection feature; or each first instance feature can be connected according to a random connection order to obtain a connection feature, which is not limited here.

[0097] For example, when the number of first instance features is 4 and each first instance feature is a 64-dimensional vector, each first instance feature may be connected, and the obtained connected feature is a 256-dimensional vector.

[0098] Step S142: Map the connection feature to the dimension of the third scene feature to obtain a mapping feature.

[0099] Here, the connection feature can be mapped according to the dimension of the third scene feature to obtain a mapping feature with the same dimension as the third scene feature. During implementation, any appropriate mapping method can be used to map the connection feature, which is not limited here.

[0100] For example, each first instance feature may be connected in the manner shown in the following formula 2 to obtain a connection feature, and the connection feature may be linearly mapped to the dimension of the third scene feature to obtain a mapping feature:

[0101] I=f linear (concat(o1,o2,……,o k )) (2);

[0102] Among them, concat(o1,o2,……,o k ) represents the first instance feature o1, o2, ..., o k To connect, linear () represents a linear mapping, and I is the mapping feature.

[0103] Step S143, determining a first instance loss value based on the third scene feature, the fourth scene feature and the mapping feature.

[0104] Here, based on the third scene feature, the fourth scene feature and the mapping feature, the loss between the representation of the scene in the first sample image and the representation of the instance, that is, the first instance loss value, can be determined. During implementation, the first instance loss value can be determined based on the third scene feature, the fourth scene feature and the mapping feature in any appropriate manner according to actual conditions, and is not limited here.

[0105] In the above embodiment, each first instance feature is connected to obtain a connection feature, and the connection feature is mapped to the dimension of the third scene feature to obtain a mapping feature, and the first instance loss value is determined based on the third scene feature, the fourth scene feature and the mapping feature. In this way, since there is a correlation between the scene and the instance in the scene, the first instance loss value is determined based on the third scene feature, the fourth scene feature and the mapping feature when determining the target loss value, and the semantic guidance model of the approximate scene view can be used to learn the representation of the instance, thereby effectively maintaining the consistency between the representation of the scene in the first sample image and the representation of the instance, improving the learning ability of the trained first model for the semantic affinity between the scene and the instance, and further improving the generalization ability of the trained first model.

[0106] In some embodiments, the above step S143 may include the following steps S151 to S153:

[0107] Step S151, determining a first sub-loss value based on the similarity between the third scene feature and the mapping feature.

[0108] Here, any suitable similarity loss function may be used to determine the first sub-loss value based on the similarity between the third scene feature and the mapping feature, and this is not limited here.

[0109] Step S152: determine a second sub-loss value based on the similarity between the fourth scene feature and the mapping feature.

[0110] Here, any suitable similarity loss function can be used to determine the second sub-loss value based on the similarity between the fourth scene feature and the mapping feature, and is not limited here. In implementation, the similarity loss function used to determine the second sub-loss value can be the same as or different from the similarity loss function used to determine the second sub-loss value.

[0111] Step S153: determine a first instance loss value based on the first sub-loss value and the second sub-loss value.

[0112] Here, the first instance loss value can be determined based on the first sub-loss value and the second sub-loss value in an appropriate manner according to the actual situation, and the embodiments of the present application are not limited. For example, the sum of the first sub-loss value and the second sub-loss value can be determined as the first instance loss value, or the average of the first sub-loss value and the second sub-loss value can be determined as the first instance loss value, and different weights can be used to perform weighted summation of the first sub-loss value and the second sub-loss value to obtain the first instance loss value.

[0113] In some embodiments, the above step S151 may include: determining a first sub-loss value based on the negative cosine similarity between the third scene feature and the mapping feature; the above step S152 may include: determining a second sub-loss value based on the negative cosine similarity between the fourth scene feature and the mapping feature.

[0114] In implementation, the method shown in the following formula 3 can be used to determine the first sub-loss value based on the negative cosine similarity between the third scene feature and the mapping feature:

[0115]

[0116] Among them, s1 represents the first scene image, I represents the mapping feature, and g x (s1) represents the third scene feature of the first scene image obtained by using the second model. In addition, the method for determining the second loss value based on the negative cosine similarity between the fourth scene feature and the mapping feature can refer to the method for determining the first sub-loss value based on the negative cosine similarity between the third scene feature and the mapping feature.

[0117] In some embodiments, the second instance feature based on each first instance feature and each candidate instance in the second scene image described in step S133 includes the following steps S161 to S162:

[0118] Step S161: Match each of the first instance features and each of the second instance features to obtain at least one pair of first instance features and second instance features having a target matching relationship.

[0119] Here, the first instance feature and the second instance feature having a target matching relationship may be two instance features having similarities. In implementation, the matching between each first instance feature and each second instance feature may be one-to-one, one-to-many, many-to-one, or many-to-many. Those skilled in the art may match each first instance feature and each second instance feature in any appropriate matching manner according to actual conditions, which is not limited here.

[0120] In some embodiments, the similarity between each first instance feature and each second instance feature may be determined, and the first instance feature and the second instance feature whose similarity is greater than a similarity threshold may be determined as a pair of first instance features and second instance features having a target matching relationship.

[0121] In some implementations, at least one pair of a first instance feature and a second instance feature having a target matching relationship may be determined based on an optimal transmission algorithm.

[0122] Step S162: Determine a second instance loss value based on each pair of first instance features and second instance features having a target matching relationship.

[0123] Here, any appropriate similarity loss function may be used to determine the similarity loss between each pair of first instance features and second instance features having a target matching relationship, and based on each similarity loss, a target loss value may be determined.

[0124] In the above embodiment, each first instance feature and each second instance feature are matched to obtain at least one pair of first instance features and second instance features having a target matching relationship, and the second instance loss value is determined based on each pair of first instance features and second instance features having a target matching relationship. In this way, different instances in the scene can be distinguished, and by maintaining the consistency between each pair of first instance features and second instance features having a target matching relationship, the ability of the trained first model to learn the distinguishing representations of different instances in the scene can be improved, thereby improving the ability of the trained first model to distinguish different instances in the scene, and further improving the generalization ability of the trained first model.

[0125] In some embodiments, the above step S161 may include:

[0126] Step S171: Based on the optimal transmission algorithm, each of the first instance features and each of the second instance features are matched to obtain at least one pair of first instance features and second instance features having a target matching relationship.

[0127] Here, in the optimal transport algorithm, it is assumed that M suppliers are needed to transport goods to N demanders, and the mth supplier holds b m units of goods, and the nth demander needs a n The unit cost of transporting goods from the mth supplier to the nth demander is c mn The goal of the optimal transportation algorithm is to find an optimal transportation plan According to the optimal transportation plan, all goods from M suppliers can be transported to N demanders at the minimum transportation cost, where M and N are both positive integers, m is a positive integer greater than 0 and less than or equal to M, and n is a positive integer greater than 0 and less than or equal to N. For example, the optimal transportation plan can satisfy the constraints shown in the following formula 4:

[0128]

[0129] Among them, b m and a n They are respectively called the marginal weight of the mth supplier and the marginal weight of the nth demander.

[0130] In some implementations, based on the optimal transmission algorithm, in the process of matching each first instance feature with each second instance feature, a first feature vector set O=[o1, o2, 2, o k ], construct a second feature vector set T = [t1, t2, 2, t k ], where the number of first instance features is k, and the number of second instance features is also k. Each node o in the first feature vector set can be m As the supplier, each node t in the second feature vector set n As the demand side. Since nodes with similar representations tend to incur less transmission cost between each other, while nodes with unrelated representations tend to incur more transmission cost, it is possible to m With t n The distance between m With t n The unit transmission cost c mn , where m and n are both positive integers not exceeding k. For example, see the following formula 5:

[0131]

[0132] Among them, m With t n The transmission cost between mn It can be o m With t n The cosine distance between .

[0133] Node o m The marginal weight b m and node t n The marginal weight a n It can be determined as shown in the following formula 6:

[0134]

[0135] Among them, max{} can ensure that the marginal weight is non-negative.

[0136] Based on the optimal transmission algorithm, the optimal transmission plan between each first instance feature and each second instance feature can be determined Optimal transmission plan The optimal transport plan may include the transmission allocation between each first instance feature and each second instance feature. In implementation, any suitable method may be used to determine the optimal transport plan. This is not limited here. For example, the Sinkhorn-Knopp algorithm can be used to quickly iterate the above formula 4 to obtain the optimal transportation plan

[0137] In some embodiments, the first instance feature and the second instance feature whose transmission cost satisfies a preset cost condition may be determined as the first instance feature and the second instance feature having a target matching relationship. For example, the first instance feature and the second instance feature whose transmission cost is less than a set cost threshold may be determined as the first instance feature and the second instance feature having a target matching relationship.

[0138] In some embodiments, the first instance feature and the second instance feature whose transmission allocation amount satisfies the preset allocation amount condition may be determined as the first instance feature and the second instance feature having a target matching relationship. For example, the first instance feature and the second instance feature whose transmission allocation amount is greater than a set allocation amount threshold may be determined as the first instance feature and the second instance feature having a target matching relationship.

[0139] The above step S162 may include:

[0140] Step S172: determining a second instance loss value based on the transmission cost between each pair of first instance features and second instance features having a target matching relationship.

[0141] Here, the transmission cost between each pair of the first instance feature and the second instance feature having a target matching relationship may be determined based on the unit transmission cost and the transmission allocation amount between the first instance feature and the second instance feature.

[0142] In some embodiments, for each pair of a first instance feature and a second instance feature having a target matching relationship, the transmission cost between the first instance feature and the second instance feature can be determined based on the product of the unit transmission cost and the transmission allocation amount between the first instance feature and the second instance feature.

[0143] During implementation, the second instance loss value can be determined based on the transmission cost between each pair of first instance features and second instance features having a target matching relationship in an appropriate manner according to the actual situation, and the embodiments of the present application are not limited. For example, the sum of the transmission costs between each pair of first instance features and second instance features having a target matching relationship can be determined as the second instance loss value, or the average value of the transmission costs between each pair of first instance features and second instance features having a target matching relationship can be determined as the second instance loss value, and different weights can be used to perform weighted summation of the transmission costs between each pair of first instance features and second instance features having a target matching relationship to obtain the second instance loss value, and the sum of the transmission costs between each pair of first instance features and second instance features having a target matching relationship can also be offset according to a specific offset value to obtain the second instance loss value.

[0144] For example, the loss value of the second instance may be determined in the manner shown in the following formula 7:

[0145]

[0146] Among them, o m With t n The negative cosine similarity between As o m With t n The unit transmission cost between For optimal transmission plan The first instance feature o m With the second instance feature t n The transmission distribution between the first instance features o m With the second instance feature t n The unit transfer cost between Transmission allocation Multiply them together to get the first instance feature om and the second instance feature t n The transmission cost between the first instance feature and the second instance feature is accumulated for each pair of target matching relationships, and the second instance loss value can be obtained.

[0147] In the above embodiment, based on the optimal transmission algorithm, each first instance feature and each second instance feature are matched to obtain at least one pair of first instance features and second instance features having a target matching relationship, and the second instance loss value is determined based on the transmission cost between each pair of first instance features and second instance features having a target matching relationship. In this way, the distinction between different instances in the scene can be converted into an optimal transmission matching problem between the representations of all candidate instances in the overlapping area of ​​the first scene image and the second scene image, so that at least one pair of first instance features and second instance features having a target matching relationship can be quickly and accurately determined, and the consistency between the first instance features and second instance features having a target matching relationship can be maintained. In addition, since the optimal transmission matching can be achieved when the representation of each instance is similar to itself and different from the representations of other instances, the model can be guided to distinguish and learn the representations between different co-occurring instances, thereby further improving the ability of the trained first model to distinguish different instances in the scene.

[0148] In some embodiments, the method may further include the following steps S181a to S182a:

[0149] Step S181a: acquiring, from the first scene image, a first instance image corresponding to each candidate instance in the overlapping area.

[0150] Here, at least one candidate instance may be included in the overlapping area of ​​the first scene image and the second scene image, and each candidate instance may correspond to a candidate area in the first scene image, and the first instance image corresponding to each candidate instance is the image of the candidate area corresponding to the candidate instance. In implementation, the image of the candidate area corresponding to each candidate instance may be extracted from the first scene image to obtain the first instance image corresponding to each candidate instance, or after extracting the image of the candidate area corresponding to each candidate instance from the first scene image, the image of each candidate area may be enhanced by resizing, rotating, grayscale conversion, etc. to obtain the first instance image corresponding to each candidate instance, which is not limited here.

[0151] Step S182a: Using the first model, extract features from each of the first instance images to obtain features of each of the first instance images.

[0152] Here, in some implementations, for each first instance image, a first model may be used to perform feature extraction on the first instance image to obtain a first instance feature of the first instance image.

[0153] In some embodiments, each first instance image may be superimposed to obtain a superimposed image, and features of the superimposed image may be extracted using the first model to obtain features of the superimposed image, and the features of the superimposed image may include the first instance features of each first instance image. In this way, by superimposing each first instance image as a whole and performing feature extraction, the first instance features of each first instance image extracted may contain more common features.

[0154] In some embodiments, the above step S181a may include the following steps S191a to S192a:

[0155] Step S191a, acquiring a first candidate region image corresponding to each candidate instance in the overlapping region from the first scene image;

[0156] Step S192a, adjusting the size of each of the first candidate region images to a target size, and obtaining a first instance image corresponding to each of the candidate instances.

[0157] Here, the target size may be pre-set or determined according to the size of the first image sample and / or the first scene image, which is not limited here. For example, the target size may be 96*96 pixels.

[0158] In some embodiments, when the target instance loss value includes the second instance loss value, the method further includes the following steps S181b to S182b:

[0159] Step S181b, acquiring a second instance image corresponding to each candidate instance in the overlapping area from the second scene image;

[0160] Step S182b: Using the second model, perform feature extraction on each of the second instance images to obtain each of the second instance features.

[0161] Here, the above steps S181b to S182b correspond to steps S181a to S182a in the aforementioned embodiment respectively, and the implementation methods of the aforementioned steps S181a to S182a may be referred to during implementation.

[0162] In some embodiments, the above step S181b may include the following steps S191b to S192b:

[0163] Step S191b, acquiring a second candidate region image corresponding to each candidate instance in the overlapping region from the second scene image;

[0164] Step S192b: adjusting the size of each of the second candidate region images to a target size, and obtaining a second instance image corresponding to each of the candidate instances.

[0165] Here, the above steps S191b to S192b correspond to steps S191a to S192a in the aforementioned embodiment respectively, and the implementation methods of the aforementioned steps S191a to S192a may be referred to during implementation.

[0166] In some embodiments, the method further includes: if the number of candidate instances included in the overlapping area is less than a set number, determining at least one new candidate instance in the overlapping area so that the overlapping area includes the set number of candidate instances.

[0167] Here, the overlapping area between the first scene image and the second scene image may include at least one candidate instance, or may not include any candidate instance. When the number of candidate instances included in the overlapping area is less than a set number, at least one newly added candidate instance may be determined in the overlapping area so that the number of candidate instances included in the overlapping area reaches the set number.

[0168] During implementation, the set number can be determined according to actual conditions and is not limited here. For example, the set number can be 2, 4, or 8.

[0169] In some embodiments, at least one candidate instance frame may be generated in the overlapping region, and at least one newly added candidate instance may be determined based on the generated candidate instance frame. In implementation, at least one candidate instance frame may be generated in any suitable manner, which is not limited here. For example, a naive strategy may be used to randomly generate at least one candidate instance frame in the overlapping region, and the naive strategy may include setting the minimum size of the candidate instance frame to 64 pixels, setting the aspect ratio range to between 1 / 3 and 3 / 1, and setting the maximum intersection-over-union ratio between the candidate instance frames to 0.5.

[0170] In the above embodiment, when the number of candidate instances included in the overlapping area is less than the set number, at least one newly added candidate instance is determined in the overlapping area so that the overlapping area includes the set number of candidate instances. In this way, the overlapping area between the first scene image and the second scene image can include at least the set number of candidate instances, thereby further improving the correlation between the first scene image and the second scene image, and further improving the performance of the trained first model. In addition, the ability of the trained first model to learn the semantic affinity between scenes and instances, as well as the ability to distinguish different instances in the scene, can be further improved. In this way, the ability of the trained first model to learn multi-granularity representations can be further improved, thereby improving the generalization ability of the trained first model.

[0171] An embodiment of the present application provides a model training method, which can be executed by a processor of a computer device. Figure 2 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes the following steps S201 to S204:

[0172] Step S201: determining a first scene image and a second scene image based on a first image sample; the first scene image and the second scene image have an overlapping area.

[0173] Step S202: Using the first model to be trained, feature extraction is performed on the first scene image and the second scene image respectively to obtain a first scene feature of the first scene image and a second scene feature of the second scene image; and using the second model, feature extraction is performed on the first scene image and the second scene image respectively to obtain a third scene feature of the first scene image and a fourth scene feature of the second scene image.

[0174] Step S203: determining a target loss value based on the first scene feature, the second scene feature, the third scene feature, and the fourth scene feature.

[0175] Step S204: when the target loss value does not meet the preset conditions, the model parameters of the first model are updated to obtain an updated first model.

[0176] Here, the preset conditions may include but are not limited to the target loss value being less than a set loss value threshold, the target loss value changing and converging, etc. During implementation, the preset conditions may be set according to actual conditions and are not limited here.

[0177] The method for updating the model parameters of the first model may be determined according to actual conditions, and may include but is not limited to at least one of the gradient descent method, the momentum update method, the Newton momentum method, etc., which is not limited here.

[0178] In some implementations, the above method may further include the following steps S211 to S214:

[0179] Step S211: determine the first scene image and the second scene image determined based on the next first image sample as the current scene image and the current second scene image respectively.

[0180] Here, the next first image sample may be the same image as the current first image sample, or may be an image different from the current first image sample.

[0181] Step S212, using the currently updated first model, respectively extract features from the current first scene image and the current second scene image to obtain first scene features of the current first scene image and second scene features of the current second scene image, and using the second model, respectively extract features from the current first scene image and the current second scene image to obtain third scene features of the current first scene image and fourth scene features of the current second scene image.

[0182] Step S213, determining a current target loss value based on the first scene feature and the second scene feature, and the third scene feature and the fourth scene feature.

[0183] Step S214, when the target loss value satisfies the preset condition or the number of times the model parameters of the first model are updated reaches a threshold number, the currently updated first model is determined as the trained first model.

[0184] In some embodiments, the updating of the model parameters of the first model in step S204 to obtain an updated first model may include:

[0185] Step S221, updating the model parameters of the first model and the model parameters of the second model respectively to obtain an updated first model and an updated second model.

[0186] In some embodiments, when the target loss value does not meet the preset conditions, the model parameters of the first model and the model parameters of the second model can be updated respectively to obtain the updated first model and the updated second model, and a new target loss value can be determined based on the updated first model and the updated second model, and whether to continue to update the updated first model and the updated second model is determined by judging whether the new target loss value meets the preset conditions. When the new target loss value meets the preset conditions, it can be determined not to continue to update the updated first model and the updated second model, and the updated first model can be determined as the trained first model; when the new target loss value does not meet the preset conditions or the number of times the model parameters of the first model are updated reaches the number threshold, the updated first model and the updated second model can be continued to be updated, and the final updated first model is determined as the trained first model.

[0187] In the above embodiment, in the process of updating the model parameters of the first model, the model parameters of the second model are also updated, so that the learning capabilities of the first model and the second model can enhance each other, thereby further improving the performance of the trained first model.

[0188] In some implementations, the above step S221 may include the following steps S231 to S232:

[0189] Step S231: Based on the current model parameters of the first model, momentum update is performed on the model parameters of the second model to obtain an updated second model.

[0190] Here, those skilled in the art may adopt any suitable momentum update method to perform momentum update on the model parameters of the second model based on the current model parameters of the first model according to actual conditions during implementation, and the embodiments of the present application are not limited thereto.

[0191] In some implementations, the current model parameters of the first model and the current model parameters of the second model may be weighted summed based on the set weights to obtain an updated second model. For example, the model parameters of the second model may be momentum updated in the manner shown in the following formula 8:

[0192] ξ j+1 =m*ξ j +(1-m)*θ j (8);

[0193] Among them, x j and θ j are the current model parameters of the second model and the current model parameters of the first model, respectively, j+1is the model parameter of the updated second model, and m is the set momentum coefficient. In some embodiments, m may be a value greater than or equal to 0.99 and less than 1, for example, m is 0.995.

[0194] Step S232: using a gradient update method to update the current model parameters of the first model to obtain an updated first model.

[0195] Here, any suitable gradient update algorithm can be used to update the current model parameters of the first model, and the embodiment of the present application is not limited. For example, the gradient update algorithm may include but is not limited to at least one of batch gradient descent, stochastic gradient descent, mini-batch gradient descent, etc.

[0196] In the above embodiment, based on the current model parameters of the first model, the model parameters of the second model are momentum updated to obtain an updated second model, and the current model parameters of the first model are updated by gradient updating to obtain an updated first model. In this way, the first model and the second model can be updated at different rates, which can reduce the collapse of the model and further improve the performance of the trained first model.

[0197] An embodiment of the present application provides a model training method, which can be executed by a processor of a computer device. Figure 3 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the method includes the following steps S301 to S306:

[0198] Step S301: Acquire at least one candidate instance in the first image sample.

[0199] Here, at least one candidate instance in the first image sample may be randomly determined, or may be obtained by performing target detection on the first image sample using any suitable unsupervised algorithm, or may be manually annotated, which is not limited here. For example, the unsupervised detection algorithm may include but is not limited to at least one of a sliding window method, a candidate region algorithm, a selective search algorithm, and the like.

[0200] Step S302: Determine at least one target candidate instance from at least one of the candidate instances.

[0201] Here, at least one target candidate instance may be determined from at least one candidate instance in any suitable manner, which is not limited here.

[0202] In some embodiments, a candidate instance screening condition may be pre-set, and based on the candidate instance screening condition, at least one target candidate instance that meets the candidate instance screening condition may be screened out from at least one candidate instance. During implementation, appropriate candidate instance screening conditions may be set according to actual conditions, and are not limited here. For example, each candidate instance may correspond to a candidate area in the first image sample, and the candidate instance screening condition may include but is not limited to at least one of the minimum scale, aspect ratio range, and maximum intersection-over-union ratio of the candidate area corresponding to the candidate instance. In this way, a target candidate instance that meets the candidate instance screening condition may be obtained, and the computational complexity of model training may be reduced to a certain extent, thereby improving the efficiency of model training.

[0203] In some implementations, a set number of target candidate instances may be determined from at least one of the candidate instances. In implementation, the set number may be determined based on actual conditions and is not limited here. For example, the set number may be 2, 4, or 8.

[0204] Step S303: determining, based on the first image sample and the at least one target candidate instance, a first scene image and a second scene image having an overlapping area, wherein the overlapping area includes the at least one target candidate instance.

[0205] Here, two scene regions each including the at least one target candidate instance can be determined from the first image sample, and the first scene image and the second scene image can be determined respectively based on the images of the two scene regions in the first image sample. Since the scene regions corresponding to the first scene image and the second scene image both include the at least one target candidate instance, there is an overlapping region between the first scene image and the second scene image, and the at least one target candidate instance is included in the overlapping region.

[0206] In some embodiments, two scene regions each including the at least one target candidate instance may be randomly determined from the first image sample, and a first scene image and a second scene image may be determined based on images of the two scene regions in the first image sample.

[0207] Step S304: Using the first model to be trained, feature extraction is performed on the first scene image and the second scene image respectively to obtain a first scene feature of the first scene image and a second scene feature of the second scene image; and using the second model, feature extraction is performed on the first scene image and the second scene image respectively to obtain a third scene feature of the first scene image and a fourth scene feature of the second scene image.

[0208] Step S305: determining a target loss value based on the first scene feature and the second scene feature, and the third scene feature and the fourth scene feature.

[0209] Step S306: Based on the target loss value, update the model parameters of the first model at least once to obtain the trained first model.

[0210] Here, the above steps S304 to S306 correspond to the above steps S102 to S104 respectively, and the implementation of the above steps S102 to S104 may be referred to.

[0211] In the embodiment of the present application, at least one candidate instance in the first image sample is obtained, at least one target candidate instance is determined from the at least one candidate instance, and based on the first image sample and the at least one target candidate instance, a first scene image and a second scene image having an overlapping area are determined, wherein the overlapping area includes the at least one target candidate instance. In this way, the first scene image and the second scene image having an overlapping area can be quickly determined.

[0212] In some embodiments, the above step S301 may include:

[0213] Step S311: perform target detection on the first image sample in an unsupervised manner to obtain at least one candidate instance.

[0214] Here, any suitable unsupervised algorithm can be used to implement unsupervised object detection on the first image sample, so as to reduce the labor cost in the model training process.

[0215] In some embodiments, the above step S303 may include the following steps S321 to S323:

[0216] Step S321 : performing a first image augmentation process and a second image augmentation process on the first image sample respectively to obtain a first augmented image and a second augmented image.

[0217] Here, the first image augmentation processing and the second image augmentation processing may include but are not limited to at least one of random scaling, random cropping, random flipping, random resizing, color jittering, grayscale processing, Gaussian blurring, random erasing, etc. The augmentation methods used in the first image augmentation processing and the second image augmentation processing may be the same or different. During implementation, those skilled in the art may select a suitable method to perform the first augmentation processing and the second augmentation processing on the first image sample according to actual conditions, and the embodiments of the present application are not limited thereto.

[0218] Step S322: Acquire a first scene image including the at least one target candidate instance from the first augmented image.

[0219] Step S323: Acquire a second scene image including the at least one target candidate instance from the second augmented image.

[0220] Here, the first scene image and the second scene image including at least one target candidate instance can be obtained from the first augmented image and the second augmented image respectively in an appropriate manner according to the actual situation, which is not limited here. For example, the first scene image and the second scene image including at least one target candidate instance can be obtained by cropping from the first augmented image and the second augmented image respectively according to the set scene image size. The scene image size may include but is not limited to the area, aspect ratio, etc. of the image.

[0221] In the above embodiment, the first augmented image and the second augmented image are obtained by performing the first image augmentation processing and the second image augmentation processing on the first image sample respectively, and the first scene image and the second scene image including at least one target candidate instance are obtained from the first augmented image and the second augmented image respectively. In this way, there can be a difference between the first scene image and the second scene image, so that the learning capabilities of the first model and the second model can be differentiated, so as to reduce the collapse of model training and improve the learning ability of the trained first model.

[0222] An embodiment of the present application provides a model training method, which can be executed by a processor of a computer device. Figure 4 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method includes the following steps S401 to S406:

[0223] Step S401: determining a first scene image and a second scene image based on a first image sample; the first scene image and the second scene image have an overlapping area.

[0224] Step S402: Using the first model to be trained, feature extraction is performed on the first scene image and the second scene image respectively to obtain a first scene feature of the first scene image and a second scene feature of the second scene image; and using the second model, feature extraction is performed on the first scene image and the second scene image respectively to obtain a third scene feature of the first scene image and a fourth scene feature of the second scene image.

[0225] Step S403: determining a target loss value based on the first scene feature, the second scene feature, the third scene feature, and the fourth scene feature.

[0226] Step S404: Based on the target loss value, update the model parameters of the first model at least once to obtain the trained first model.

[0227] Here, the above steps S401 to S404 correspond to the above steps S101 to S104 respectively, and the implementation of the above steps S101 to S104 may be referred to.

[0228] Step S405: determine an initial third model based on the trained first model.

[0229] Here, in some embodiments, the output layer in the trained first model can be adjusted according to the actual business scenario, and the adjusted first model can be determined as the initial third model. For example, when the task in the current business scenario is an image classification task, the output layer in the first model can be replaced with a feedforward neural network for image classification; when the task in the current business scenario is an image segmentation task, the output layer in the first model can be replaced with a feedforward neural network for image segmentation.

[0230] Step S406: based on at least one second image sample, update the model parameters of the third model to obtain the trained third model.

[0231] Here, the second image sample may have annotation information or may not have annotation information, and may be a single-target image or a multi-target image. During implementation, those skilled in the art may determine a suitable second image sample according to an actual business scenario, which is not limited here.

[0232] In some implementations, model parameters of the third model may be fine-tuned and trained based on at least one second image sample to obtain the trained third model.

[0233] In the embodiment of the present application, an initial third model is determined based on the trained first model, and the model parameters of the third model are updated based on at least one second image sample to obtain a trained third model. In this way, the model parameters of the trained first model can be migrated to a model for performing other tasks to be applied to a variety of business scenarios, and the training efficiency of the third model and the performance of the trained third model can be improved.

[0234] An embodiment of the present application provides an image processing method, which can be executed by a processor of a computer device. Figure 5 A schematic diagram of the implementation flow of a model training method provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the method includes the following steps S501 to S502:

[0235] Step S501, obtaining an image to be processed;

[0236] Step S502, using the trained fourth model to process the image to be processed to obtain a processing result; wherein the fourth model includes at least one of the following: a first model obtained by the model training method described in the above embodiment, and a third model obtained by the model training method described in the above embodiment.

[0237] Here, the image to be processed may be any suitable image. During implementation, those skilled in the art may select a suitable image to be processed according to the actual application scenario, and the embodiments of the present application are not limited thereto.

[0238] The processing of the image to be processed using the fourth model may be determined according to the actual application scenario, and may include but is not limited to at least one of classification processing, target detection processing, image segmentation processing, etc. For example, in an industrial defect detection scenario, the target detection processing may include but is not limited to nut defect detection processing, bolt defect detection processing, cotter pin defect detection processing, etc.

[0239] In some implementations, the above step S502 may include: using the trained fourth model to perform at least one of the following processing on the image to be processed: classification processing, target detection processing, image segmentation processing, to obtain a processing result.

[0240] In an embodiment of the present application, since the model training method described in the above embodiment can achieve a self-supervised training process on the first image sample by maintaining the consistency between the first scene feature and the second scene feature, and the third scene feature and the fourth scene feature, obtained after the first model and the second model process the first scene image and the second scene image with overlapping areas, respectively, the performance of the first model after training is improved, and it is applicable to model training on both single-target images and multi-target images, it can greatly reduce the high requirements and high dependence on the quality of sample data in the training process, and has better versatility. Therefore, based on the first model and / or the third model obtained by the model training method described in the above embodiment, classification processing, target detection processing and / or image segmentation processing, etc. are performed on the processed image, and the accuracy of the processing results can be improved.

[0241] The embodiment of the present application provides a self-supervised pre-training method for a general visual model, which can realize a self-supervised learning process of multi-granularity visual representation using single target images and / or multi-target images. The method can be trained using any type of image data, thereby improving the robustness of self-supervised training, and the trained model has better performance and can be applied to a variety of application scenario tasks. Fig. 6AA schematic diagram of the implementation process of a self-supervised pre-training method provided in an embodiment of the present application is shown in FIG. Fig. 6A As shown, the method may include the following steps S601 to S610:

[0242] Step S601: determining at least one candidate instance from a first image sample.

[0243] In implementation, any suitable unsupervised detection algorithm may be used to detect the target instance in the first image sample to obtain at least one candidate instance. For example, a selective search algorithm may be used to unsupervisedly obtain at least one candidate instance with a high recall rate from the first image sample.

[0244] Step S602, randomly determining a first scene area and a second scene area with an overlapping area from the first image sample, and determining a first scene image and a second scene image based on the first scene area and the second scene area, respectively; wherein the overlapping area includes a set number of candidate instances.

[0245] Here, there is an overlapping area between the first scene image and the second scene image. Figure 6B A schematic diagram of determining a first scene image and a second scene image provided in an embodiment of the present application, such as Figure 6B As shown, a first scene area s1 and a second scene area s2 having overlapping areas are randomly determined from a first image sample p, and a first scene image s1' and a second scene image s2' are determined based on the first scene area s1 and the second scene area s2, respectively. The first scene image s1' and the second scene image s2' both contain candidate instances b1 and b2, and there is similarity between the first scene image s1' and the second scene image s2'. Compared to determining two scene images from a first image sample in a completely random manner, when the first image sample is a multi-target image, the two scene images determined in a completely random manner may have semantic inconsistencies due to their long distances, thereby causing the model to be unable to learn. However, the first scene image and the second scene image determined in the self-supervised pre-training method provided in the embodiment of the present application have global similarity, and can be applied to the learning of single-target images and / or multi-target images.

[0246] Step S603: Using the first model to be trained, feature extraction is performed on the first scene image and the second scene image respectively to obtain a first scene feature of the first scene image and a second scene feature of the second scene image; and using the second model, feature extraction is performed on the first scene image and the second scene image respectively to obtain a third scene feature of the first scene image and a fourth scene feature of the second scene image.

[0247] Step S604: acquiring a first instance image corresponding to each candidate instance in the overlapping area from the first scene image, and acquiring a second instance image corresponding to each candidate instance in the overlapping area from the second scene image.

[0248] For example, see Figure 6B , a first instance image b2' corresponding to the candidate instance b2 in the overlapping area can be obtained from the first scene image s1', and a second instance image b1' corresponding to the candidate instance b1 in the overlapping area can be obtained from the second scene image s2'. The first instance image b2' and the second instance image b1' correspond to co-occurring but different instances, and the first scene image s1' and the first instance image b2', as well as the second scene image s2' and the second instance image b1' are related. Therefore, the semantic affinity between natural scenes and instances in the scenes can be utilized during the training process, and co-occurring but different instances in the images can be distinguished.

[0249] Step S605, using the first model, performing feature extraction on each first instance image to obtain the first instance feature of each candidate instance in the first scene image, and using the second model, performing feature extraction on each second instance image to obtain the second instance feature of each candidate instance in the second scene image.

[0250] Step S606, determining the target scene loss value based on the similarity between the first scene feature and the fourth scene feature, and the similarity between the second scene feature and the third scene feature.

[0251] Step S607, connect each first instance feature together and perform linear mapping to obtain a mapping feature with the same dimension as the first scene image, and determine the first instance loss value based on the similarity between the third scene feature and the mapping feature and the similarity between the fourth scene feature and the mapping feature.

[0252] Step S608: Based on the optimal transmission algorithm, determine the optimal transmission loss between each first instance feature and each second instance feature, and determine the optimal transmission loss as the second instance loss value.

[0253] Step S609, determining a target loss value based on the target scene loss value, the first instance loss value and the second instance loss value.

[0254] Step S610: Based on the target loss value, update the first model and the second model at least once to obtain a trained first model.

[0255] Figure 6C A schematic diagram of an implementation architecture of a self-supervised pre-training method provided in an embodiment of the present application is shown in FIG. Figure 6CAs shown, the implementation of the method includes scene instance extraction 10, feature extraction 20 and loss calculation 30; wherein:

[0256] Through scene instance extraction 10, at least one candidate instance i can be determined from the first image sample p, and a first scene area s3 and a second scene area s4 having overlapping areas can be randomly determined from the first image sample p, and a first scene image s3' and a second scene image s4' can be respectively determined based on the first scene area s3 and the second scene area s4, wherein the first scene image s3' and the second scene image s4' both contain candidate instances i1 and i2, and first instance images i1' and i2' corresponding to the candidate instances i1 and i2 are respectively extracted from the first scene image s3', and second instance images i1" and i2" corresponding to the candidate instances i1 and i2 are respectively extracted from the second scene image s4';

[0257] Through feature extraction 20, the first scene image s3' and the second scene image s4' can be input into the first model M1 and the second model M2 respectively to obtain the first scene feature f of the first scene image s3' output by the first model M1. o1 and the second scene feature f of the second scene image s4′ o2 , and the third scene feature f of the first scene image s3' output by the second model M2 t1 and the fourth scene feature f of the second scene image s4′ t2 ;

[0258] Through feature extraction 20, the first instance images i1' and i2' can also be input into the first model M1 to obtain first instance features o1 and o2 of the candidate instances i1 and i2 in the first scene image respectively, and the second instance images i1" and i2" can be input into the second model M2 to obtain second instance features t1 and t2 of the candidate instances i1 and i2 in the second scene image respectively;

[0259] Through the loss calculation 30, based on the first scene feature f o1 and the fourth scene feature f t2 The similarity between the second scene feature f o2 and the third scene feature f t1 The similarity between them determines the target scene loss value L s-s ; Connect each first instance feature o1 and o2 together and perform linear mapping to obtain a mapping feature f with the same dimension as the first scene image c , and based on the third scene feature f t1 And the mapping feature f c The similarity between the fourth scene feature f t2 And the mapping feature f c The similarity between them determines the loss value L of the first instances-i Based on the optimal transmission algorithm, determine the optimal transmission loss between the first instance features o1 and o2 and the second instance features t1 and t2, and determine the optimal transmission loss as the second instance loss value L i-i ; Set the target scene loss value L s-s , the first instance loss value L s-i and the second instance loss value L i-i Add them together to get the target loss value.

[0260] In some embodiments, the self-supervised pre-training method provided in the embodiments of the present application can be applied to industrial visual defect detection tasks. On the one hand, it does not require labeling of the training data. On the other hand, it greatly improves the accuracy of industrial visual defect detection and provides effective assistance to later maintenance personnel in managing faulty components.

[0261] In the embodiment of the present application, a self-supervised algorithm can be used to train a large amount of unlabeled data, which can greatly reduce the high requirements and high dependence of self-supervised training on the quality of training data, and enable the trained model to have the ability to extract multi-granular features, thereby improving the generalization ability of the model, so that the trained model parameters can be migrated to the models of various downstream tasks. In addition, by introducing the semantic affinity between the instances and the scenes in the overlapping area of ​​the two scene areas for representation learning, the representation of the scene can be focused on the instances in the scene and ignore the background, thereby improving the performance of the trained model.

[0262] Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the model training device 700 includes: a first determination module 710, a first extraction module 720, a second determination module 730 and a first update module 740, wherein:

[0263] A first determination module 710 is used to determine a first scene image and a second scene image based on a first image sample; the first scene image and the second scene image have an overlapping area;

[0264] A first extraction module 720 is used to perform feature extraction on the first scene image and the second scene image respectively using the first model to be trained to obtain a first scene feature of the first scene image and a second scene feature of the second scene image, and to perform feature extraction on the first scene image and the second scene image respectively using the second model to obtain a third scene feature of the first scene image and a fourth scene feature of the second scene image;

[0265] A second determination module 730, configured to determine a target loss value based on the first scene feature and the second scene feature, and the third scene feature and the fourth scene feature;

[0266] The first updating module 740 is used to update the model parameters of the first model at least once based on the target loss value to obtain the trained first model.

[0267] In some embodiments, the second determination module is further used to: determine a target scene loss value based on the first scene feature and the second scene feature, and the third scene feature and the fourth scene feature; determine a target loss value based on the target scene loss value.

[0268] In some embodiments, the second determination module is also used to: determine a first scene loss value based on the similarity between the first scene feature and the fourth scene feature; determine a second scene loss value based on the similarity between the second scene feature and the third scene feature; determine the target scene loss value based on the first scene loss value and the second scene loss value.

[0269] In some embodiments, the overlapping area includes at least one candidate instance; the second determination module is also used to: determine a target loss value based on the target scene loss value and the target instance loss value; wherein the target instance loss value includes at least one of the following: a first instance loss value and a second instance loss value; when the target instance loss value includes the first instance loss value, the first instance loss value is determined based on the third scene feature and the fourth scene feature, and the first instance feature of each candidate instance in the first scene image; each of the first instance features is extracted using the first model; when the target instance loss value includes the second instance loss value, the second instance loss value is determined based on each of the first instance features and the second instance feature of each candidate instance in the second scene image; each of the second instance features is extracted using the second model.

[0270] In some embodiments, the third scene feature and the fourth scene feature have the same dimension; the second determination module is also used to: connect each of the first instance features to obtain a connection feature; map the connection feature to the dimension of the third scene feature to obtain a mapping feature; and determine the first instance loss value based on the third scene feature, the fourth scene feature and the mapping feature.

[0271] In some embodiments, the second determination module is also used to: determine a first sub-loss value based on the similarity between the third scene feature and the mapping feature; determine a second sub-loss value based on the similarity between the fourth scene feature and the mapping feature; and determine a first instance loss value based on the first sub-loss value and the second sub-loss value.

[0272] In some embodiments, the second determination module is also used to: match each of the first instance features and each of the second instance features to obtain at least one pair of first instance features and second instance features having a target matching relationship; and determine a second instance loss value based on each pair of first instance features and second instance features having a target matching relationship.

[0273] In some embodiments, the second determination module is also used to: match each of the first instance features and each of the second instance features based on an optimal transmission algorithm to obtain at least one pair of first instance features and second instance features having a target matching relationship; determine the second instance loss value based on the transmission cost between each pair of first instance features and second instance features having a target matching relationship.

[0274] In some embodiments, the device also includes: a first acquisition module, used to: acquire the first instance image corresponding to each candidate instance in the overlapping area from the first scene image; a second extraction module, used to: use the first model to extract features from each first instance image to obtain each first instance feature.

[0275] In some embodiments, the first acquisition module is also used to: acquire a first candidate area image corresponding to each candidate instance in the overlapping area from the first scene image; adjust the size of each first candidate area image to a target size to obtain a first instance image corresponding to each candidate instance.

[0276] In some embodiments, when the target instance loss value includes the second instance loss value, the device also includes: a second acquisition module, used to: acquire the second instance image corresponding to each candidate instance in the overlapping area from the second scene image; a third extraction module, used to: use the second model to extract features from each second instance image to obtain each second instance feature.

[0277] In some embodiments, the device further includes: a new adding module, which is used to determine at least one new candidate instance in the overlapping area when the number of candidate instances included in the overlapping area is less than a set number, so that the overlapping area includes the set number of candidate instances.

[0278] In some embodiments, the first updating module is further used to update the model parameters of the first model to obtain an updated first model when the target loss value does not meet a preset condition.

[0279] In some embodiments, the first updating module is further used to update the model parameters of the first model and the model parameters of the second model respectively to obtain an updated first model and an updated second model.

[0280] In some embodiments, the first update module is further used to: based on the current model parameters of the first model, momentum update the model parameters of the second model to obtain an updated second model; and update the current model parameters of the first model using a gradient update method to obtain an updated first model.

[0281] In some embodiments, the first determination module is also used to: obtain at least one candidate instance in the first image sample; determine at least one target candidate instance from at least one of the candidate instances; and determine a first scene image and a second scene image having an overlapping area based on the first image sample and the at least one target candidate instance, wherein the overlapping area includes the at least one target candidate instance.

[0282] In some embodiments, the first determination module is further used to: perform target detection on the first image sample in an unsupervised manner to obtain at least one candidate instance.

[0283] In some embodiments, the first determination module is further used to: perform first image augmentation processing and second image augmentation processing on the first image sample, respectively, to obtain a first augmented image and a second augmented image; obtain a first scene image including the at least one target candidate instance from the first augmented image; and obtain a second scene image including the at least one target candidate instance from the second augmented image.

[0284] In some embodiments, the device also includes: a third determination module, used to determine an initial third model based on the trained first model; a second updating module, used to update the model parameters of the third model based on at least one second image sample to obtain the trained third model.

[0285] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the image processing device 800 includes: a third acquisition module 810 and a processing module 820, wherein:

[0286] The third acquisition module 810 is used to acquire the image to be processed;

[0287] The processing module 820 is used to use the trained fourth model to process the image to be processed to obtain a processing result; wherein the fourth model includes at least one of the following: a first model obtained by the model training method described in the above embodiment, and a third model obtained by the model training method described in the above embodiment.

[0288] In some embodiments, the processing module is further used to use the trained fourth model to perform at least one of the following processing on the image to be processed: classification processing, target detection processing, image segmentation processing, and obtain a processing result.

[0289] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0290] It should be noted that in the embodiment of the present application, if the above-mentioned model training method or image processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0291] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps in the above method when executing the program.

[0292] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0293] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

[0294] It should be noted here that the description of the above storage medium, computer program product and device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the storage medium, computer program product and device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0295] It should be noted that Fig. 9 A schematic diagram of a hardware entity of a computer device in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the hardware entity of the computer device 900 includes: a processor 901, a communication interface 902 and a memory 903, wherein:

[0296] Processor 901 generally controls the overall operation of computer device 900 .

[0297] The communication interface 902 enables the computer device to communicate with other terminals or servers through a network.

[0298] The memory 903 is configured to store instructions and applications executable by the processor 901, and can also cache data to be processed or processed by the processor 901 and each module in the computer device 900 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission between the processor 901, the communication interface 902 and the memory 903 can be carried out through the bus 904.

[0299] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0300] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0301] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0302] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0303] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0304] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0305] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0306] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Determine a first scene image and a second scene image based on the first image sample; the first scene image and the second scene image have an overlapping area; Using the first model to be trained, feature extraction is performed on the first scene image and the second scene image respectively to obtain a first scene feature of the first scene image and a second scene feature of the second scene image, and using the second model, feature extraction is performed on the first scene image and the second scene image respectively to obtain a third scene feature of the first scene image and a fourth scene feature of the second scene image; the network structure of the first model includes at least: a backbone network, a projection head and a prediction head; the network structure of the second model includes at least: a backbone network and a projection head; Determining a target loss value based on the first scene feature and the second scene feature, and the third scene feature and the fourth scene feature; Based on the target loss value, the model parameters of the first model are updated at least once to obtain the trained first model.

2. The method according to claim 1, characterized in that The determining a target loss value based on the first scene feature and the second scene feature, and the third scene feature and the fourth scene feature includes: Determine a target scene loss value based on the first scene feature and the second scene feature, and the third scene feature and the fourth scene feature; Based on the target scene loss value, a target loss value is determined.

3. The method according to claim 2, characterized in that The determining the target scene loss value based on the first scene feature and the second scene feature, and the third scene feature and the fourth scene feature includes: Determining a first scene loss value based on a similarity between the first scene feature and the fourth scene feature; Determining a second scene loss value based on a similarity between the second scene feature and the third scene feature; Based on the first scene loss value and the second scene loss value, the target scene loss value is determined.

4. The method according to claim 2 or 3, characterized in that: The overlapping area includes at least one candidate instance; The determining a target loss value based on the target scene loss value includes: Determine a target loss value based on the target scene loss value and the target instance loss value; wherein the target instance loss value includes at least one of the following: a first instance loss value and a second instance loss value; The determining of the target loss value based on the first scene feature and the second scene feature, and the third scene feature and the fourth scene feature, further includes: In a case where the target instance loss value includes the first instance loss value, determining the first instance loss value based on the third scene feature and the fourth scene feature, and a first instance feature of each of the candidate instances in the first scene image; each of the first instance features is extracted using the first model; In the case where the target instance loss value includes the second instance loss value, the second instance loss value is determined based on each of the first instance features and the second instance features of each of the candidate instances in the second scene image; each of the second instance features is extracted using the second model.

5. The method according to claim 4, characterized in that The third scene feature and the fourth scene feature have the same dimension; and determining the first instance loss value based on the third scene feature and the fourth scene feature and the first instance feature of each candidate instance in the first scene image includes: Connecting each of the first instance features to obtain a connected feature; Mapping the connection feature to the dimension of the third scene feature to obtain a mapping feature; Based on the third scene feature, the fourth scene feature and the mapping feature, a first instance loss value is determined.

6. The method according to claim 5, characterized in that The determining the first instance loss value based on the third scene feature, the fourth scene feature and the mapping feature comprises: Determining a first sub-loss value based on a similarity between the third scene feature and the mapping feature; Determining a second sub-loss value based on a similarity between the fourth scene feature and the mapping feature; Based on the first sub-loss value and the second sub-loss value, a first instance loss value is determined.

7. The method according to claim 5 or 6, characterized in that: The determining the second instance loss value based on each first instance feature and a second instance feature of each candidate instance in the second scene image comprises: Matching each of the first instance features and each of the second instance features to obtain at least one pair of first instance features and second instance features having a target matching relationship; Based on each pair of the first instance feature and the second instance feature having a target matching relationship, a second instance loss value is determined.

8. The method according to claim 7, characterized in that The matching of each of the first instance features and each of the second instance features to obtain at least one pair of first instance features and second instance features having a target matching relationship includes: Based on the optimal transmission algorithm, each of the first instance features and each of the second instance features are matched to obtain at least one pair of first instance features and second instance features having a target matching relationship; The determining of the second instance loss value based on each pair of the first instance feature and the second instance feature having a target matching relationship comprises: A second instance loss value is determined based on a transmission cost between each pair of first instance features and second instance features having a target matching relationship.

9. The method according to claim 8, characterized in that The method further comprises: Acquire a first instance image corresponding to each candidate instance in the overlapping area from the first scene image; Using the first model, feature extraction is performed on each of the first instance images to obtain each of the first instance features.

10. The method according to claim 9, characterized in that The acquiring, from the first scene image, a first instance image corresponding to each candidate instance in the overlapping area comprises: Acquire a first candidate region image corresponding to each candidate instance in the overlapping region from the first scene image; The size of each of the first candidate region images is adjusted to a target size to obtain a first instance image corresponding to each of the candidate instances.

11. The method according to any one of claims 8 to 10, characterized in that In a case where the target instance loss value includes the second instance loss value, the method further includes: Acquire, from the second scene image, a second instance image corresponding to each candidate instance in the overlapping area; Using the second model, feature extraction is performed on each second instance image to obtain each second instance feature.

12. The method according to any one of claims 8 to 10, characterized in that The method further comprises: When the number of candidate instances included in the overlapping region is less than a set number, at least one newly added candidate instance is determined in the overlapping region so that the overlapping region includes the set number of candidate instances.

13. The method according to any one of claims 8 to 10, characterized in that The updating of the model parameters of the first model at least once based on the target loss value to obtain the trained first model includes: When the target loss value does not meet the preset conditions, the model parameters of the first model are updated to obtain an updated first model.

14. The method according to claim 13, characterized in that The updating of the model parameters of the first model to obtain an updated first model includes: The model parameters of the first model and the model parameters of the second model are updated respectively to obtain an updated first model and an updated second model.

15. The method according to claim 14, characterized in that The updating of the model parameters of the first model and the model parameters of the second model respectively to obtain an updated first model and an updated second model comprises: Based on the current model parameters of the first model, momentum update the model parameters of the second model to obtain an updated second model; The current model parameters of the first model are updated by using a gradient update method to obtain an updated first model.

16. The method according to claim 14 or 15, characterized in that The determining of the first scene image and the second scene image based on the first image sample comprises: Acquire at least one candidate instance in the first image sample; Determine at least one target candidate instance from at least one of the candidate instances; Based on the first image sample and the at least one target candidate instance, a first scene image and a second scene image having an overlapping area are determined, wherein the overlapping area includes the at least one target candidate instance.

17. The method according to claim 16, characterized in that The obtaining of at least one candidate instance in the first image sample comprises: An unsupervised approach is used to perform target detection on the first image sample to obtain at least one candidate instance.

18. The method according to claim 17, characterized in that The determining, based on the first image sample and the at least one target candidate instance, of a first scene image and a second scene image having an overlapping area, wherein the overlapping area includes the at least one target candidate instance, comprises: Performing a first image augmentation process and a second image augmentation process on the first image sample respectively to obtain a first augmented image and a second augmented image; Acquire a first scene image including the at least one target candidate instance from the first augmented image; A second scene image including the at least one target candidate instance is acquired from the second augmented image.

19. The method according to claim 17 or 18, characterized in that The method further comprises: Determine an initial third model based on the trained first model; Based on at least one second image sample, the model parameters of the third model are updated to obtain the trained third model.

20. An image processing method, characterized in that: The method comprises: Get the image to be processed; The image to be processed is processed using the trained fourth model to obtain a processing result; wherein the fourth model includes at least one of the following: a first model obtained by the model training method described in any one of claims 1 to 18, and a third model obtained by the model training method described in claim 19.

21. The method according to claim 20, characterized in that The using the trained fourth model to process the image to be processed to obtain a processing result includes: Using the trained fourth model, at least one of the following processes is performed on the image to be processed: classification processing, target detection processing, and image segmentation processing to obtain a processing result.

22. A model training device, characterized in that: include: A first determining module, configured to determine a first scene image and a second scene image based on the first image sample; There is an overlapping area between the first scene image and the second scene image; a first extraction module, configured to perform feature extraction on the first scene image and the second scene image respectively using the first model to be trained to obtain a first scene feature of the first scene image and a second scene feature of the second scene image, and to perform feature extraction on the first scene image and the second scene image respectively using the second model to obtain a third scene feature of the first scene image and a fourth scene feature of the second scene image; The network structure of the first model at least includes: a backbone network, a projection head and a prediction head; the network structure of the second model at least includes: a backbone network and a projection head; A second determination module, configured to determine a target loss value based on the first scene feature and the second scene feature, and the third scene feature and the fourth scene feature; The first updating module is used to update the model parameters of the first model at least once based on the target loss value to obtain the trained first model.

23. An image processing device, characterized in that: include: A third acquisition module is used to acquire the image to be processed; The second detection module is used to use the trained fourth model to perform target detection on the image to be processed to obtain a detection result; wherein the fourth model includes at least one of the following: a first model obtained by using the model training method described in any one of claims 1 to 18, and a third model obtained by using the model training method described in claim 19.

24. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 21 are implemented.

25. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 21 are implemented.

26. A computer program product, comprising a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, the steps of the method according to any one of claims 1 to 21 are implemented.

Images