Model determination method, device, electronic device and memory

Abstract

The present disclosure provides a model determination method, device, electronic device and memory, which relate to the field of computer technology, in particular to the field of computer vision and deep learning, and can be applied to scenes such as image processing and image recognition. The specific implementation scheme is: obtaining a first image sample and a first text sample; training the first image sample and the first text sample to obtain a first target model, wherein the first target model learns the local features of the first text sample; obtaining a second image sample and a second text sample, and training the first target model based on the second image sample and the second text sample to obtain a second target model, wherein the second target model learns the global features of the second text sample; determining the second target model as the initialization model of the third target model, and solving the technical problem that the training effect of the initialization model is low.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the fields of computer vision and deep learning, and can be applied to scenes such as image processing and image recognition, and specifically to a model determination method, device, electronic device and memory. Background Art

[0002] At present, the pre-training scheme for the model is usually implemented by a one-step training method, that is, the image samples and text samples are directly input into the pre-training network to perform the pre-training task, which makes the training index of the initialization model not high. Summary of the invention

[0003] The present disclosure provides a model determination method, device, electronic device and memory.

[0004] According to one aspect of the present disclosure, a model determination method is provided. The method may include: obtaining a first image sample and a first text sample, wherein the text in the first text sample is used to describe the target image in the first image sample; training the first image sample and the first text sample to obtain a first target model, wherein the first target model learns the local features of the first text sample; obtaining a second image sample and a second text sample, and training the first target model based on the second image sample and the second text sample to obtain a second target model, wherein the text in the second text sample is used to describe the target image in the second image sample, and the second target model learns the global features of the second text sample; determining the second target model as the initialization model of the third target model.

[0005] According to one aspect of the present disclosure, another model determination method is provided. The method may include: sending a model training request to a server, wherein the model training request includes a first image sample and a first text sample, wherein the text in the first text sample is used to describe the target image in the first image sample; receiving an initialization model sent by the server in response to the model training request, wherein the initialization model is obtained by the server training the first target model based on the second image sample and the second text sample, the first target model is obtained by the server training the first image sample and the first text sample, the first target model learns the local features of the first text sample, the text in the second text sample is used to describe the target image in the second image sample, and the initialization model learns the global features of the second text sample.

[0006] According to one aspect of the present disclosure, another image processing method is provided, including: obtaining an image to be processed; inputting the image to be processed into a third target model, wherein the third target model is obtained according to a model determination method according to an embodiment of the present disclosure; and obtaining a processing result of the third target model.

[0007] According to one aspect of the present disclosure, a model determination device is provided. The device may include: a first acquisition unit, used to acquire a first image sample and a first text sample, wherein the text in the first text sample is used to describe the target image in the first image sample; a training unit, used to train the first image sample and the first text sample to obtain a first target model, wherein the first target model learns the local features of the first text sample; a processing unit, used to acquire a second image sample and a second text sample, and train the first target model based on the second image sample and the second text sample to obtain a second target model, wherein the text in the second text sample is used to describe the target image in the second image sample, and the second target model learns the global features of the second text sample; a determination unit, used to determine the second target model as the initialization model of the third target model.

[0008] According to one aspect of the present disclosure, another model determination device is also provided. The device may include: a sending unit, used to send a model training request to a server, wherein the model training request includes a first image sample and a first text sample, wherein the text in the first text sample is used to describe the target image in the first image sample; a receiving unit, used to receive an initialization model sent by the server in response to the model training request, wherein the initialization model is obtained by the server training the first target model based on the second image sample and the second text sample, the first target model is obtained by the server training the first image sample and the first text sample, the first target model learns the local features of the first text sample, the text in the second text sample is used to describe the target image in the second image sample, and the initialization model learns the global features of the second text sample.

[0009] According to one aspect of the present disclosure, an image processing device is also provided. The device may include: a third acquisition unit, used to acquire an image to be processed; an input unit, used to input the image to be processed into a third target model, wherein the third target model is obtained by the model determination method of an embodiment of the present disclosure; and a fourth acquisition unit, used to acquire a processing result of the third target model.

[0010] According to one aspect of the present disclosure, an electronic device is also provided, which may include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the model determination method of an embodiment of the present disclosure.

[0011] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable a computer to execute the model determination method of an embodiment of the present disclosure.

[0012] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the model determination method of the embodiment of the present disclosure is implemented.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1A is a flow chart of a model determination method according to an embodiment of the present disclosure;

[0016] Figure 1B is a flow chart of another model determination method according to an embodiment of the present disclosure;

[0017] Figure 1C is a flow chart of an image processing method according to an embodiment of the present disclosure;

[0018] Figure 2 is a schematic diagram of a multi-stage image and text pre-training system implemented according to the present disclosure;

[0019] Figure 3 is a schematic diagram of a Deit model structure according to an embodiment of the present disclosure;

[0020] Figure 4 is a schematic diagram of matching image features and text features according to an embodiment of the present disclosure;

[0021] Figure 5A is a schematic diagram of a model determination device according to an embodiment of the present disclosure;

[0022] Figure 5B is a schematic diagram of another model determination device according to an embodiment of the present disclosure;

[0023] Figure 5C is a schematic diagram of an image processing device according to an embodiment of the present disclosure;

[0024] Figure 6 is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] Figure 1A FIG. 1 is a flow chart of a model determination method according to an embodiment of the present disclosure. Figure 1A As shown, the method may include the following steps:

[0027] Step S102: Acquire a first image sample and a first text sample.

[0028] In the technical solution provided in the above step 102 of the present disclosure, the text (text data) in the first text sample is used to describe the target image (image data) in the first image sample.

[0029] The model determination method of this embodiment is a model determination method for image and text pre-training, which can include two stages. In the first stage, image and text pre-training requires a large amount of data. This embodiment can obtain a first image sample and a first text sample as training samples for the first stage. The first text sample corresponds to the first image sample, wherein the first text sample may include a large amount of text, the first image sample may include a large amount of images, and the image may include a picture. Each text can be used to describe the target image of the large amount of images in the first image sample, that is, each text in the first text sample corresponds one-to-one to the target image in the first image sample, and each text in the first text sample and the corresponding target image can also be called an image-text pair.

[0030] Optionally, this embodiment may crawl the first image sample and the first text sample by using an Internet crawler.

[0031] Optionally, the first image sample and the first text sample of this embodiment may not require manual annotation and cleaning to save labor costs.

[0032] Optionally, the above method can be implemented in a data processing module of the first stage.

[0033] Step S104: training the first image sample and the first text sample to obtain a first target model.

[0034] In the technical solution provided in the above step 104 of the present disclosure, after obtaining the first image sample and the first text sample, the first image sample and the first text sample can be trained to obtain a first target model, wherein the first target model learns local features of the first text sample.

[0035] In the first stage of this embodiment, the first image sample and the first text sample may be trained, and machine learning training may be performed on the first image sample and the first text sample, for example, contrastive learning training may be performed on the first image sample and the first text sample to obtain a first target model, and the first target model may learn some local features of the first text sample.

[0036] Step S106: Acquire a second image sample and a second text sample, and train the first target model based on the second image sample and the second text sample to obtain a second target model.

[0037] In the technical solution provided in the above step 106 of the present disclosure, after the first image sample and the first text sample are trained to obtain the first target model, the text in the second text sample is used to provide a textual description of the target image in the second image sample, and the second target model learns the global features of the second text sample.

[0038] The model determination method of this embodiment includes a second stage. In the second stage, a second image sample and a second text sample are obtained as training samples of the second stage, and the second text sample corresponds to the second image sample, wherein the second text sample may include a large number of texts, the second image sample may include a large number of images, and the image may include a picture, and each text may be used to describe the target image of the large number of images in the second image sample, that is, each text in the second text sample corresponds to the target image in the second image sample one by one, and each text in the second text sample and the corresponding target image may also be called an image-text pair.

[0039] Optionally, in this embodiment, the second image sample and the second text sample may be crawled by an Internet crawler.

[0040] Optionally, the above method can be implemented in the data processing module of the second stage.

[0041] This embodiment can continue to train the first target model based on the second image sample and the second text sample to obtain the second target model. It can perform machine learning training on the first target model based on the second image sample and the second text sample. For example, comparative learning training is performed on the first image sample and the first text sample to obtain the second target model. The second target model can learn the global features of the second text sample, which is also the overall features of the second text sample.

[0042] Step S108: determine the second target model as the initialization model of the third target model.

[0043] In the technical solution provided in the above step 108 of the present disclosure, after the first target model is trained based on the second image sample and the second text sample to obtain the second target model, the second target model is determined as the initialization model of the third target model.

[0044] In this embodiment, the second target model is determined as the initialization model of the third target model, and the initialization model is used to train the third target model. The third target model can be an image detection model, an image segmentation model, an image classification model, etc.

[0045] It should be noted that the above-mentioned third target model of this embodiment is an image detection model, an image segmentation model, and an image classification model. It is only an example of the embodiment of the present disclosure, and does not mean that the third target model of the embodiment of the present disclosure is only an image detection model, an image segmentation model, and an image classification model. Any model that can be obtained by training the initialization model is within the scope of this embodiment, and no examples are given one by one here.

[0046] Since the initialization model of this embodiment not only takes into account the local features of the first text sample but also takes into account the global features of the second text sample during the generation process, the pre-training index is improved. The pre-training index refers to an index used to represent the training effect of the initialization model.

[0047] Through the above steps S102 to S108 of the present application, a first image sample and a first text sample are obtained, wherein the text in the first text sample is used to describe the target image in the first image sample; the first image sample and the first text sample are trained to obtain a first target model, wherein the first target model learns the local features of the first text sample; a second image sample and a second text sample are obtained, and the first target model is trained based on the second image sample and the second text sample to obtain a second target model, wherein the text in the second text sample is used to describe the target image in the second image sample, and the second target model learns the global features of the second text sample; the second target model is determined as the initialization model of the third target model. In other words, this embodiment adopts a multi-stage pre-training method, by allowing the model to better learn some local features of the text in the first stage of pre-training, and allowing the model to pay more attention to the global features of the text in the second stage, thereby improving the pre-training index, thereby solving the technical problem of low training effect of the initialization model, and achieving the technical effect of providing improved training effect of the initialization model.

[0048] The above method of this embodiment is further introduced below.

[0049] As an optional implementation, the method further includes: obtaining entity words of the first text sample; step S104, training the first image sample and the first text sample to obtain a first target model includes: training the first image sample and the entity words to obtain a first target model.

[0050] In this embodiment, in the first stage, after obtaining the first text sample, entity words can be obtained from the first text sample, and entity words can be extracted from the first text sample. Optionally, this embodiment can use a natural language processing model (Nature language processing, referred to as NLP) to extract entity words from the first text sample, and then train the first image sample and the entity words to obtain a first target model. For example, comparative learning training is performed on the entity words of the first image sample and the first text sample to obtain the first target model. In this way, the first target model can focus more on the training of entity words, so that some local features of the first text sample can be learned, and the modal interaction effect is improved, avoiding the problem of no deep modal interaction in the related technology.

[0051] As an optional implementation, training the first image sample and entity words to obtain the first target model includes: obtaining the first image feature of the first image sample; obtaining the first text feature of the entity word; training the first image feature and the first text feature to obtain the first target model.

[0052] In this embodiment, when the first image sample and the entity word are trained to obtain the first target model, the first image feature of the first image sample may be first obtained, and the first image sample may be input into an image encoder (Image Encoder), and the first image feature may be extracted from the first image sample by the image encoder. For example, the first image feature may be I1, I2, ..., I N This embodiment can also obtain the first text feature of the entity word of the first text sample, which can be inputting the entity word of the first text sample into a text encoder (Text Encoder), and extracting the first text feature from the entity word of the first text sample through the text encoder. The first text feature can be T1, T2...T N After acquiring the first image feature and the first text feature, the first image feature and the first text feature may be trained, for example, comparative learning training may be performed on the first image feature and the first text feature, so as to obtain a first target model.

[0053] Optionally, the image encoder of this embodiment may use a data efficient image (Deit) model to extract the first image feature, that is, Deit applies the deformation model (Transformer) from NLP to computer vision (Comouter Vision, referred to as CV).

[0054] Optionally, the text encoder of this embodiment can use the RoBERTa model to extract the first text feature, wherein the RoBERTa model is an upgrade based on the language representation model (BERT), including improving the optimization function at the specific details level of the model; at the training strategy level, a dynamic mask method is used to train the model, which proves the shortcomings of the next sentence prediction model (Next Sentence Prediction, referred to as NSP) training strategy and adopts a larger batch size (batch size); in addition, at the data level, on the one hand, a larger data set is used, and on the other hand, Byte-Pair Encoding (BPE) is used to process text data.

[0055] As an optional implementation, training the first image feature and the first text feature to obtain the first target model includes: matching multiple first image features and multiple first text features to obtain multiple first matching results and multiple first unmatched results, wherein the first matching results include the first image features and the first text features that successfully match each other, and the first unmatched results include the first image features and the first text features that fail to match each other; determining first model parameters based on the multiple first matching results and the multiple first unmatched results; and determining the first target model based on the first model parameters.

[0056] In this embodiment, when the first image feature and the first text feature are trained to obtain the first target model, a plurality of first image features and a plurality of first text features may be matched respectively. For example, the first image features, the first text features, and the first text features may be matched respectively. N And T1, T2...T N Matching is performed to obtain a plurality of first matching results and a plurality of first unmatched results, wherein the first matching result may include a first image feature and a first text feature that successfully match each other, such as I1·T1, I2·T2, etc. N ·T N , the first unmatched result may include the first image feature and the first text feature that fail to match each other, such as I1·T2, I1·T3...I1·T N , I2·T1, I2·T3...I2·T N wait.

[0057] After determining the above-mentioned multiple first matching results and multiple first unmatched results, the first model parameter can be determined based on the multiple first matching results and the multiple first unmatched results. Optionally, this embodiment uses a loss function (InfoNCE loss) to implement using the multiple first matching results and the multiple first unmatched results, for example, by the following formula:

[0058]

[0059] Among them, x i It is used to indicate the probability that the network output belongs to the i-th category, x j It is used to indicate the probability that the network output result belongs to the jth category. Optionally, in this embodiment, the above exp(x i ) can be used to represent the matching results of multiple image features and multiple text features, while Σ j exp(x j ) can be used to represent mismatch results between multiple image features and multiple text features.

[0060] After the first model parameters are determined, a first target model may be generated using the first model parameters.

[0061] The second stage of the model determination method of this embodiment is further introduced below.

[0062] As an optional implementation, step S106, training the first target model based on the second image sample and the second text sample to obtain the second target model includes: obtaining the second image feature of the second image sample; obtaining the second text feature of the second text sample; training the first target model based on the second image feature and the second text feature to obtain the second target model.

[0063] In this embodiment, when the first target model is trained based on the second image sample and the second text sample to obtain the second target model, the second image feature of the second image sample can be first obtained, and the second image sample can be input into the image encoder, and the second image feature can be extracted from the second image sample by the image encoder. For example, the second image feature can be I1′, I2′…I N '. This embodiment can also obtain the second text feature of the second text sample by inputting the entire second text sample into a text encoder, and extracting the second text feature from the entire second text sample through the text encoder. The second text feature can be T1', T2'...T N ', so that the model pays more attention to the global features of the second text sample. After obtaining the second image features and the second text features, the first target model can be trained based on the second image features and the second text features. For example, the first target model can be trained by comparative learning based on the second image features and the second text features to obtain the second target model.

[0064] As an optional implementation, training the first target model based on the second image feature and the second text feature to obtain the second target model includes: matching multiple second image features and multiple second text features to obtain multiple second matching results and multiple second unmatched results, wherein the second matching results include second image features and second text features that successfully match each other, and the second unmatched results include second image features and second text features that fail to match each other; training the first target model based on the multiple second matching results and the multiple second unmatched results to obtain second model parameters; and determining the second target model based on the second model parameters.

[0065] In this embodiment, when the first target model is trained based on the second image feature and the second text feature to obtain the second target model, the plurality of second image features and the plurality of second text features may be matched respectively. For example, the plurality of second image features and the plurality of second text features may be matched respectively. N ′ and T1′, T2′…T N ′ to obtain a plurality of second matching results and a plurality of second unmatched results, wherein the second matching results may include a second image feature and a second text feature that successfully match each other, such as I1′·T1′, I2′·T2′…I N ′·T N ′, the second unmatched result may include a second image feature and a second text feature that fail to match each other, such as I1′·T2′, I1′·T3′…I1′·T N ′, I2′·T1′, I2′·T3′…I2′·T N 'wait.

[0066] After determining the plurality of second matching results and the plurality of second unmatched results, the first target model can be trained based on the plurality of second matching results and the plurality of second unmatched results to obtain second model parameters. Optionally, this embodiment uses InfoNCE loss to implement using the plurality of second matching results and the plurality of second unmatched results.

[0067] After the second model parameters are acquired, a second target model may be generated by the second model parameters, and then determined as an initialization model of the third target model, and the initialization model is output.

[0068] As an optional implementation, the first image sample includes first image noise data and / or the first text sample includes first text noise data.

[0069] In the related art, pre-training requires a large amount of data and has poor learning ability for noise data, such as the twin towers picture and text pre-training method in the related art. However, in the first stage of the model determination method of this embodiment, a certain amount of noise data is allowed, the first image sample can include the first image noise data, and the first text sample can include the first text noise data, that is, this embodiment does not need to perform special processing on the first image noise data and the first text noise data to save labor costs.

[0070] As an optional implementation, the second image sample includes second image noise data and / or the second text sample includes second text noise data.

[0071] In the second stage of the model determination method of this embodiment, a certain amount of noise data is also allowed. The second image sample may include second image noise data, and the second text sample may include second text noise data. That is, this embodiment does not need to perform special processing on the second image noise data and the second text noise data to save labor costs.

[0072] Figure 1B is another model determination method according to an embodiment of the present disclosure. Figure 1B As shown, the method may include the following steps:

[0073] Step S1002: Send a model training request to the server, wherein the model training request includes a first image sample and a first text sample, wherein the text in the first text sample is used to provide a textual description of a target image in the first image sample.

[0074] In the technical solution provided in the above step S1002 of the present disclosure, in order to train an initialization model with high accuracy, a large number of image samples and text samples are required for training, and the amount of data and calculation of the entire training process is large. In order to reduce the resource consumption of user devices (such as smart phones, tablet computers, laptops, PDAs, personal computers, etc.), the model can be trained by the service, and only the trained model can be deployed in the user device to facilitate user use.

[0075] In this embodiment, the above-mentioned model training request can be generated according to the user's model usage requirements. The model training request includes image samples and text samples that need to be processed, and can also include expected processing results, etc.

[0076] Optionally, in this embodiment, a graphical user interface may be provided on the user device, and the user may input a model training request in the input area of ​​the graphical user interface, so that the user device may send the model training request to the server through the network. In order to be more targeted, the server may provide different model training schemes to the user according to the type of user, and the user may select in the input area, so that the user device may generate a model training request according to the rotation result of the user, and send it to the server through the network.

[0077] Step S1004, receiving the initialization model sent by the server in response to the model training request, wherein the initialization model is obtained by the server training the first target model based on the second image sample and the second text sample, and the first target model is obtained by the server training the first image sample and the first text sample.

[0078] In the technical solution provided in the above step S1004 of the present disclosure, the first target model learns the local features of the first text sample, the text in the second text sample is used to describe the target image in the second image sample, and the initialization model learns the global features of the second text sample.

[0079] In this embodiment, the server responds to the model training request and can use the first image sample and the first text sample as training samples for the first stage, and the first text sample corresponds to the first image sample. Optionally, the first image sample and the first text sample of this embodiment do not need to be manually labeled and cleaned to save labor costs. The server can train the first image sample and the first text sample, and can perform machine learning training on the first image sample and the first text sample to obtain a first target model, and the first target model can learn some local features of the first text sample.

[0080] In the second stage, the server can obtain the second image sample and the second text sample as training samples of the second stage, and the second text sample corresponds to the second image sample. The server can continue to train the first target model based on the second image sample and the second text sample to obtain an initialization model, and can perform machine learning training on the first target model based on the second image sample and the second text sample to obtain an initialization model. The initialization model can learn the global features of the second text sample, which is also the overall feature of the second text sample. The initialization model can be used to train an image detection model, an image segmentation model, an image classification model, etc.

[0081] Furthermore, in order to significantly reduce the computing burden of user devices, the trained initialization model can be directly deployed in the server. The user device connects to the server through a specific interface and sends a model acquisition request to the server through the network. The user device obtains the initialization model sent by the server in response to the model acquisition request through the network and uses it as the initialization model of the second target model, thereby achieving the purpose of model pre-training.

[0082] The processing method of this embodiment can be a multi-stage image and text pre-training method, which can use an NLP model to assist in extracting entity words, and can allow the model to better learn some local features of the text in the first stage of pre-training; in the second stage, the model will pay more attention to the overall features of the text, and through the combination of local features and overall features, it can better improve the pre-training effect of the initialization model, thereby solving the technical problem of low pre-training effect of the initialization model.

[0083] Figure 1C FIG. 1 is a flow chart of an image processing method according to an embodiment of the present disclosure. Figure 1CAs shown, the method may include the following steps:

[0084] Step S10002, obtaining the image to be processed.

[0085] In the technical solution provided in the above step S10002 of the present disclosure, the image to be processed may be an image that needs to be processed, for example, an image that needs to be processed by image detection, image segmentation, image classification, image recognition, etc. The processing type can be flexibly determined according to the image application scenario, for example, according to the road scene, education scene, vegetation growth prediction scene, weather prediction scene, etc., and no specific limitation is made here.

[0086] Optionally, this embodiment may acquire the image to be processed by an image acquisition device, for example, by acquiring the image to be processed by a camera deployed in a certain space.

[0087] Step S10004: input the image to be processed into a third target model, wherein the third target model is obtained according to the model determination method of an embodiment of the present disclosure.

[0088] In the technical solution provided in the above step S10004 of the present disclosure, the collected image to be processed can be input into the third target model. Optionally, the third target model of this embodiment is obtained by training an initialization model, and the initialization model can be obtained by training the first target model through the second image sample and the second text sample, and the global features of the second text sample are learned. The text in the second text sample is used to describe the target image in the second image sample. The first target model is obtained by training the first image sample and the first text sample by the server, and the local features of the first text sample are learned. For example, the initialization model can be a recurrent neural network model, and no specific limitation is made here.

[0089] Optionally, in this embodiment, when training the initialization model to obtain the third target model, a large amount of sample data may be collected in advance, and the sample data may include a large number of image samples, which may be annotated to obtain multiple labels, and the labels may be labels related to image processing such as image detection, image segmentation, image classification, and image recognition. Then, the initialization model is trained using the sample data and the corresponding labels to obtain the third target model.

[0090] Optionally, in the sample data, this embodiment may extract features from each sample data through a convolutional neural network to obtain a feature vector including multiple features. For example, the feature vector includes features related to the above-mentioned labels. The initialization model is trained by the feature vector and the corresponding label to obtain the target parameter. The target parameter may be an optimization parameter of the model. The third target model may be determined by the target parameter and the initialization model.

[0091] Optionally, the embodiment can pre-process the sample data according to the distribution consistency algorithm, denoising algorithm, etc., and then perform feature extraction, feature transformation, feature normalization, feature combination, etc. from the pre-processed data to obtain features for training the initialization model. Optionally, the embodiment can further process the features through optimization algorithms, hypothesis functions, loss functions, decision boundaries, convergence speeds, iteration strategies, etc., and train the initialization model through the processed features to obtain a third target model.

[0092] Optionally, after the third target model, this embodiment may also perform cross-validation, target evaluation, overfitting, underfitting and other evaluations on the third target model to determine the final third target model, so as to realize image detection, image segmentation, image classification, image recognition and other processing of the input image through the third target model.

[0093] Step S10006, obtaining the processing result of the third target model.

[0094] In the technical solution provided in the above step S10006 of the present disclosure, the third target model can process the image to be processed, for example, perform image detection, image segmentation, image classification, image recognition and other processing on the third target model to obtain a processing result, which may include image detection results, image segmentation results, image classification results, image recognition results and the like, and output them, for example, display the image detection results, image segmentation results, image classification results, image recognition results and the like through a graphical user interface for further analysis.

[0095] The above technical solution of the embodiment of the present disclosure is further illustrated below in combination with preferred implementation modes.

[0096] In the related art, the dual-tower pre-training scheme can perform real-time offline retrieval. However, since the dual-tower pre-training scheme does not have deep modal interaction, the retrieval effect is not as good as that of a single-tower retrieval.

[0097] In addition, in the related art, the training schemes used in the pre-training of Shuangta images and texts are all one-step training methods, where image and text pairs are directly input into the pre-training network for pre-training tasks. For example, the text samples are sorted and input into the text encoder to extract a global text feature, and then the loss is calculated for the text feature and the image feature. It is impossible to conduct deep inter-modal interaction, which leads to low pre-training indicators of the initialization model. In addition, Shuangta image and text pre-training requires a large amount of data, and its learning ability for noisy data is poor.

[0098] In view of the above problems, this embodiment provides a multi-stage image-text pre-training method. In the first stage, an NLP model is used to assist in extracting entity words so that the model can better learn some local features of the text. In the second stage, the model pays more attention to the overall features of the text, and the combination of local features and overall features is used to improve the effect of initializing the model. It is further introduced below.

[0099] Figure 2 Schematic diagram of a multi-stage image and text pre-training system implemented according to the present disclosure. Figure 2 As shown, in the data processing module of the first stage, the image-text pre-training requires a large amount of data, and a large amount of first image samples and corresponding first text samples can be crawled through the Internet crawler. The first image sample and the corresponding first text sample can tolerate certain noise data to save labor costs, so that the input data of the first stage of this embodiment can be the first image sample and the first text sample (Noisy Product Image-Text Data) containing certain noise data. Optionally, this embodiment can use a large number of unlabeled first text samples and first image samples as training samples for the first stage. Optionally, in the data processing module of the first stage, the NLP model can be used to extract the entity words of the first text sample, and only the entity words are input into the text encoder for calculation to obtain the first text feature, and the first image sample is input into the image encoder for calculation to obtain the first image feature, and then the first text feature and the first image feature are compared and studied to obtain the first target model.

[0100] The processing method in the data processing module of the second stage of this embodiment may be similar to the processing method in the data processing module of the first stage, but the second stage targets the entire text sample, and the entire text sample is input into the text encoder for calculation to obtain the first text feature, and the image sample is input into the image encoder for calculation to obtain the second image feature, and then the second text feature and the second image feature are used to compare and learn the first target model to obtain the second target model, which is also the initialization model of the third target model. The third target model may be an image detection model, an image segmentation model, an image classification model, etc., and no specific restrictions are made here.

[0101] In this embodiment, the text encoder module uses the RoBERTa model to extract text features. The RoBERTa model is an upgrade based on the BERT model. The image encoder uses the Deit model to extract image features. Figure 3 As shown, Figure 3 This is a schematic diagram of a Deit model structure according to an embodiment of the present disclosure. By inputting data class tokens, patch tokens, and distinguishing tokens, and processing them through a self-attention mechanism and a fully connected network (FFN), the output result obtained can be used to obtain image features. The Deit of this embodiment applies transformers from NLP to computer vision.

[0102] In this embodiment, the contrastive learning model module mainly uses InfoNCE loss, and the specific calculation formula is as follows:

[0103]

[0104] Among them, x i It is used to indicate the probability that the network output belongs to the i-th category, x j It is used to indicate the probability that the network output result belongs to the jth category. Optionally, in this embodiment, the above exp(x i ) can be used to represent the matching results of image features and text features, and Σ j exp(x j ) can be used to indicate the matching result of image features and text features that failed to match. Figure 4 As shown. Among them, Figure 4 FIG. 1 is a schematic diagram of matching image features and text features according to an embodiment of the present disclosure. Figure 4 As shown, the image encoder extracts image features I1, I2, ..., I from the input image sample. N, the text encoder extracts text features T1, T2, ... T from the input text sample N , for image features I1, I2, ... N and T1, T2, ... T N Matching is performed respectively to obtain matching results, wherein the matching results on the diagonal are the results of successful matching of text features and image features, and the matching results outside the diagonal are the matching results of failed matching of text features and image features.

[0105] The above method of this embodiment can be applied to multi-stage multi-tower pre-training, and the NLP model can be used to assist in extracting entity words, so that the model can better learn some local features of the text sample in the first stage of pre-training. The second stage model will pay more attention to the overall features of the text sample (noise data does not require special processing), and the pre-training effect of the initialization model can be better improved by combining local features with overall features.

[0106] The present disclosure also provides a method for executing Figure 1A The model determination device of the model determination method of the illustrated embodiment.

[0107] Figure 5A A schematic diagram of a model determination device according to an embodiment of the present disclosure. Figure 5A As shown, the model determination device 50 may include: a first acquisition unit 51, a training unit 52, a processing unit 53 and a determination unit 54.

[0108] The first acquisition unit 51 is used to acquire a first image sample and a first text sample, wherein the text in the first text sample is used to describe the target image in the first image sample.

[0109] The training unit 52 is used to train the first image sample and the first text sample to obtain a first target model, wherein the first target model learns the local features of the first text sample.

[0110] The processing unit 53 is used to obtain a second image sample and a second text sample, and train the first target model based on the second image sample and the second text sample to obtain a second target model, wherein the text in the second text sample is used to provide a textual description of the target image in the second image sample, and the second target model learns the global features of the second text sample.

[0111] The determining unit 54 is configured to determine the second target model as an initialization model of the third target model.

[0112] Optionally, the device further includes: a second acquisition unit, used to acquire entity words of the first text sample; the training unit 52 includes: a training module, used to train the first image sample and the entity words to obtain a first target model.

[0113] Optionally, the training module includes: a first acquisition submodule, used to acquire a first image feature of a first image sample; a second acquisition submodule, used to acquire a first text feature of an entity word; and a first training submodule, used to train the first image feature and the first text feature to obtain a first target model.

[0114] Optionally, the first training submodule is used to train the first image feature and the first text feature to obtain a first target model through the following steps: matching multiple first image features and multiple first text features to obtain multiple first matching results and multiple first unmatched results, wherein the first matching results include the first image features and the first text features that successfully match each other, and the first unmatched results include the first image features and the first text features that fail to match each other; determining first model parameters based on the multiple first matching results and the multiple first unmatched results; determining the first target model based on the first model parameters.

[0115] Optionally, the processing unit 53 includes: a first acquisition module, used to acquire the second image features of the second image sample; a second acquisition module, used to acquire the second text features of the second text sample; and a first training module, used to train the first target model based on the second image features and the second text features to obtain the second target model.

[0116] Optionally, the first training module includes: a matching submodule, used to match multiple second image features and multiple second text features to obtain multiple second matching results and multiple second unmatched results, wherein the second matching results include second image features and second text features that successfully match each other, and the second unmatched results include second image features and second text features that fail to match each other; a second training submodule, used to train the first target model based on the multiple second matching results and the multiple second unmatched results to obtain second model parameters; and a determination submodule, used to determine the second target model based on the second model parameters.

[0117] Optionally, the first image sample includes first image noise data and / or the first text sample includes first text noise data.

[0118] Optionally, the second image sample includes second image noise data and / or the second text sample includes second text noise data.

[0119] The present disclosure also provides a method for executing Figure 1B The model determination device of the model determination method of the illustrated embodiment.

[0120] Figure 5B is a schematic diagram of another model determination device according to an embodiment of the present disclosure. Figure 5B As shown, the model determination device 500 may include: a sending unit 501 and a receiving unit 502.

[0121] The sending unit 501 is used to send a model training request to the server, wherein the model training request includes a first image sample and a first text sample, wherein the text in the first text sample is used to describe the target image in the first image sample.

[0122] The receiving unit 502 is used to receive an initialization model sent by the server in response to a model training request, wherein the initialization model is obtained by the server training the first target model based on the second image sample and the second text sample, the first target model is obtained by the server training the first image sample and the first text sample, the first target model learns the local features of the first text sample, the text in the second text sample is used to provide a textual description of the target image in the second image sample, and the initialization model learns the global features of the second text sample.

[0123] The present disclosure also provides a method for executing Figure 1C The image processing device of the image processing method of the embodiment is shown.

[0124] Figure 5C is a schematic diagram of an image processing device according to an embodiment of the present disclosure. Figure 5C As shown, the image processing device 5000 may include: a third acquisition unit 5001 , an input unit 5002 , and a fourth acquisition unit 5003 .

[0125] The third acquisition unit 5001 is used to acquire the image to be processed.

[0126] The input unit 5002 is used to input the image to be processed into the third target model, wherein the third target model is obtained by the model determination method of the embodiment of the present disclosure.

[0127] The fourth acquisition unit 5003 is used to acquire the processing result of the third target model.

[0128] In this embodiment, a multi-stage pre-training method is adopted. In the first stage of pre-training, the model is allowed to better learn some local features of the text, and in the second stage, the model is allowed to pay more attention to the global features of the text, thereby improving the pre-training indicators, thereby solving the technical problem of low training effect of the initialization model and achieving the technical effect of providing improved training effect of the initialization model.

[0129] It should be noted that the above-mentioned units and modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to this: the above-mentioned units and modules are all located in the same processor; or, the above-mentioned units and modules are located in different processors in any combination.

[0130] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0131] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device. The electronic device may include: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the model determination method of the embodiment of the present disclosure.

[0132] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0133] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the model determination method of the embodiment of the present disclosure.

[0134] Optionally, in this embodiment, the non-volatile storage medium may be configured to store a computer program for performing the following steps:

[0135] S1, obtaining a first image sample and a first text sample, wherein the text in the first text sample is used to describe a target image in the first image sample;

[0136] S2, training the first image sample and the first text sample to obtain a first target model, wherein the first target model learns local features of the first text sample;

[0137] S3, obtaining a second image sample and a second text sample, and training the first target model based on the second image sample and the second text sample to obtain a second target model, wherein the text in the second text sample is used to describe the target image in the second image sample, and the second target model learns the global features of the second text sample;

[0138] S4, determining the second target model as an initialization model of the third target model.

[0139] Optionally, in this embodiment, the non-volatile storage medium may be configured to store a computer program for performing the following steps:

[0140] S1, sending a model training request to a server, wherein the model training request includes a first image sample and a first text sample, wherein the text in the first text sample is used to describe a target image in the first image sample;

[0141] S2, receiving the initialization model sent by the server in response to the model training request, wherein the initialization model is obtained by the server training the first target model based on the second image sample and the second text sample, the first target model is obtained by the server training the first image sample and the first text sample, the first target model learns the local features of the first text sample, the text in the second text sample is used to describe the target image in the second image sample, and the initialization model learns the global features of the second text sample.

[0142] Optionally, in this embodiment, the non-volatile storage medium may be configured to store a computer program for performing the following steps:

[0143] S1, obtaining the image to be processed;

[0144] S2, inputting the image to be processed into a third target model, wherein the third target model is obtained according to the model determination method of the embodiment of the present disclosure;

[0145] S3, obtaining the processing result of the third target model.

[0146] Optionally, in this embodiment, the non-transitory computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0147] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0148] S1, obtaining a first image sample and a first text sample, wherein the text in the first text sample is used to describe a target image in the first image sample;

[0149] S2, training the first image sample and the first text sample to obtain a first target model, wherein the first target model learns local features of the first text sample;

[0150] S3, obtaining a second image sample and a second text sample, and training the first target model based on the second image sample and the second text sample to obtain a second target model, wherein the text in the second text sample is used to describe the target image in the second image sample, and the second target model learns the global features of the second text sample;

[0151] S4, determining the second target model as an initialization model of the third target model.

[0152] Optionally, when the above computer program is executed by a processor, it further implements the following steps:

[0153] S1, sending a model training request to a server, wherein the model training request includes a first image sample and a first text sample, wherein the text in the first text sample is used to describe a target image in the first image sample;

[0154] S2, receiving the initialization model sent by the server in response to the model training request, wherein the initialization model is obtained by the server training the first target model based on the second image sample and the second text sample, the first target model is obtained by the server training the first image sample and the first text sample, the first target model learns the local features of the first text sample, the text in the second text sample is used to describe the target image in the second image sample, and the initialization model learns the global features of the second text sample.

[0155] Optionally, when the above computer program is executed by a processor, it further implements the following steps:

[0156] S1, obtaining the image to be processed;

[0157] S2, inputting the image to be processed into a third target model, wherein the third target model is obtained according to the model determination method of the embodiment of the present disclosure;

[0158] S3, obtaining the processing result of the third target model.

[0159] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0160] The program code for implementing the model determination method of the present disclosure in this embodiment can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable model determination device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0161] Figure 6 is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0162] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0163] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0164] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the model determination method. For example, in some embodiments, the model determination method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the model determination method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the model determination method in any other appropriate manner (e.g., by means of firmware).

[0165] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0166] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable model determination device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0167] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0168] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0169] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0170] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0171] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0172] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model determination method, comprising: Acquire a first image sample and a first text sample, wherein the text in the first text sample is used to describe the target image in the first image sample; Training the first image sample and the first text sample to obtain a first target model, wherein the first target model learns local features of the first text sample; Acquire a second image sample and a second text sample, and train the first target model based on the second image sample and the second text sample to obtain a second target model, wherein the text in the second text sample is used to describe the target image in the second image sample, and the second target model learns the global features of the second text sample; The second target model is determined as an initialization model of the third target model.

2. The method according to claim 1, wherein: The method further comprises: obtaining entity words of the first text sample; Training the first image sample and the first text sample to obtain a first target model includes: training the first image sample and the entity word to obtain a first target model.

3. The method according to claim 2, wherein: Training the first image sample and the entity word to obtain a first target model includes: Acquire a first image feature of the first image sample; Acquire a first text feature of the entity word; The first image feature and the first text feature are trained to obtain the first target model.

4. The method according to claim 3, wherein: Training the first image feature and the first text feature to obtain the first target model includes: Matching the plurality of first image features and the plurality of first text features to obtain a plurality of first matching results and a plurality of first unmatched results, wherein the first matching results include the first image features and the first text features that successfully match each other, and the first unmatched results include the first image features and the first text features that fail to match each other; determining a first model parameter based on the plurality of first matching results and the plurality of first non-matching results; The first target model is determined based on the first model parameters.

5. The method according to claim 1, wherein: Training the first target model based on the second image sample and the second text sample to obtain the second target model includes: Acquire a second image feature of the second image sample; Acquire a second text feature of the second text sample; The first target model is trained based on the second image feature and the second text feature to obtain the second target model.

6. The method according to claim 5, wherein: The first target model is trained based on the second image feature and the second text feature to obtain the second target model, comprising: Matching the plurality of second image features and the plurality of second text features to obtain a plurality of second matching results and a plurality of second unmatched results, wherein the second matching results include the second image features and the second text features that successfully match each other, and the second unmatched results include the second image features and the second text features that fail to match each other; Training the first target model based on the plurality of second matching results and the plurality of second unmatched results to obtain second model parameters; The second target model is determined based on the second model parameters.

7. The method according to any one of claims 1 to 6, wherein: The first image sample comprises first image noise data and / or the first text sample comprises first text noise data.

8. The method according to any one of claims 1 to 6, wherein: The second image sample comprises second image noise data and / or the second text sample comprises second text noise data.

9. An image processing method, comprising: Get the image to be processed; Inputting the image to be processed into a third target model, wherein the third target model is obtained by the model determination method according to any one of claims 1 to 8; The processing result of the third target model is obtained.

10. A model determination device, comprising: A first acquisition unit, configured to acquire a first image sample and a first text sample, wherein the text in the first text sample is used to describe the target image in the first image sample; A training unit, configured to train the first image sample and the first text sample to obtain a first target model, wherein the first target model learns local features of the first text sample; a processing unit, configured to obtain a second image sample and a second text sample, and train the first target model based on the second image sample and the second text sample to obtain a second target model, wherein the text in the second text sample is used to describe the target image in the second image sample, and the second target model learns the global features of the second text sample; A determining unit is used to determine the second target model as an initialization model of the third target model.

11. The device according to claim 10, wherein: The device further includes: a second acquisition unit, configured to acquire entity words of the first text sample; The training unit includes: a training module, which is used to train the first image sample and the entity word to obtain a first target model.

12. The device according to claim 11, wherein The training modules include: A first acquisition submodule, used to acquire a first image feature of the first image sample; A second sub-acquisition module is used to acquire a first text feature of the entity word; The first training submodule is used to train the first image feature and the first text feature to obtain the first target model.

13. The device according to claim 12, wherein: The first training submodule is used to train the first image feature and the first text feature through the following steps to obtain the first target model: Matching the plurality of first image features and the plurality of first text features to obtain a plurality of first matching results and a plurality of first unmatched results, wherein the first matching results include the first image features and the first text features that successfully match each other, and the first unmatched results include the first image features and the first text features that fail to match each other; determining a first model parameter based on the plurality of first matching results and the plurality of first non-matching results; The first target model is determined based on the first model parameters.

14. The device according to claim 10, wherein: The processing unit comprises: A first acquisition module, used for acquiring a second image feature of the second image sample; A second acquisition module, used to acquire a second text feature of the second text sample; The first training module is used to train the first target model based on the second image feature and the second text feature to obtain the second target model.

15. The device according to claim 14, wherein: The first training module includes: a matching submodule, configured to match the plurality of second image features and the plurality of second text features to obtain a plurality of second matching results and a plurality of second unmatched results, wherein the second matching results include the second image features and the second text features that successfully match each other, and the second unmatched results include the second image features and the second text features that fail to match each other; A second training submodule, configured to train the first target model based on the plurality of second matching results and the plurality of second non-matching results to obtain second model parameters; A determination submodule is used to determine the second target model based on the second model parameters.

16. An image processing device, comprising: A third acquisition unit, used to acquire an image to be processed; An input unit, used for inputting the image to be processed into a third target model, wherein the third target model is obtained by the model determination method according to any one of claims 1 to 8; The fourth acquisition unit is used to acquire the processing result of the third target model.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images