Image-text retrieval model training method and device, electronic equipment and storage medium

Abstract

Embodiments of the invention provide an image-text retrieval model training method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining a sample data set; based on the sample data set, a combined loss function of the to-be-trained model is obtained, the combined loss function is composed of a first loss function and a second loss function, the first loss function is used for guiding the to-be-trained model to learn first detection knowledge corresponding to the sample data set, and the second loss function is used for guiding the to-be-trained model to learn second detection knowledge; the second loss function is used for guiding the to-be-trained model to fix existing second detection knowledge of the to-be-trained model; and training the to-be-trained model based on the combined loss function and the sample data set until convergence to obtain an image-text retrieval model. On the basis of the first loss function used for guiding the model to learn new knowledge, the second loss function used for guiding the model to fix existing old knowledge is added to obtain the combined loss function, and the model is trained based on the combined loss function, so that the knowledge forgetting problem occurring along with the continuation of the model training process is reduced.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technologies, and in particular, to a method, an apparatus, an electronic device, and a storage medium for training an image-text retrieval model. Background Art

[0002] Visual-text retrieval, also known as image-text retrieval, is one of the important technologies in current retrieval systems, and has important practical application values in application scenarios such as cross-modal search engines, image-text post retrieval, and geographical location retrieval.

[0003] Currently, the image-text retrieval function in a retrieval system is usually implemented based on a pre-trained image-text retrieval model. Therefore, the training effect of the image-text retrieval model will directly affect the performance of the image-text retrieval function.

[0004] However, in the prior art, during the continuous training process of an image-text retrieval model, as the training samples are continuously updated, the problem of knowledge forgetting will occur in the model, thereby affecting the training efficiency of the model and the performance of the model after training. Summary of the Invention

[0005] Embodiments of the present disclosure provide a method, an apparatus, an electronic device, and a storage medium for training an image-text retrieval model to overcome the problem of knowledge forgetting occurring during the model training process.

[0006] In a first aspect, embodiments of the present disclosure provide a method for training an image-text retrieval model, including:

[0007] Obtain a sample data set, where the sample data set includes at least two image-text sample data pairs, and each image-text sample data pair includes a sample image and a sample description text corresponding to the sample image, and the sample description text is used to characterize the image content of the sample image; based on the sample data set, obtain a combined loss function of a model to be trained, where the combined loss function is composed of a first loss function and a second loss function, the first loss function is used to guide the model to be trained to learn first detection knowledge corresponding to the sample data set, and the second loss function is used to guide the model to be trained to fix existing second detection knowledge of the model to be trained; train the model to be trained based on the combined loss function and the sample data set until convergence, to obtain an image-text retrieval model, where the image-text retrieval model is used to generate a corresponding target image according to a description text, or to generate a corresponding description text according to a target image.

[0008] In a second aspect, embodiments of the present disclosure provide an apparatus for training an image-text retrieval model, including:

[0009] An acquisition module, configured to acquire a sample data set, where the sample data set includes at least two text-image sample data pairs, each text-image sample data pair includes a sample image and a sample description text corresponding to the sample image, and the sample description text is used to characterize the image content of the sample image;

[0010] A processing module, configured to obtain a combined loss function of a model to be trained based on the sample data set, where the combined loss function is composed of a first loss function and a second loss function, the first loss function is used to guide the model to be trained to learn first detection knowledge corresponding to the sample data set, and the second loss function is used to guide the model to be trained to fix existing second detection knowledge of the model to be trained;

[0011] A training module, configured to train the model to be trained based on the combined loss function and the sample data set until convergence, to obtain a text-image retrieval model, where the text-image retrieval model is used to generate a corresponding target image according to a description text, or to generate a corresponding description text according to a target image.

[0012] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0013] The memory stores computer-executable instructions;

[0014] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the text-image retrieval model training method described in the first aspect and various possible designs of the first aspect above.

[0015] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the processor executes the computer-executable instructions, the text-image retrieval model training method described in the first aspect and various possible designs of the first aspect above is implemented.

[0016] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, and when the computer program is executed by a processor, the text-image retrieval model training method described in the first aspect and various possible designs of the first aspect above is implemented.

[0017] The method, device, electronic device, and storage medium for training a graphic-text retrieval model provided in this embodiment obtain a sample data set, where the sample data set includes at least two graphic-text sample data pairs. The graphic-text sample data pair includes a sample image and a sample description text corresponding to the sample image. The sample description text is used to characterize the image content of the sample image. Based on the sample data set, a combined loss function of the model to be trained is obtained. The combined loss function is composed of a first loss function and a second loss function. The first loss function is used to guide the model to be trained to learn the first detection knowledge corresponding to the sample data set, and the second loss function is used to guide the model to be trained to fix the existing second detection knowledge of the model to be trained. The model to be trained is trained based on the combined loss function and the sample data set until convergence to obtain a graphic-text retrieval model. The graphic-text retrieval model is used to generate a corresponding target image according to a description text, or to generate a corresponding description text according to a target image. By adding a second loss function for guiding the model to fix the existing old knowledge on the basis of the first loss function for guiding the model to learn new knowledge, a combined loss function is obtained, and the model is trained based on the combined loss function, thereby reducing the problem of knowledge forgetting that occurs as the model training process continues and improving the training efficiency of the model and the retrieval performance of the trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following briefly introduces the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 FIG. is an application scenario diagram of the method for training a graphic-text retrieval model provided in an embodiment of the present disclosure;

[0020] Figure 2 FIG. is a flowchart of the method for training a graphic-text retrieval model provided in an embodiment of the present disclosure Figure 1 ;

[0021] Figure 3 is Figure 2 a flowchart of the specific implementation manner of step S102 in the illustrated embodiment;

[0022] Figure 4 is Figure 3 a flowchart of the specific implementation manner of step S1023 in the illustrated embodiment;

[0023] Figure 5 is Figure 4Flowchart of the specific implementation of step S1023C in the illustrated embodiment;

[0024] Figure 6 Schematic flow of the graphic retrieval model training method provided by an embodiment of the present disclosure Figure 2 ;

[0025] Figure 7 is Figure 6 Flowchart of a possible implementation of step S201 in the illustrated embodiment;

[0026] Figure 8 Schematic diagram of the process of generating a preprocessed image provided by an embodiment of the present disclosure;

[0027] Figure 9 is Figure 6 Flowchart of another possible implementation of step S201 in the illustrated embodiment;

[0028] Figure 10 Block diagram of the graphic retrieval model training device provided by an embodiment of the present disclosure;

[0029] Figure 11 Schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure;

[0030] Figure 12 Schematic diagram of the hardware structure of the electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are some but not all of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts shall fall within the scope of protection of the present disclosure.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for the user to select authorization or rejection.

[0033] The application scenarios of the embodiments of the present disclosure are explained below:

[0034] Figure 1This is an application scenario diagram of the method for training a graphic-text retrieval model provided by an embodiment of the present disclosure. The method for training a graphic-text retrieval model provided by an embodiment of the present disclosure can be applied to the pre-training scenario of a graphic-text detection model. The execution subject of this embodiment can be a graphic-text retrieval model training device or an electronic device. Among them, the graphic-text retrieval model training device can be implemented in software and / or hardware. Further, the device can be integrated in an electronic device with certain data processing capabilities. Among them, the electronic device can include, but is not limited to, a mobile terminal with big data processing capabilities, and a fixed terminal with big data processing capabilities such as a desktop computer, a server, and a supercomputer. Refer to Figure 1 As shown in, taking the execution subject of this method embodiment as a server for model training as an example, the model developer user sends / downloads the initialization model and training samples to the server through the terminal device, and the server trains the initialization model based on the training samples until the initialization model converges, and then obtains the trained graphic-text retrieval model. After that, the trained graphic-text retrieval model can be further deployed to a specific retrieval system, and the corresponding graphic-text retrieval function can be provided to the user through the graphic-text retrieval model.

[0035] In the prior art, in the process of continuously training a graphic-text retrieval model, as the training samples are continuously updated, on the one hand, the graphic-text detection model will continuously learn new knowledge from the training samples, especially for some specific vertical fields, such as product search in e-commerce scenarios, emoji retrieval in dialogue scenarios, etc. By increasing the sample data in a specific vertical field to fine-tune the model, the retrieval performance of the graphic-text retrieval model in this vertical field can be improved. On the other hand, as the training samples continue to increase, the problem of knowledge forgetting will occur in the model, which will affect the training efficiency of the model and the performance of the trained model.

[0036] The embodiment of the present disclosure provides a method for training a graphic-text retrieval model to solve the above problems.

[0037] Refer to Figure 2 , Figure 2 This is a flowchart of the method for training a graphic-text retrieval model provided by an embodiment of the present disclosure Figure 1 . The method of this embodiment can be applied to an electronic device with data processing capabilities, such as a server. The method for training a graphic-text retrieval model includes:

[0038] Step S101: Obtain a sample data set. The sample data set includes at least two graphic-text sample data pairs. The graphic-text sample data pair includes a sample image and a sample description text corresponding to the sample image. The sample description text is used to characterize the image content of the sample image.

[0039] Exemplarily, the sample data set is the training data used to train the model to be trained so that the trained model (image-text retrieval model) has the image-text retrieval ability. Among them, the sample data set includes one or more image-text sample data pairs. Each image-text sample data pair contains a sample image and a corresponding sample description text. The sample description text is used to characterize the image content of the sample image. Therefore, each image-text sample data pair contains information representing the mapping relationship between the image and the text. By training the model based on the above image-text sample data pairs, the model can learn the above mapping relationship and thus possess the image-text retrieval ability, that is, retrieve the text describing the image content according to the image; or retrieve the corresponding image according to the text describing the image content.

[0040] Further, during the model training process, before the model converges, the model to be trained needs to be trained in multiple rounds. The sample data sets used in each training round are different. Therefore, for example, if the model is trained for S rounds, correspondingly, the sample data set used in the S-th round of training can be expressed as:

[0041]

[0042] Among them, represents the sample data set used in the S-th round of training, represents the sample image in the i-th image-text sample data pair in the sample data set corresponding to the S-th round of training, represents the sample description text in the i-th image-text sample data pair in the sample data set corresponding to the S-th round of training, N (S) represents the number of image-text sample data pairs in the sample data set used in the S-th round of training.

[0043] Step S102: Based on the sample data set, obtain the combined loss function of the model to be trained. Among them, the combined loss function is composed of a first loss function and a second loss function. The first loss function is used to guide the model to be trained to learn the first detection knowledge corresponding to the sample data set, and the second loss function is used to guide the model to be trained to fix the existing second detection knowledge of the model to be trained.

[0044] Exemplarily, refer to Figure 1The schematic diagram of the application scenario shown. After the server for training the model obtains the above training samples, it first loads a preset initialization model, and then uses this initialization model as the model to be trained for multiple rounds of training. Specifically, this initialization model is implemented based on a pre-trained model for comparing text and images. The pre-trained model for text and images is a pre-trained neural network model for matching images and text, and its specific principle will not be elaborated as it is prior art. Of course, the initialization model can also be implemented based on other models with similar functions or structures, and will not be exemplified one by one here. Further, a loss function is set in the server for continuously improving the model to be trained. In this embodiment, the loss function for continuously improving the model to be trained is a combined loss function. The combined loss function includes a first loss function and a second loss function. The first loss function is used to guide the model to be trained to learn the first detection knowledge corresponding to the sample data set, and the second loss function is used to guide the model to be trained to fix the existing second detection knowledge of the model to be trained. Specifically, the first loss function is used to guide the model to be trained to train on new knowledge in the sample data set. By calculating the difference between the retrieval description text obtained after the model to be trained processes the sample image and the sample description text, a residual value is obtained, and then the model parameters are adjusted based on the residual value to enable the model to be trained to learn the knowledge in the training sample set. In a possible implementation, the implementation method of the first loss function is:

[0045]

[0046] where L new is the first loss function; is the image feature of the sample image in the i-th text-image sample data pair in the sample data set, is the text feature of the sample description text in the i-th text-image sample data pair in the sample data set, is generated by encoding the sample image through the model to be trained, is generated by encoding the sample description text through the model to be trained; <·,·> represents the dot product between two features; N is the number of text-image sample data pairs in the sample data set; τ is a learnable hyperparameter, which is initialized to a positive minimum value, such as 0.01, at the network initialization stage and is automatically adjusted as the network is trained. Through the above first loss function, the distance between paired samples (the sample image and the corresponding sample description text in the same text-image sample data pair) can be shortened, while the distance between unpaired samples (the sample image and the sample description text in different text-image sample data pairs) can be increased, so that the model to be trained can learn the knowledge of mapping sample images and sample description texts.

[0047] On the other hand, the second loss function is used to guide the model to be trained to fix the existing second detection knowledge of the model to be trained, that is, to retain the knowledge learned in the previous rounds of the model to be trained, so as to delay the forgetting of old knowledge caused by learning new knowledge (through the first loss function). Specifically, as Figure 3 shown, the specific implementation manner of step S102 includes:

[0048] Step S1021: Obtain a new knowledge model corresponding to the model to be trained. The new knowledge model is the result of training the model to be trained based on the first loss function and the sample data set;

[0049] Step S1022: Encode the sample data set through the model to be trained and the new knowledge model respectively to obtain the first graphic and text features corresponding to the model to be trained and the second graphic and text features corresponding to the new knowledge model;

[0050] Exemplarily, during the process of training the model to be trained in multiple rounds, for the S-th round of model training, the sample data set used is The sample data set is which contains N graphic and text sample data pairs; then, based on this sample data set and the above first loss function, after training the model to be trained, a model that has learned the new knowledge in the sample data set is obtained, that is, the new knowledge model, which can be expressed as Mode (S) ; on the other hand, for the model to be trained that has not been trained, which is equivalent to the model after the previous round of training and is equivalent to the old knowledge model, it can be expressed as Mode (S-1) , that is, the model to be trained referred to in the steps of this embodiment.

[0051] After that, the new knowledge model Mode (S) and the model to be trained Mode (S-1) (old knowledge model) are used to encode the sample data set to obtain the first graphic and text features corresponding to the model to be trained and the second graphic and text features corresponding to the new knowledge model. Specifically, since the model to be trained and the new knowledge model both originate from the same initialization model, both models contain an image encoder and a text encoder. Correspondingly, the first graphic and text features include the first image feature and the first text feature, and the second graphic and text features include the second image feature and the second text feature. Exemplarily, the implementation steps of step S1022 include:

[0052] Step S1022A: Input the sample images and the corresponding sample description texts in each graphic and text sample data pair into the image encoder and the text encoder of the model to be trained respectively to obtain the first image feature and the first text feature.

[0053] Step S1022B: Input the sample images and corresponding sample description texts in each graphic and text sample data pair into the image encoder and text encoder of the new knowledge model respectively to obtain second image features and second text features.

[0054] Specifically, the generation method for generating the first image features and the first text features using the model to be trained can be represented by Equation (3):

[0055]

[0056] Wherein, represents the first image features corresponding to the i-th graphic and text sample data pair, represents the first text features corresponding to the i-th graphic and text sample data pair. Correspondingly, the first graphic and text features can be expressed as v i represents the sample image in the i-th graphic and text sample data pair, t i represents the sample description text in the i-th graphic and text sample data pair; f ′ v represents the image encoder of the model to be trained, f ′ t represents the image encoder of the model to be trained.

[0057] On the other hand, use the new knowledge model to process the sample images and corresponding sample description texts in the graphic and text sample data pair to obtain the second image features corresponding to the i-th graphic and text sample data pair and the second text features corresponding to the i-th graphic and text sample data pair. Correspondingly, the second graphic and text features can be expressed as The specific process is similar to the production process of the first image features and the first text features and will not be elaborated here.

[0058] Step S1023: Obtain a second loss function according to the similarity between the first graphic and text features and the second graphic and text features.

[0059] After that, obtain the second loss function according to the similarity between the first graphic and text features and the second graphic and text features obtained in the above steps. Specifically, as Figure 4 shown, the specific implementation manner of Step S1023 includes:

[0060] Step S1023A: Obtain a first similarity square matrix according to the first graphic and text features corresponding to each graphic and text sample data pair in the sample dataset. The first similarity square matrix represents the content relevance between each pair of sample images and sample description texts after being encoded by the model to be trained.

[0061] Step S1023B: Obtain a second similarity matrix based on the second graphic and text features corresponding to each graphic and text sample data in the sample dataset; the second similarity matrix represents the content relevance between each pair of sample images and sample description texts after being encoded by the new knowledge model.

[0062] Step S1023C: Perform fusion based on the first similarity matrix and the second similarity matrix to obtain a second loss function.

[0063] Among them, a matrix, also known as a square matrix, refers to a matrix with equal numbers of rows and columns. Each matrix value in the similarity matrix represents the content relevance (affinity) between any sample image and any sample description text. The larger the matrix value, the greater the content relevance. Specifically, still taking the model training process in the S-th round as an example, according to the sample dataset The obtained first graphic and text feature is The second graphic and text feature is On this basis, the implementation method for obtaining the first similarity matrix according to the first graphic and text feature can be represented by Equation (4):

[0064]

[0065] Among them, represents the matrix value in the i-th row and j-th column of the first similarity matrix; N is the number of graphic and text sample data pairs in the sample dataset That is, the matrix dimension of the first similarity matrix and the second similarity matrix; τ is a learnable hyperparameter.

[0066] Correspondingly, the implementation method for obtaining the second similarity matrix according to the second graphic and text feature can be represented by Equation (5):

[0067]

[0068] Among them, represents the matrix value in the i-th row and j-th column of the second similarity matrix; for the description of other parameters, refer to the description of the corresponding parameters in Equation (4).

[0069] After that, perform fusion based on the first similarity matrix and the second similarity matrix obtained in the above steps of Equation (4) and Equation (5) to obtain a second loss function. In a possible implementation method, specifically, for example, the calculation method of the second loss function is shown in Equation (6):

[0070]

[0071] Among them, is the second loss function, is the first similarity matrix obtained in the previous steps; It is the second similarity matrix obtained in the previous steps. For details, please refer to the relevant introductions of the first similarity matrix and the second similarity matrix in the previous steps.

[0072] In the steps of this embodiment, the second loss function generated by using the above method is equivalent to, in the process of knowledge distillation, taking the old knowledge contained in the old knowledge model as a constraint, so that while the new model continuously learns the new data set, it alleviates the forgetting of the old data set, thereby improving the efficiency of model training and the model performance.

[0073] However, further, in the actual application process, when there are errors in the old knowledge contained in the old knowledge model, it will cause the wrong knowledge to be synchronized and solidified while solidifying the old knowledge by using the above method, resulting in the accumulation of wrong knowledge in the trained model (new knowledge model), and further reducing the retrieval performance of the trained image-text retrieval model. To solve the above problems, this embodiment further proposes another implementation scheme for generating the second loss function. Specifically, as Figure 5 shown, the specific implementation manner of step S1023C includes:

[0074] Step S1023C-1: Create an empty matrix of the target dimension, where the target dimension is the matrix dimension of the first similarity matrix or the second similarity matrix.

[0075] Step S1023C-2: If the first matrix value on the diagonal in the i-th row of the first similarity matrix is the maximum value among all the matrix values in the i-th row of the first similarity matrix, then assign the matrix values of the i-th row of the first similarity matrix to the matrix values of the i-th row of the empty matrix correspondingly.

[0076] Step S1023C-3: If the first matrix value on the diagonal in the i-th row of the first similarity matrix is not the maximum value among all the matrix values in the i-th row of the first similarity matrix, and the matrix value on the diagonal in the i-th row of the second similarity matrix is the maximum value among all the matrix values in the i-th row of the second similarity matrix, then regularize the matrix values of the i-th row of the first similarity matrix based on the matrix values of the i-th row of the second similarity matrix to obtain the corresponding regularization sequence, and assign the regularization matrix values in the regularization sequence to the matrix values of the i-th row of the empty matrix correspondingly.

[0077] Step S1023C-4: If, within the i-th row of the first similarity matrix, the first matrix value located on the diagonal is not the maximum value among all the matrix values within the i-th row of the first similarity matrix, and within the i-th row of the second similarity matrix, the second matrix value located on the diagonal is not the maximum value among all the matrix values within the i-th row of the second similarity matrix, then assign the value 1 to the first matrix value within the i-th row of the first similarity matrix. Subsequently, normalize all the matrix values within the i-th row of the first similarity matrix to obtain the regularization sequence corresponding to the i-th row, and assign each regularized matrix value within the regularization sequence to the matrix values of the i-th row of the empty matrix, where i is any positive integer less than or equal to the target dimension.

[0078] Step S1023C-5: Generate an updated first similarity matrix based on the assigned empty matrix.

[0079] Step S1023C-6: Generate a second loss function based on the updated first similarity matrix and the first similarity matrix.

[0080] Specifically, first create an empty matrix with the target dimension, where the target dimension is, for example, the matrix dimension N of the first similarity matrix or the second similarity matrix in the above embodiments. Subsequently, based on the content of the first similarity matrix representing old knowledge, perform case-by-case processing:

[0081] In the first case, when the first matrix value located on the diagonal within the i-th row of the first similarity matrix is the maximum value among all the matrix values within the i-th row of the first similarity matrix, then assign the matrix values of the i-th row of the first similarity matrix to the matrix values of the i-th row of the empty matrix. For example, the matrix values of the i-th row of the first similarity matrix are [v0, v1, v2, v3, v4], where the first matrix value located on the diagonal is v2, and v2 is the maximum value among [v0, v1, v2, v3, v4], then assign [v0, v1, v2, v3, v4] to the i-th row of the empty matrix.

[0082] In the second case, if, within the \(i\)-th row of the first similarity matrix, the first matrix value located on the diagonal is not the maximum value among all the matrix values within the \(i\)-th row of the first similarity matrix, and within the \(i\)-th row of the second similarity matrix, the matrix value located on the diagonal is the maximum value among all the matrix values within the \(i\)-th row of the second similarity matrix. For example, the matrix values of the \(i\)-th row of the first similarity matrix are \([v0, v1, v2, v3, v4]\), and the matrix values of the \(i\)-th row of the second similarity matrix are \([v5, v6, v7, v8, v9]\). Among them, the first matrix value located on the diagonal in the \(i\)-th row of the first similarity matrix is \(v1\), and \(v1\) is not the maximum value among \([v0, v1, v2, v3, v4]\); at the same time, the first matrix value located on the diagonal in the \(i\)-th row of the second similarity matrix is \(v6\), and \(v6\) is the maximum value among \([v5, v6, v7, v8, v9]\). In this case, then regularize the matrix values of the \(i\)-th row of the first similarity matrix based on the matrix values of the \(i\)-th row of the second similarity matrix to obtain the corresponding regularization sequence \([Re(v0), Re(v1), Re(v2), Re(v3), Re(v4)]\) for the \(i\)-th row. After that, assign the regularization sequence \([Re(v0), Re(v1), Re(v2), Re(v3), Re(v4)]\) to the \(i\)-th row of the empty matrix. Here, \(Re()\) represents the regularization process, and the specific calculation method of regularization is prior art and will not be elaborated here.

[0083] In the third case, if the first matrix value located on the diagonal within the i-th row of the first similarity matrix is not the maximum value among all the matrix values within the i-th row of the first similarity matrix, and the second matrix value located on the diagonal within the i-th row of the second similarity matrix is also not the maximum value among all the matrix values within the i-th row of the second similarity matrix. For example, the matrix values of the i-th row of the first similarity matrix are [v0, v1, v2, v3, v4], and the matrix values of the i-th row of the second similarity matrix are [v5, v6, v7, v8, v9]. Among them, the first matrix value located on the diagonal within the i-th row of the first similarity matrix is v1, and v1 is not the maximum value among [v0, v1, v2, v3, v4]; at the same time, the first matrix value located on the diagonal within the i-th row of the second similarity matrix is v6, and v6 is also not the maximum value among [v5, v6, v7, v8, v9]. In this case, the first matrix value within the i-th row of the first similarity matrix is assigned 1, that is, v1 within [v0, v1, v2, v3, v4] is assigned 1, resulting in [v0, 1, v2, v3, v4]; then, all the matrix values within the i-th row of the first similarity matrix are regularized to obtain the regularization sequence corresponding to the i-th row, that is, [Re(v0), Re(1), Re(v2), Re(v3), Re(v4)], and then this regularization sequence [Re(v0), Re(1), Re(v2), Re(v3), Re(v4)] is assigned to the i-th row of the empty matrix.

[0084] Among them, the i-th row is any row of the first similarity matrix. The above strategy can be used to detect each row (the 1st to N-th rows) of the first similarity matrix through parallel processing or sequential traversal processing, and according to the detection results, the corresponding rows of the empty matrix are filled. Thus, after each row of the first similarity matrix is detected and the corresponding assignment processing for the empty matrix is executed, a fully assigned (each row is assigned) matrix is obtained, and this matrix is used as the updated first similarity matrix. Until, using the calculation method shown in Equation (6), the second loss function is obtained using the updated first similarity matrix and the first similarity matrix.

[0085] In the steps of this embodiment, by evaluating the matrix values at the diagonal positions of the first similarity matrix, the first similarity matrix is corrected to reduce the errors in the old knowledge corresponding to the first similarity matrix, thereby reducing the accumulation of incorrect knowledge caused during the process of solidifying the old knowledge, and further improving the retrieval performance of the trained image-text retrieval model.

[0086] Step S1024: Obtain the combined loss function of the model to be trained according to the first loss function and the second loss function.

[0087] Exemplarily, based on the introduction in the previous steps, after obtaining the first loss function and the second loss function After that, the sum of the two is used as the combined loss function, or the weights corresponding to the first loss function and the second loss function are obtained, and the weighted sum of the two is used as the combined loss function. Specifically, the expression of the combined loss function is shown in Equation (7):

[0088]

[0089] where is the combined loss function; w1 is the weight coefficient of the first loss function ; w2 is the weight coefficient of the second loss function .

[0090] Step S103: Train the model to be trained based on the combined loss function and the sample data set until convergence to obtain a graphic-text retrieval model, where the graphic-text retrieval model is used to generate a corresponding first image according to the first description text, or is used to generate a corresponding second description text according to the second image.

[0091] Exemplarily, after obtaining the combined loss function, based on this combined loss function, start the S-th round of model training. Input the sample data set into the model to be trained to obtain the corresponding model prediction result. For example, input the sample image into the model to be trained to obtain the predicted description text predicted by the model to be trained; or input the sample description text into the model to be trained to obtain the predicted image predicted by the model to be trained. After that, use the combined loss function to calculate the residual between the prediction result and the label data (non-input data) of the corresponding graphic-text sample data pair in the sample data set, and fine-tune the model parameters of the model to be trained based on this parameter, so as to complete the training of the model to be trained in this round. After that, repeat the above training process, for example, return to step S101, obtain a new set of sample data sets again, then obtain the combined loss function of the corresponding model to be trained based on the new set of sample data sets, and train the model to be trained again, and so on, until the model reaches the convergence condition to obtain a graphic-text retrieval model. This graphic-text retrieval model is used to generate a corresponding first image according to the first description text (i.e., retrieve an image according to the text), or is used to generate a corresponding second description text according to the second image (i.e., retrieve text according to the image). The specific implementation process of model training will not be elaborated here.

[0092] In this embodiment, by obtaining a sample data set, the sample data set includes at least two text-image sample data pairs. The text-image sample data pair includes a sample image and a sample description text corresponding to the sample image. The sample description text is used to characterize the image content of the sample image. Based on the sample data set, a combined loss function of the model to be trained is obtained. The combined loss function is composed of a first loss function and a second loss function. The first loss function is used to guide the model to be trained to learn the first detection knowledge corresponding to the sample data set, and the second loss function is used to guide the model to be trained to fix the existing second detection knowledge of the model to be trained. The model to be trained is trained based on the combined loss function and the sample data set until convergence, and a text-image retrieval model is obtained. The text-image retrieval model is used to generate a corresponding target image according to the description text, or to generate a corresponding description text according to the target image. By adding a second loss function for guiding the model to fix the existing old knowledge on the basis of the first loss function for guiding the model to learn new knowledge, a combined loss function is obtained, and the model is trained based on the combined loss function, thereby reducing the problem of knowledge forgetting that occurs as the model training process continues, and improving the training efficiency of the model and the retrieval performance of the trained model.

[0093] Reference Figure 6 , Figure 6 is a schematic flowchart of the method for training a text-image retrieval model provided by an embodiment of the present disclosure Figure 2 . This embodiment further adds a step of preprocessing the sample data set on the basis of the embodiment shown in Figure 2 , and refines the specific implementation process of step S102. The method for training a text-image retrieval model includes:

[0094] Step S201: Obtain a sample data set. The sample data set includes at least two text-image sample data pairs. The text-image sample data pair includes a sample image and a sample description text corresponding to the sample image. The sample description text is used to characterize the image content of the sample image.

[0095] Step S202: Preprocess the sample images in the sample data set to obtain preprocessed images. The preprocessed images lack the effective pixel information corresponding to at least one pixel point compared with the sample images.

[0096] Exemplarily, preprocessing the sample images in the sample data set to obtain preprocessed images may include performing processing steps such as resolution adjustment, left-right flipping, and up-down flipping on the sample images in each text-image sample data pair in the sample data set, so that the generated preprocessed images have higher randomness. The specific implementation method can be set or not set according to needs, and is not limited here. On this basis, the effective content of the sample images can also be deleted, so as to improve the generalization ability of the model and the ability to search for low-quality images.

[0097] Exemplarily, in one possible implementation, as Figure 7 shown, the implementation of step S202 includes:

[0098] Step S2021: Expand blank pixels outward from the edge of the sample image to obtain an expanded image, and the image size of the expanded image is larger than the initial image size of the sample image.

[0099] Step S2022: Crop the expanded image based on the initial image size to obtain a preprocessed image.

[0100] Exemplarily, Figure 8 is a schematic diagram of a process for generating a preprocessed image provided by an embodiment of the present disclosure. As Figure 8 shown, first, based on a sample image with a size of, for example, 224 * 224 pixels, expand blank pixels outward from the edge of the sample image. For example, expand M blank pixels in sequence up, down, left, and right; the RGB value of the expanded blank pixels is, for example, (0, 0, 0). Then, randomly crop the expanded image based on the initial image size (224 * 224) to obtain an image with the same size as the initial image size (224 * 224), that is, the preprocessed image.

[0101] In another possible implementation, as Figure 9 shown, the implementation of step S202 includes:

[0102] Step S2023: Obtain an occlusion size within a preset range.

[0103] Step S2024: Randomly determine an occlusion area within the sample image based on the occlusion size, and randomly fill the pixel information of the pixel points within the occlusion area to obtain a preprocessed image.

[0104] Exemplarily, first, based on preset configuration information, obtain an occlusion size within a preset range, which is smaller than the image size of the sample image. Specifically, for example, if the size of the sample image is 224 * 224, then randomly select a proportion of the occlusion area from (1000, 20000), randomly select an aspect ratio of the occlusion area from (0.3, 3.3), and then generate a rectangular area using the above area ratio and aspect ratio. The area size of the rectangular area is the occlusion size. Then, use this rectangular area to randomly occlude any area of the image in the training set and fill it with random RGB values, thereby obtaining an image with some parts changed, that is, the preprocessed image.

[0105] Wherein, Figure 7 and Figure 9The two methods for generating a preprocessed image shown can be executed separately to obtain the preprocessed image, or can be executed successively to obtain the preprocessed image. For example, Figure 7 the corresponding steps can be executed first, and then according to Figure 7 the result obtained from the corresponding steps, Figure 9 the corresponding steps are executed. Specifically, it includes:

[0106] Step S202A: Expand blank pixels outward from the edge of the sample image to obtain an expanded image, and the image size of the expanded image is larger than the initial image size of the sample image.

[0107] Step S202B: Crop the expanded image based on the initial image size to obtain a pre-frontal preprocessed image.

[0108] Step S202C: Obtain the occlusion size within a preset range.

[0109] Step S202D: Randomly determine an occlusion area within the pre-frontal preprocessed image based on the occlusion size, and randomly fill the pixel information of the pixel points within the occlusion area to obtain the preprocessed image.

[0110] Of course, in another implementation manner, Figure 9 the corresponding steps can also be executed first, and then according to Figure 9 the result obtained from the corresponding steps, Figure 7 the corresponding steps are executed, which will not be elaborated here. In this embodiment, by preprocessing the sample image to obtain the preprocessed image, and then performing the subsequent model training process based on the preprocessed image, the generalization ability of the model can be further improved, and the retrieval performance of the image-text retrieval model can be improved.

[0111] Step S203: Generate a preprocessed sample data set based on the preprocessed image and the corresponding sample description text.

[0112] Step S204: Obtain the combined loss function of the model to be trained based on the preprocessed sample data set, where the combined loss function is composed of a first loss function and a second loss function.

[0113] Step S205: Train the model to be trained based on the combined loss function and the preprocessed sample data set until convergence to obtain the image-text retrieval model, where the image-text retrieval model is used to generate a corresponding first image according to the first description text, or is used to generate a corresponding second description text according to the second image.

[0114] In this embodiment, the implementation manners of steps 203-step 205 are the same as those of steps S101-step S103 in the embodiment Figure 2 shown in the present disclosure, and will not be elaborated one by one here.

[0115] An embodiment of the present disclosure also discloses a method for a graphic and text retrieval model, including:

[0116] S1. Obtain a retrieval input, where the retrieval input includes a first image or a second description text;

[0117] S2. Process the retrieval input through the graphic and text retrieval model to obtain a retrieval result. The retrieval result includes a first description text corresponding to the first image, or the retrieval result includes a second image corresponding to the second description text; wherein, the graphic and text retrieval model is obtained by Figures 2 - 9 being trained by the graphic and text retrieval model training method shown in any embodiment.

[0118] Corresponding to the graphic and text retrieval model training method in the above embodiment, Figure 10 it is a structural block diagram of a graphic and text retrieval model training device provided by an embodiment of the present disclosure. For the sake of convenience of description, only the parts related to the embodiment of the present disclosure are shown. Referring to Figure 10 , the graphic and text retrieval model training device 3 includes:

[0119] An acquisition module 31, configured to acquire a sample data set. The sample data set includes at least two graphic and text sample data pairs. The graphic and text sample data pair includes a sample image and a sample description text corresponding to the sample image. The sample description text is used to characterize the image content of the sample image;

[0120] A processing module 32, configured to obtain a combined loss function of the model to be trained based on the sample data set. The combined loss function is composed of a first loss function and a second loss function. The first loss function is used to guide the model to be trained to learn first detection knowledge corresponding to the sample data set, and the second loss function is used to guide the model to be trained to fix the existing second detection knowledge of the model to be trained;

[0121] A training module 33, configured to train the model to be trained based on the combined loss function and the sample data set until convergence, to obtain a graphic and text retrieval model, where the graphic and text retrieval model is used to generate a corresponding first image according to the first description text, or is used to generate a corresponding second description text according to the second image.

[0122] According to one or more embodiments of the present disclosure, the processing module 32 is specifically configured to: obtain a new knowledge model corresponding to the model to be trained. The new knowledge model is the result of training the model to be trained based on the first loss function and the sample data set; respectively encode the sample data set through the model to be trained and the new knowledge model to obtain a first graphic and text feature corresponding to the model to be trained and a second graphic and text feature corresponding to the new knowledge model; obtain a second loss function according to the similarity between the first graphic and text feature and the second graphic and text feature; generate a combined loss function according to the first loss function and the second loss function.

[0123] According to one or more embodiments of the present disclosure, an image encoder and a text encoder are included in the model to be trained. The first image-text feature includes a first image feature and a first text feature, and the second image-text feature includes a second image feature and a second text feature. When the processing module 32 encodes the sample data set through the model to be trained and the new knowledge model respectively to obtain the corresponding first image-text feature of the model to be trained and the corresponding second image-text feature of the new knowledge model, it is specifically configured to: input the sample image and the corresponding sample description text in each image-text sample data pair into the image encoder and the text encoder of the model to be trained respectively to obtain a first image feature and a first text feature; input the sample image and the corresponding sample description text in each image-text sample data pair into the image encoder and the text encoder of the new knowledge model respectively to obtain a second image feature and a second text feature.

[0124] According to one or more embodiments of the present disclosure, when the processing module 32 obtains the second loss function according to the similarity between the first image-text feature and the second image-text feature, it is specifically configured to: obtain a first similarity matrix according to the first image-text features corresponding to the image-text sample data pairs in the sample data set, where the first similarity matrix represents the content relevance between each pair of sample images and sample description texts after being encoded by the model to be trained; obtain a second similarity matrix according to the second image-text features corresponding to the image-text sample data pairs in the sample data set; the second similarity matrix represents the content relevance between each pair of sample images and sample description texts after being encoded by the new knowledge model; fuse the first similarity matrix and the second similarity matrix to obtain the second loss function.

[0125] According to one or more embodiments of the present disclosure, when the processing module 32 fuses according to the first similarity square matrix and the second similarity square matrix to obtain the second loss function, it is specifically configured to: create an empty square matrix with a target dimension, where the target dimension is the square matrix dimension of the first similarity square matrix or the second similarity square matrix; if the first square matrix value located on the diagonal in the i-th row of the first similarity square matrix is the maximum value among all the square matrix values in the i-th row of the first similarity square matrix, then assign the square matrix values of the i-th row of the first similarity square matrix to the square matrix values of the i-th row of the empty square matrix correspondingly; if the first square matrix value located on the diagonal in the i-th row of the first similarity square matrix is not the maximum value among all the square matrix values in the i-th row of the first similarity square matrix, and the square matrix value located on the diagonal in the i-th row of the second similarity square matrix is the maximum value among all the square matrix values in the i-th row of the second similarity square matrix, then regularize the square matrix values of the i-th row of the first similarity square matrix based on the square matrix values of the i-th row of the second similarity square matrix to obtain a regularization sequence corresponding to the i-th row, and assign the regularization square matrix values in the regularization sequence to the square matrix values of the i-th row of the empty square matrix correspondingly; if the first square matrix value located on the diagonal in the i-th row of the first similarity square matrix is not the maximum value among all the square matrix values in the i-th row of the first similarity square matrix, and the second square matrix value located on the diagonal in the i-th row of the second similarity square matrix is also not the maximum value among all the square matrix values in the i-th row of the second similarity square matrix, then assign 1 to the first square matrix value in the i-th row of the first similarity square matrix, and then regularize all the square matrix values in the i-th row of the first similarity square matrix to obtain a regularization sequence corresponding to the i-th row, and assign the regularization square matrix values in the regularization sequence to the square matrix values of the i-th row of the empty square matrix correspondingly, where i is any positive integer less than or equal to the target dimension; generate an updated first similarity square matrix according to the assigned empty square matrix; generate a second loss function based on the updated first similarity square matrix and the first similarity square matrix.

[0126] According to one or more embodiments of the present disclosure, after obtaining the sample data set, the processing module 32 is further configured to: preprocess the sample images in the sample data set to obtain preprocessed images, where the preprocessed images lack the effective pixel information corresponding to at least one pixel point compared with the sample images; generate a preprocessed sample data set based on the preprocessed images and the corresponding sample description texts; when the processing module 32 obtains the combined loss function of the model to be trained based on the sample data set, it is specifically configured to: obtain the combined loss function of the model to be trained based on the preprocessed sample data set.

[0127] According to one or more embodiments of the present disclosure, when the processing module 32 preprocesses the sample image in the sample data set to obtain the preprocessed image, it is specifically used to: expand blank pixels from the edge of the sample image outward to obtain an expanded image, where the image size of the expanded image is larger than the initial image size of the sample image; and crop the expanded image based on the initial image size to obtain the preprocessed image.

[0128] According to one or more embodiments of the present disclosure, when the processing module 32 preprocesses the sample image in the sample data set to obtain the preprocessed image, it is specifically used to: obtain the occlusion size within a preset range; randomly determine an occlusion area in the sample image based on the occlusion size, and randomly fill the pixel information of the pixel points in the occlusion area to obtain the preprocessed image.

[0129] According to one or more embodiments of the present disclosure, the model to be trained is implemented based on a pre-trained model of text-image comparison.

[0130] The acquisition module 31, processing module 32 and training module 33 are connected in sequence. The image-text retrieval model training device 3 provided in this embodiment can implement the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0131] Figure 11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure is shown in FIG. Figure 11 As shown, the electronic device 4 includes:

[0132] A processor 41, and a memory 42 communicatively connected to the processor 41;

[0133] Memory 42 stores computer-executable instructions;

[0134] The processor 41 executes the computer execution instructions stored in the memory 42 to implement the following Figures 2 - 9 The image-text retrieval model training method in the illustrated embodiment.

[0135] Optionally, the processor 41 and the memory 42 are connected via a bus 43 .

[0136] For related instructions, please refer to Figures 2 - 9 The relevant descriptions and effects corresponding to the steps in the corresponding embodiments can be understood, and no further details are given here.

[0137] The present invention provides a computer-readable storage medium that stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the computer-executable instructions are used to implement the present invention. Figures 2 - 9 The image-text retrieval model training method provided in any one of the corresponding embodiments.

[0138] An embodiment of the present disclosure provides a computer program product, including a computer program which, when executed by a processor, implements the graphic and text retrieval model training method provided by any one of the corresponding embodiments of the present disclosure. Figures 2 - 9

[0139] To implement the above embodiments, an embodiment of the present disclosure also provides an electronic device.

[0140] Referring to Figure 12 , which shows a schematic structural diagram of an electronic device 900 suitable for implementing the embodiments of the present disclosure. The electronic device 900 may be a terminal device or a server. Among them, the terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 12 The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0141] As Figure 12 shown, the electronic device 900 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 901, which may perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 902 or the program loaded from the storage device 908 into the random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. The input / output (I / O) interface 905 is also connected to the bus 904.

[0142] Generally, the following devices may be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 12An electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or have all the devices shown. Instead, more or fewer devices may be implemented or had.

[0143] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a computer-readable medium, the computer program including program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, the above-described functions defined in the methods of the embodiments of the present disclosure are performed.

[0144] It should be noted that the above-mentioned computer-readable medium in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. And in the present disclosure, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which the computer-readable program code is carried. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable signal medium can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0145] The above-mentioned computer-readable medium can be included in the above-mentioned electronic device; or it can exist separately and not be assembled into the electronic device.

[0146] The above computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0147] Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0149] The units or modules involved in the embodiments described in the present disclosure may be implemented in software or in hardware. In some cases, the name of the unit or module does not constitute a limitation on the unit itself.

[0150] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chip (SOCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0151] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM or Flash Memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0152] In a first aspect, according to one or more embodiments of the present disclosure, there is provided a method for training a graphic and text retrieval model, including:

[0153] Obtain a sample data set, where the sample data set includes at least two graphic and text sample data pairs, the graphic and text sample data pair includes a sample image, and a sample description text corresponding to the sample image, the sample description text being used to characterize the image content of the sample image; based on the sample data set, obtain a combined loss function of the model to be trained, where the combined loss function is composed of a first loss function and a second loss function, the first loss function being used to guide the model to be trained to learn first detection knowledge corresponding to the sample data set, the second loss function being used to guide the model to be trained to fix existing second detection knowledge of the model to be trained; train the model to be trained based on the combined loss function and the sample data set until convergence to obtain a graphic and text retrieval model, where the graphic and text retrieval model is used to generate a corresponding first image according to a first description text, or to generate a corresponding second description text according to a second image.

[0154] According to one or more embodiments of the present disclosure, obtaining a combined loss function of a model to be trained based on the sample data set includes: obtaining a new knowledge model corresponding to the model to be trained, where the new knowledge model is a result of training the model to be trained based on the first loss function and the sample data set; respectively encoding the sample data set through the model to be trained and the new knowledge model to obtain a first graphic-text feature corresponding to the model to be trained and a second graphic-text feature corresponding to the new knowledge model; obtaining a second loss function according to the similarity between the first graphic-text feature and the second graphic-text feature; and generating the combined loss function according to the first loss function and the second loss function.

[0155] According to one or more embodiments of the present disclosure, the model to be trained includes an image encoder and a text encoder, the first graphic-text feature includes a first image feature and a first text feature, and the second graphic-text feature includes a second image feature and a second text feature; the step of respectively encoding the sample data set through the model to be trained and the new knowledge model to obtain a first graphic-text feature corresponding to the model to be trained and a second graphic-text feature corresponding to the new knowledge model includes: inputting the sample image and the corresponding sample description text in each graphic-text sample data pair into the image encoder and the text encoder of the model to be trained respectively to obtain a first image feature and a first text feature; and inputting the sample image and the corresponding sample description text in each graphic-text sample data pair into the image encoder and the text encoder of the new knowledge model respectively to obtain a second image feature and a second text feature.

[0156] According to one or more embodiments of the present disclosure, the step of obtaining a second loss function according to the similarity between the first graphic-text feature and the second graphic-text feature includes: obtaining a first similarity square matrix according to the first graphic-text features corresponding to the graphic-text sample data pairs in the sample data set, where the first similarity square matrix represents the content relevance between each pair of sample images and sample description texts after being encoded by the model to be trained; obtaining a second similarity square matrix according to the second graphic-text features corresponding to the graphic-text sample data pairs in the sample data set; the second similarity square matrix represents the content relevance between each pair of sample images and sample description texts after being encoded by the new knowledge model; and fusing the first similarity square matrix and the second similarity square matrix to obtain a second loss function.

[0157] According to one or more embodiments of the present disclosure, fusing according to the first similarity square matrix and the second similarity square matrix to obtain a second loss function includes: creating an empty square matrix with a target dimension, where the target dimension is the square matrix dimension of the first similarity square matrix or the second similarity square matrix; if within the i-th row of the first similarity square matrix, the first square matrix value located on the diagonal is the maximum value among all the square matrix values within the i-th row of the first similarity square matrix, then assign the square matrix values of the i-th row of the first similarity square matrix to the square matrix values of the i-th row of the empty square matrix correspondingly; if within the i-th row of the first similarity square matrix, the first square matrix value located on the diagonal is not the maximum value among all the square matrix values within the i-th row of the first similarity square matrix, and within the i-th row of the second similarity square matrix, the square matrix value located on the diagonal is the maximum value among all the square matrix values within the i-th row of the second similarity square matrix, then regularize the square matrix values of the i-th row of the first similarity square matrix based on the square matrix values of the i-th row of the second similarity square matrix to obtain a regularization sequence corresponding to the i-th row, and assign the regularization square matrix values within the regularization sequence to the square matrix values of the i-th row of the empty square matrix correspondingly; if within the i-th row of the first similarity square matrix, the first square matrix value located on the diagonal is not the maximum value among all the square matrix values within the i-th row of the first similarity square matrix, and within the i-th row of the second similarity square matrix, the second square matrix value located on the diagonal is also not the maximum value among all the square matrix values within the i-th row of the second similarity square matrix, then assign 1 to the first square matrix value within the i-th row of the first similarity square matrix, and then regularize all the square matrix values within the i-th row of the first similarity square matrix to obtain a regularization sequence corresponding to the i-th row, and assign the regularization square matrix values within the regularization sequence to the square matrix values of the i-th row of the empty square matrix correspondingly, where i is any positive integer less than or equal to the target dimension; generate an updated first similarity square matrix according to the assigned empty square matrix; generate a second loss function based on the updated first similarity square matrix and the first similarity square matrix.

[0158] According to one or more embodiments of the present disclosure, after obtaining the sample data set, it further includes: preprocessing the sample images in the sample data set to obtain preprocessed images, where the preprocessed images lack the effective pixel information corresponding to at least one pixel point compared to the sample images; generating a preprocessed sample data set based on the preprocessed images and the corresponding sample description texts; the obtaining of the combined loss function of the model to be trained based on the sample data set includes: obtaining the combined loss function of the model to be trained based on the preprocessed sample data set.

[0159] According to one or more embodiments of the present disclosure, preprocessing the sample images in the sample dataset to obtain preprocessed images includes: expanding blank pixels outward from the edges of the sample images to obtain expanded images, where the image size of the expanded images is larger than the initial image size of the sample images; and cropping the expanded images based on the initial image size to obtain the preprocessed images.

[0160] According to one or more embodiments of the present disclosure, preprocessing the sample images in the sample dataset to obtain preprocessed images includes: obtaining an occlusion size within a preset range; randomly determining an occlusion area within the sample image based on the occlusion size, and randomly filling the pixel information of the pixel points within the occlusion area to obtain the preprocessed images.

[0161] According to one or more embodiments of the present disclosure, the to-be-trained model is implemented based on a pre-trained model for comparing text and images.

[0162] In a second aspect, according to one or more embodiments of the present disclosure, there is provided a training device for a text-image retrieval model, including:

[0163] An acquisition module, configured to acquire a sample dataset, where the sample dataset includes at least two text-image sample data pairs, the text-image sample data pairs include sample images and corresponding sample description texts, and the sample description texts are used to characterize the image content of the sample images;

[0164] A processing module, configured to obtain a combined loss function of the to-be-trained model based on the sample dataset, where the combined loss function is composed of a first loss function and a second loss function, the first loss function is used to guide the to-be-trained model to learn first detection knowledge corresponding to the sample dataset, and the second loss function is used to guide the to-be-trained model to fix existing second detection knowledge of the to-be-trained model;

[0165] A training module, configured to train the to-be-trained model based on the combined loss function and the sample dataset until convergence to obtain a text-image retrieval model, where the text-image retrieval model is used to generate a corresponding first image according to a first description text, or to generate a corresponding second description text according to a second image.

[0166] According to one or more embodiments of the present disclosure, the processing module is specifically configured to: obtain a new knowledge model corresponding to the model to be trained, where the new knowledge model is the result of training the model to be trained based on the first loss function and the sample data set; respectively encode the sample data set through the model to be trained and the new knowledge model to obtain a first graphic and text feature corresponding to the model to be trained and a second graphic and text feature corresponding to the new knowledge model; obtain a second loss function according to the similarity between the first graphic and text feature and the second graphic and text feature; generate the combined loss function according to the first loss function and the second loss function.

[0167] According to one or more embodiments of the present disclosure, the model to be trained includes an image encoder and a text encoder, the first graphic and text feature includes a first image feature and a first text feature, and the second graphic and text feature includes a second image feature and a second text feature; when the processing module encodes the sample data set through the model to be trained and the new knowledge model respectively to obtain the first graphic and text feature corresponding to the model to be trained and the second graphic and text feature corresponding to the new knowledge model, it is specifically configured to: input the sample image and the corresponding sample description text in each graphic and text sample data pair into the image encoder and the text encoder of the model to be trained respectively to obtain a first image feature and a first text feature; input the sample image and the corresponding sample description text in each graphic and text sample data pair into the image encoder and the text encoder of the new knowledge model respectively to obtain a second image feature and a second text feature.

[0168] According to one or more embodiments of the present disclosure, when the processing module obtains the second loss function according to the similarity between the first graphic and text feature and the second graphic and text feature, it is specifically configured to: obtain a first similarity matrix according to the first graphic and text features corresponding to each graphic and text sample data pair in the sample data set, where the first similarity matrix represents the content relevance between each pair of sample images and sample description texts after being encoded by the model to be trained; obtain a second similarity matrix according to the second graphic and text features corresponding to each graphic and text sample data pair in the sample data set; the second similarity matrix represents the content relevance between each pair of sample images and sample description texts after being encoded by the new knowledge model; fuse the first similarity matrix and the second similarity matrix to obtain the second loss function.

[0169] According to one or more embodiments of the present disclosure, when the processing module fuses the first similarity matrix and the second similarity matrix to obtain a second loss function, it is specifically configured to: create an empty matrix with a target dimension, where the target dimension is the matrix dimension of the first similarity matrix or the second similarity matrix; if, within the i-th row of the first similarity matrix, the first matrix value located on the diagonal is the maximum value among all the matrix values in the i-th row of the first similarity matrix, then assign the matrix values of the i-th row of the first similarity matrix to the matrix values of the i-th row of the empty matrix correspondingly; if, within the i-th row of the first similarity matrix, the first matrix value located on the diagonal is not the maximum value among all the matrix values in the i-th row of the first similarity matrix, and within the i-th row of the second similarity matrix, the matrix value located on the diagonal is the maximum value among all the matrix values in the i-th row of the second similarity matrix, then regularize the matrix values of the i-th row of the first similarity matrix based on the matrix values of the i-th row of the second similarity matrix to obtain a regularization sequence corresponding to the i-th row, and assign the regularization matrix values within the regularization sequence to the matrix values of the i-th row of the empty matrix correspondingly; if, within the i-th row of the first similarity matrix, the first matrix value located on the diagonal is not the maximum value among all the matrix values in the i-th row of the first similarity matrix, and within the i-th row of the second similarity matrix, the second matrix value located on the diagonal is also not the maximum value among all the matrix values in the i-th row of the second similarity matrix, then assign 1 to the first matrix value within the i-th row of the first similarity matrix, and then regularize all the matrix values within the i-th row of the first similarity matrix to obtain a regularization sequence corresponding to the i-th row, and assign the regularization matrix values within the regularization sequence to the matrix values of the i-th row of the empty matrix correspondingly, where i is any positive integer less than or equal to the target dimension; generate an updated first similarity matrix according to the assigned empty matrix; generate a second loss function based on the updated first similarity matrix and the first similarity matrix.

[0170] According to one or more embodiments of the present disclosure, after obtaining the sample data set, the processing module is further configured to: preprocess the sample images in the sample data set to obtain preprocessed images, where the preprocessed images lack effective pixel information corresponding to at least one pixel point compared to the sample images; generate a preprocessed sample data set based on the preprocessed images and the corresponding sample description texts; when the processing module obtains the combined loss function of the model to be trained based on the sample data set, it is specifically configured to: obtain the combined loss function of the model to be trained based on the preprocessed sample data set.

[0171] According to one or more embodiments of the present disclosure, when the processing module preprocesses the sample images in the sample dataset to obtain preprocessed images, it is specifically configured to: expand blank pixels outward from the edges of the sample images to obtain an expanded image, where the image size of the expanded image is larger than the initial image size of the sample images; and crop the expanded image based on the initial image size to obtain the preprocessed images.

[0172] According to one or more embodiments of the present disclosure, when the processing module preprocesses the sample images in the sample dataset to obtain preprocessed images, it is specifically configured to: obtain an occlusion size within a preset range; randomly determine an occlusion area within the sample image based on the occlusion size, and randomly fill the pixel information of the pixel points within the occlusion area to obtain the preprocessed images.

[0173] According to one or more embodiments of the present disclosure, the model to be trained is implemented based on a pre-trained model for comparing text and images.

[0174] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, including: at least one processor and a memory;

[0175] The memory stores computer-executable instructions;

[0176] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method for training the text-image retrieval model as described in the first aspect above and various possible designs of the first aspect.

[0177] In a fourth aspect, according to one or more embodiments of the present disclosure, there is provided a computer-readable storage medium, in which computer-executable instructions are stored, and when the processor executes the computer-executable instructions, the method for training the text-image retrieval model as described in the first aspect above and various possible designs of the first aspect is implemented.

[0178] In a fifth aspect, according to one or more embodiments of the present disclosure, there is provided a computer program product, including a computer program, and when the computer program is executed by the processor, the method for training the text-image retrieval model as described in the first aspect above and various possible designs of the first aspect is implemented.

[0179] The above description is only a preferred embodiment of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) having similar functions disclosed in the present disclosure.

[0180] In addition, although the operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0181] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms for implementing the claims.

Claims

1. A method for training a graphic and text retrieval model, characterized in that Including: Obtain a sample data set, where the sample data set includes at least two image-text sample data pairs. The image-text sample data pair includes a sample image and a sample description text corresponding to the sample image, and the sample description text is used to characterize the image content of the sample image; Based on the sample data set, obtain a combined loss function of the model to be trained. Among them, the combined loss function is composed of a first loss function and a second loss function. The first loss function is used to guide the model to be trained to learn the first detection knowledge corresponding to the sample data set, and the second loss function is used to guide the model to be trained to fix the existing second detection knowledge of the model to be trained; Based on the combined loss function and the sample data set, train the model to be trained until convergence to obtain an image-text retrieval model. Among them, the image-text retrieval model is used to generate a corresponding first image according to the first description text, or to generate a corresponding second description text according to the second image.

2. The method according to claim 1, wherein The obtaining the combined loss function of the model to be trained based on the sample data set includes: Obtain a new knowledge model corresponding to the model to be trained. The new knowledge model is the result of training the model to be trained based on the first loss function and the sample data set; Encode the sample data set through the model to be trained and the new knowledge model respectively to obtain the first image-text feature corresponding to the model to be trained and the second image-text feature corresponding to the new knowledge model; Obtain a second loss function according to the similarity between the first image-text feature and the second image-text feature; Generate the combined loss function according to the first loss function and the second loss function.

3. The method according to claim 2, wherein The model to be trained includes an image encoder and a text encoder. The first image-text feature includes a first image feature and a first text feature, and the second image-text feature includes a second image feature and a second text feature; The encoding the sample data set through the model to be trained and the new knowledge model respectively to obtain the first image-text feature corresponding to the model to be trained and the second image-text feature corresponding to the new knowledge model includes: Input the sample image and the corresponding sample description text in each image-text sample data pair into the image encoder and the text encoder of the model to be trained respectively to obtain a first image feature and a first text feature; Input the sample image and the corresponding sample description text in each image-text sample data pair into the image encoder and the text encoder of the new knowledge model respectively to obtain a second image feature and a second text feature.

4. The method according to claim 2, characterized in that, The obtaining the second loss function according to the similarity between the first image-text feature and the second image-text feature includes: According to the first image-text features corresponding to each image-text sample data pair in the sample data set, obtain a first similarity matrix, and the first similarity matrix characterizes the content relevance between each pair of sample images and sample description texts after being encoded by the model to be trained; Based on the second graphic features corresponding to each of the graphic sample data pairs in the sample data set, a second similarity matrix is obtained; the second similarity matrix characterizes the content relevance between each pair of sample images and sample description texts after being encoded by the new knowledge model; Based on the fusion of the first similarity matrix and the second similarity matrix, a second loss function is obtained.

5. The method according to claim 4, characterized in that, Based on the fusion of the first similarity matrix and the second similarity matrix, obtaining a second loss function includes: Create an empty matrix with the target dimension, where the target dimension is the matrix dimension of the first similarity matrix or the second similarity matrix; If, within the i-th row of the first similarity matrix, the first matrix value on the diagonal is the maximum value among all the matrix values in the i-th row of the first similarity matrix, then assign the matrix values of the i-th row of the first similarity matrix to the matrix values of the i-th row of the empty matrix correspondingly; If, within the i-th row of the first similarity matrix, the first matrix value on the diagonal is not the maximum value among all the matrix values in the i-th row of the first similarity matrix, and within the i-th row of the second similarity matrix, the matrix value on the diagonal is the maximum value among all the matrix values in the i-th row of the second similarity matrix, then regularize the matrix values of the i-th row of the first similarity matrix based on the matrix values of the i-th row of the second similarity matrix to obtain a regularization sequence corresponding to the i-th row, and assign the regularization matrix values within the regularization sequence to the matrix values of the i-th row of the empty matrix correspondingly; If, within the i-th row of the first similarity matrix, the first matrix value on the diagonal is not the maximum value among all the matrix values in the i-th row of the first similarity matrix, and within the i-th row of the second similarity matrix, the second matrix value on the diagonal is also not the maximum value among all the matrix values in the i-th row of the second similarity matrix, then assign 1 to the first matrix value within the i-th row of the first similarity matrix, and then regularize all the matrix values within the i-th row of the first similarity matrix to obtain a regularization sequence corresponding to the i-th row, and assign the regularization matrix values within the regularization sequence to the matrix values of the i-th row of the empty matrix correspondingly, where i is any positive integer less than or equal to the target dimension; Based on the assigned empty matrix, generate an updated first similarity matrix; Based on the updated first similarity matrix and the first similarity matrix, generate a second loss function.

6. The method according to claim 1, characterized in that After obtaining the sample data set, it further includes: Preprocess the sample images in the sample data set to obtain preprocessed images, where the preprocessed images lack the effective pixel information corresponding to at least one pixel point compared to the sample images; Based on the preprocessed images and the corresponding sample description texts, generate a preprocessed sample data set; The obtaining the combined loss function of the model to be trained based on the sample data set includes: Based on the preprocessed sample data set, obtain the combined loss function of the model to be trained.

7. The method according to claim 6, wherein The preprocessing the sample images in the sample data set to obtain preprocessed images includes: Perform blank pixel expansion outward from the edge of the sample image to obtain an expanded image, where the image size of the expanded image is larger than the initial image size of the sample image; Crop the expanded image based on the initial image size to obtain the preprocessed image.

8. The method according to claim 6, characterized in that The preprocessing of the sample images in the sample dataset to obtain preprocessed images includes: Obtain the occlusion size within a preset range; Randomly determine an occlusion area within the sample image based on the occlusion size, and randomly fill the pixel information of the pixel points within the occlusion area to obtain the preprocessed image.

9. The method according to claim 6, wherein The model to be trained is implemented based on a pre-trained model for comparing text and images.

10. An apparatus for training a graphic and text retrieval model, characterized in that It includes: An acquisition module for acquiring a sample dataset, where the sample dataset includes at least two text-image sample data pairs. The text-image sample data pair includes a sample image and a sample description text corresponding to the sample image, and the sample description text is used to characterize the image content of the sample image; A processing module for obtaining a combined loss function of the model to be trained based on the sample dataset. The combined loss function is composed of a first loss function and a second loss function. The first loss function is used to guide the model to be trained to learn the first detection knowledge corresponding to the sample dataset, and the second loss function is used to guide the model to be trained to fix the existing second detection knowledge of the model to be trained; A training module for training the model to be trained based on the combined loss function and the sample dataset until convergence to obtain a text-image retrieval model, where the text-image retrieval model is used to generate a corresponding first image according to a first description text, or to generate a corresponding second description text according to a second image.

11. An electronic device, characterized in that, It includes: A processor and a memory; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory, so that the processor executes the text-image retrieval model training method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, Computer execution instructions are stored in the computer-readable storage medium. When the processor executes the computer execution instructions, the text-image retrieval model training method according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, the text-image retrieval model training method according to any one of claims 1 to 9 is implemented.

Images