A model training and retrieval method, computer program product, device, medium

Abstract

The present invention discloses a model training and retrieval method, a computer program product, a device, and a medium, and relates to the field of artificial intelligence technology, and includes obtaining an image text recognition model to be trained; obtaining a target number of training samples, wherein the training samples include image text pairs consisting of images and texts; obtaining recognition results of the image text recognition model on the training samples; processing the recognition results according to a target loss function to generate a loss value of the image text recognition model; and updating the image text recognition model according to the loss value; wherein the target loss function includes a similarity loss term, and the similarity loss term is used to constrain the similarity between the image features and the text features of the positive sample pair to approach 1, and constrain the similarity between the image features and the text features of the negative sample pair to approach 0. The present invention decouples the similarity between the positive sample pair and the negative sample pair, realizes precise control of the similarity between the positive sample pair and the negative sample pair, and can accurately recognize images and texts.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a model training and retrieval method, computer program product, device, and medium. Background Art

[0002] Image and text contrastive learning is an important research direction in the field of computer vision and natural language processing. Its goal is to enable computers to understand and reason about the relationship between images and text by jointly learning them.

[0003] In recent years, with the rapid development of deep learning, image and text contrast learning has made significant progress. However, in the process of training the image text recognition model, the ability to further enhance the similarity of positive sample pairs and reduce the similarity of negative sample pairs is limited, resulting in poor accuracy of the image text recognition model and inability to accurately identify the association between images and text. Among them, the positive sample pair refers to the image-text pair consisting of an image and the corresponding text description, and the negative sample pair refers to the image-text pair consisting of an image and other unrelated text descriptions.

[0004] In summary, how to accurately recognize images and texts is a problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0005] The purpose of the present invention is to provide a model training method, which can solve the technical problem of how to accurately recognize images and texts to a certain extent. The present invention also provides a retrieval method, a computer program product, an electronic device and a computer-readable storage medium.

[0006] In a first aspect, a model training method is provided, comprising:

[0007] Get the image text recognition model to be trained;

[0008] Obtaining a target number of training samples, wherein the training samples include image-text pairs consisting of an image and a text;

[0009] Obtaining a recognition result of the image text recognition model on the training sample;

[0010] Processing the recognition result according to the target loss function to generate a loss value of the image text recognition model;

[0011] Update the image text recognition model according to the loss value;

[0012] The target loss function includes a similarity loss term, which is used to constrain the similarity between the image features and the text features of the positive sample pair to approach 1, and to constrain the similarity between the image features and the text features of the negative sample pair to approach 0.

[0013] On the other hand, the image text recognition model includes an image encoding module and a text encoding module;

[0014] Obtaining the recognition result of the image text recognition model on the training sample includes:

[0015] Obtaining an image embedding vector obtained by extracting the image in the training sample by the image encoding module, wherein the image embedding vector includes an image classification embedding vector;

[0016] Obtaining a text embedding vector obtained by extracting the text in the training sample by the text encoding module, wherein the text embedding vector includes a text classification embedding vector;

[0017] The image embedding vector and the text embedding vector are used as the recognition result.

[0018] On the other hand, the recognition result is processed according to the target loss function to generate a loss value of the image text recognition model, including:

[0019] Combining the target number of image classification embedding vectors into a column vector;

[0020] Combining the target number of text classification embedding vectors into a row vector;

[0021] Forming a similarity matrix based on the column vector and the row vector;

[0022] Determining a target dimension for elements in the similarity matrix;

[0023] Generate a first matrix with rows and columns of the target dimension, the first matrix including a matrix with diagonal elements of 1 and other elements of 0;

[0024] Generate a second matrix with rows and columns of the target dimension, the second matrix including a matrix with diagonal elements of 0 and other elements of 1;

[0025] Processing the similarity matrix based on the first matrix and the second matrix to generate a value of the similarity loss term;

[0026] The value of the similarity loss term and the recognition result are processed according to the target loss function to generate a loss value of the image text recognition model.

[0027] On the other hand, processing the similarity matrix based on the first matrix and the second matrix to generate the value of the similarity loss term includes:

[0028] Applying a constant coefficient and an F-norm, performing a matrix position-wise multiplication process on the similarity matrix based on the first matrix and the second matrix, and generating a value of the similarity loss term.

[0029] On the other hand, the objective loss function includes a constraint term of the similarity loss term, the constraint term changes with the number of training times, and gradually maximizes the similarity of positive sample pairs and minimizes the similarity of negative sample pairs.

[0030] On the other hand, the value of the similarity loss term and the recognition result are processed according to the target loss function to generate a loss value of the image text recognition model, including:

[0031] Determining a cycle number for training the image text recognition model;

[0032] Determine the constraint item which takes the periodic ordinal number as an independent variable and increases monotonically;

[0033] Processing the current cycle ordinal number based on the constraint item to generate a value of the constraint item;

[0034] The value of the constraint item, the value of the similarity loss item and the recognition result are processed according to the target loss function to generate a loss value of the image text recognition model.

[0035] On the other hand, the current cycle ordinal number is processed based on the constraint item to generate the value of the constraint item, including:

[0036] An activation function is performed on the current cycle ordinal number based on the constraint item to generate a value of the constraint item.

[0037] On the other hand, the value of the constraint item, the value of the similarity loss item and the recognition result are processed according to the target loss function to generate the loss value of the image text recognition model, including:

[0038] Generate an image-text contrast loss item from each image to all texts in the training sample according to the text classification embedding vector and the image classification embedding vector;

[0039] Generate a text-image contrast loss term from each text in the training sample to all images according to the text classification embedding vector and the image classification embedding vector;

[0040] Generate a target contrast loss term according to the image-text contrast loss term and the text-image contrast loss term;

[0041] The target contrast loss term, the value of the constraint term and the value of the similarity loss term are processed according to the target loss function to generate a loss value of the image text recognition model.

[0042] On the other hand, according to the text classification embedding vector and the image classification embedding vector, an image-text contrast loss item from each image to all texts in the training sample is generated, including:

[0043] Applying a logarithmic function, an exponential function, a vector modulus and a temperature coefficient of contrast loss, an image-text contrast loss term from each image to all texts in the training sample is generated according to the text classification embedding vector and the image classification embedding vector.

[0044] On the other hand, according to the text classification embedding vector and the image classification embedding vector, a text image contrast loss item from each text in the training sample to all images is generated, including:

[0045] A logarithmic function, an exponential function, a vector modulus and a temperature coefficient of contrast loss are applied to generate a text-image contrast loss term from each text in the training sample to all images according to the text classification embedding vector and the image classification embedding vector.

[0046] On the other hand, generating a target contrast loss term according to the image-text contrast loss term and the text-image contrast loss term includes:

[0047] A constant coefficient is applied to perform weighted averaging on the image-text contrast loss term and the text-image contrast loss term to generate a target contrast loss term.

[0048] On the other hand, the target contrast loss term, the value of the constraint term, and the value of the similarity loss term are processed according to the target loss function to generate a loss value of the image text recognition model, including:

[0049] The target contrast loss term, the value of the constraint term and the value of the similarity loss term are multiplied according to the target loss function to generate a loss value of the image text recognition model.

[0050] In a second aspect, a retrieval method is provided, comprising:

[0051] Acquire an object to be retrieved, wherein the object to be retrieved includes text or an image;

[0052] Acquire a candidate matching object; when the object to be retrieved is text, the candidate matching object is an image; when the object to be retrieved is an image, the candidate matching object is text;

[0053] Processing the object to be retrieved by using a trained image text recognition model to obtain features of the object to be retrieved;

[0054] Processing the candidate matching object by the image text recognition model to obtain features of the candidate matching object;

[0055] Calculating the similarity value between the feature of the object to be retrieved and the feature of each candidate matching object;

[0056] Taking the similarity value greater than or equal to a preset threshold as a target value;

[0057] Determine the candidate matching object corresponding to the target value as the search result of the object to be searched;

[0058] Among them, the training process of the image text recognition model includes: obtaining a target number of training samples, the training samples include image-text pairs consisting of images and texts; obtaining the recognition results of the image text recognition model for the training samples; processing the recognition results according to the target loss function to generate a loss value of the image text recognition model; updating the image text recognition model according to the loss value; wherein the target loss function includes a similarity loss term, and the similarity loss term is used to constrain the similarity between the image features and the text features of the positive sample pair to approach 1, and constrain the similarity between the image features and the text features of the negative sample pair to approach 0.

[0059] According to a third aspect, a computer program product is provided, comprising a computer program / instruction, which implements the steps of any of the above methods when executed by a processor.

[0060] In a fourth aspect, an electronic device is provided, including:

[0061] Memory for storing computer programs;

[0062] A processor is used to implement the steps of any of the above methods when executing the computer program.

[0063] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0064] The present invention provides a model training method, which comprises the following steps: obtaining an image text recognition model to be trained; obtaining a target number of training samples, wherein the training samples include image text pairs consisting of images and texts; obtaining recognition results of the image text recognition model on the training samples; processing the recognition results according to a target loss function to generate a loss value of the image text recognition model; and updating the image text recognition model according to the loss value; wherein the target loss function includes a similarity loss term, and the similarity loss term is used to constrain the similarity between the image features and the text features of a positive sample pair to approach 1, and to constrain the similarity between the image features and the text features of a negative sample pair to approach 0. The beneficial effect of the present invention is that in the training process of the image text recognition model, a similarity loss term is set in the target loss function for calculating the loss value, so as to constrain the similarity between the image features and the text features of the positive sample pair to approach 1 and the similarity between the image features and the text features of the negative sample pair to approach 0 by means of the similarity loss term, so that the similarity of the positive and negative sample pairs is different, thereby decoupling the similarity of the positive sample pair and the negative sample pair, and realizing targeted and accurate control of the similarity of the positive sample pair and the negative sample pair, so that a high-precision image text recognition model can be trained, and then the image and text can be accurately recognized. A retrieval method, a computer program product, an electronic device and a computer-readable storage medium provided by the present invention also solve the corresponding technical problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0066] Figure 1 A flow chart of a model training method provided by an embodiment of the present invention;

[0067] Figure 2 This is a structural diagram of the image text recognition model;

[0068] Figure 3 A flowchart of a retrieval method provided by an embodiment of the present invention;

[0069] Figure 4 It is a schematic diagram of the structure of the image retrieval system;

[0070] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0071] Figure 6 Another structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] See also Figure 1 , Figure 1 A flowchart of a model training method provided in an embodiment of the present invention.

[0074] A model training method provided by an embodiment of the present invention may include the following steps:

[0075] Step S101: Obtain an image text recognition model to be trained.

[0076] In practical applications, the image text recognition model to be trained can be obtained first. The image text recognition model is a model for recognizing images and texts, and its type and structure can be flexibly determined according to the application scenario.

[0077] Step S102: Obtain a target number of training samples, where the training samples include image-text pairs consisting of images and texts.

[0078] In practical applications, after obtaining the image text recognition model to be trained, the target number of training samples can be obtained. The target number can be flexibly determined according to the training requirements, and each training sample is an image-text pair consisting of an image and a text, wherein when the image corresponds to the text, the training sample is a positive sample, and when the image is not related to the text, the training sample is a negative sample. Specifically, the image and the text can be paired to form an "image-text pair" to ensure a clear correspondence between them. The pairing can be achieved by associating the image's annotation information or the image's file name with the text. For example, the image's label, title, description, or related text description can be used to correspond to the image. For ease of understanding, assuming that the image is a cat, when the text description is a language describing a cat, the training sample is a positive sample, and when the text description is a language describing a dog, the training sample is a negative sample.

[0079] In specific application scenarios, a large-scale unprocessed image and text dataset can be collected. After the pairing of images and texts is completed, preprocessing is required. For example, for images, the preprocessing operations can be resizing, cropping, normalization, etc. to ensure that all have consistent sizes and formats; for text, word segmentation, stop word removal, stem extraction, etc. can be performed to obtain a more standardized and comparable text representation. In addition, in order to increase the diversity and robustness of the data, data enhancement operations can also be used. For example, for images, random cropping, flipping, rotation, etc. can be performed to increase the diversity of the data; for text, random replacement, deletion, insertion, etc. can be performed to expand the diversity of text data; thus, through these operations, more diverse positive and negative sample pairs are generated, which is convenient for improving the generalization ability of the model. Through the above operations, a training data set for the present invention can be produced. Such a training data set can be used to jointly train image and text models so that they can learn the connection between vision and semantics, thereby realizing the joint representation learning of images and texts. Correspondingly, the training samples can be selected from the training data set. It should be noted that the number of samples in the training data set can be flexibly adjusted. For example, it can include at least 1 million image and text pairs. More image-text pairs will help train a more accurate model. On the contrary, if the number of image-text pairs in the data set is small, the accuracy of the model will be weakened. The images can be in RGB (red, green, blue) color format, and the text can be in Chinese or English sentences.

[0080] Step S103: Obtain the recognition result of the image text recognition model on the training sample.

[0081] In practical applications, after obtaining the image text recognition model and training samples, the training samples can be used to train the image text recognition model, that is, the training samples need to be input into the image text recognition model for processing. Correspondingly, since the recognition results of the image text recognition model for the training samples reflect the recognition ability of the image text recognition model, it is necessary to obtain the recognition results to adjust the image text recognition model.

[0082] In specific application scenarios, such as Figure 2As shown, the image text recognition model may include an image encoding module and a text encoding module; the image encoding module is used to extract features from the image to obtain an image embedding vector, the image embedding vector includes an image patch embedding vector and an image classification embedding vector, the image patch is obtained by cutting the image, and the relationship between the image classification embedding vector and the image patch embedding vector can be flexibly determined according to actual needs; the text encoding module is used to extract features from the text to obtain a text embedding vector, the text embedding vector includes a word element embedding vector and a text classification embedding vector, the word element is obtained by segmenting the text, and the relationship between the text classification embedding vector and the word element embedding vector can be flexibly determined according to actual needs. At this time, the image embedding vector and the text embedding vector can be used as the recognition result.

[0083] It should be noted that the original image processed by the image encoding module can be preprocessed. For example, the long side of the original image can be scaled to L pixels. L can be 256, 384, 512, etc. Preferably, L=384 can be set, and then the short side of the original image can be scaled proportionally; finally, the short side is padded with zeros to L pixels. In this way, after the above preprocessing operation, an image with equal width and height and L pixels can be obtained. The preprocessed image is the input image of the image encoding module. And the image encoding module E I The input image is cut into a set number of image slices. For example, if the set number is 256, the image can be divided into 256 non-overlapping image slices of the same size along the width and height of the input image. The width of each image slice is L / 16 pixels, and the height is also L / 16 pixels. For the convenience of description, the total number of image slices is defined as p, then p=256. After the image encoding module extracts features from the image slices, the image embedding vector F can be obtained. I , including p image slice embedding vectors and an image classification embedding vector for classification , that is , the dimension of each embedding vector in the image embedding vector is d, and the value of d can be flexibly determined, such as 1024, etc., that is, each embedding vector in the image embedding vector can be regarded as a d-dimensional column vector.

[0084] It should also be noted that the text encoding module E TUse the word segmenter and the corresponding word element (token) embedding vector table to extract features from the text. The word segmenter and word element embedding vector table of the text encoding module can be fixed settings and are not updated during the training process. In the process of extracting features from the text, the word segmenter is used to divide the text T into words, and the total number of words is defined as q. q can change dynamically according to the input text T. Then, the embedding vectors corresponding to these words are retrieved from the word element embedding vector table and input into the text encoding module. After the forward processing of the text encoding module, the text embedding vector F of the text T is extracted. T , F T Includes q word embedding vectors and a text classification embedding vector for classification , that is , the dimension of each embedding vector in the text embedding vector is also d, that is, each embedding vector in the text embedding vector can be regarded as a d-dimensional column vector. At this time, for the i-th training sample pair, image I i The image embedding vector obtained by the image encoding module is defined as F Ii , , text T i The text embedding vector obtained by the text encoding module is F Ti , .

[0085] Step S104: Process the recognition result according to the target loss function to generate a loss value of the image text recognition model; the target loss function includes a similarity loss term, and the similarity loss term is used to constrain the similarity between the image features and the text features of the positive sample pair to approach 1, and to constrain the similarity between the image features and the text features of the negative sample pair to approach 0.

[0086] Step S105: Update the image text recognition model according to the loss value.

[0087] In practical applications, after obtaining the recognition result, the target loss function set by the present invention needs to be applied to process the recognition result to generate the loss value of the image text recognition model, so as to quantify the recognition ability of the image text recognition model with the help of the loss value; and the target loss function in the present invention includes a similarity loss term, which is used to constrain the similarity between the image features and the text features of the positive sample pair to approach 1, and constrain the similarity between the image features and the text features of the negative sample pair to approach 0, and the specific degree of approach can be flexibly determined according to the actual operation. After obtaining the loss value, the image text recognition model can be updated according to the loss value, such as using the stochastic gradient descent algorithm to update the parameters of the image text recognition model according to the loss value.

[0088] In practical applications, in the process of processing the recognition results according to the target loss function to generate the loss value of the image text recognition model, the target number of image classification embedding vectors can be composed of column vectors; the target number of text classification embedding vectors can be composed of row vectors; a similarity matrix can be composed based on the column vectors and the row vectors; the target dimension of the elements in the similarity matrix can be determined; a first matrix with both rows and columns being the target dimension is generated, the first matrix includes a matrix with a diagonal of 1 and other elements being 0; a second matrix with both rows and columns being the target dimension is generated, the second matrix includes a matrix with a diagonal of 0 and other elements being 1; the similarity matrix is ​​processed based on the first matrix and the second matrix to generate the value of the similarity loss term; the value of the similarity loss term and the recognition result are processed according to the target loss function to generate the loss value of the image text recognition model.

[0089] In a specific application scenario, the similarity loss term may be specifically a regularized formula, that is, in the process of processing the similarity matrix based on the first matrix and the second matrix to generate the value of the similarity loss term, the constant coefficient and the F norm may be applied, and the similarity matrix may be processed by positional multiplication based on the first matrix and the second matrix to generate the value of the similarity loss term. For example, the similarity matrix may be processed based on the first matrix and the second matrix according to the similarity loss term generation formula to generate the value of the similarity loss term; the similarity loss term generation formula includes:

[0090] ;

[0091] ;

[0092] Among them, R represents the value of the similarity loss term; represents the preset constant coefficient, whose value can be flexibly adjusted, for example, it can be 0.5, 0.6, etc.; M represents the similarity matrix; represents the F-norm; Indicates that matrices are multiplied by position, that is, matrices are multiplied pixel by pixel; represents the first matrix, represents the second matrix, d represents the target dimension; represents the image classification embedding vector of the i-th training sample, represents the text classification embedding vector of the i-th training sample, , B represents the target number; tr represents transposition.

[0093] In practical applications, a monotonically increasing control coefficient can also be applied to adjust the regularization term, and the role of the constraint term can be gradually increased during the training process, so that as the model training proceeds, the similarity of the positive sample pairs can be significantly maximized and the similarity of the negative sample pairs can be minimized, thereby further improving the training accuracy of the image text recognition model. In other words, the target loss function of the present invention includes a constraint term of the similarity loss term, and the constraint term changes with the number of training times, and gradually maximizes the similarity of the positive sample pairs and minimizes the similarity of the negative sample pairs.

[0094] In a specific application scenario, in the process of processing the value of the similarity loss item and the recognition result according to the target loss function to generate the loss value of the image text recognition model, the period ordinal number for training the image text recognition model can be determined; a constraint item with the period ordinal number as the independent variable and monotonically increasing is determined, and the value of the constraint item is greater than or equal to 0; the current period ordinal number is processed based on the constraint item to generate the value of the constraint item; the value of the constraint item, the value of the similarity loss item and the recognition result are processed according to the target loss function to generate the loss value of the image text recognition model.

[0095] In a specific application scenario, in the process of processing the current cycle ordinal based on the constraint item and generating the value of the constraint item, the current cycle ordinal can be processed by an activation function based on the constraint item to generate the value of the constraint item. For example, the current cycle ordinal can be processed based on the constraint item through a constraint item generation formula to generate the value of the constraint item. The constraint item generation formula includes:

[0096] ;

[0097] Among them, t represents the ordinal number of the current cycle, , E represents the total number of training cycles; sigmold represents the activation function; e represents the natural constant; Represents the value of the constraint item.

[0098] In practical applications, in the process of processing the values ​​of constraint items, similarity loss items and recognition results according to the target loss function to generate the loss value of the image text recognition model, the stochastic gradient descent algorithm can be used to jointly train the image encoding module and the text encoding module to analyze the loss value, that is, the image-text contrast loss item from each image to all texts in the training sample can be generated according to the text classification embedding vector and the image classification embedding vector; the text-image contrast loss item from each text in the training sample to all images can be generated according to the text classification embedding vector and the image classification embedding vector; the target contrast loss item is generated according to the image-text contrast loss item and the text-image contrast loss item; the target contrast loss item, the values ​​of the constraint item and the similarity loss item are processed according to the target loss function to generate the loss value of the image text recognition model.

[0099] From this implementation process, it can be seen that the present invention takes into account the corresponding relationship between the image and the text, or the irrelevance between the image and the text, so according to the text classification embedding vector and the image classification embedding vector, an image-text comparison loss item is generated from each image to all texts in the training sample to analyze the correlation between each image and each text; according to the text classification embedding vector and the image classification embedding vector, a text-image comparison loss item is generated from each text in the training sample to all images to analyze the correlation between each text and each image; according to the image-text comparison loss item and the text-image comparison loss item, a target comparison loss item is generated to synthesize the correlation analysis results between the text and the image; finally, according to the target loss function, the target comparison loss item, the constraint item value and the similarity loss item value are processed to generate the loss value of the image text recognition model, which is equivalent to comprehensively evaluating the recognition ability of the image text recognition model by the correlation between all images and all texts, thereby improving the effectiveness of the loss value and facilitating the high-precision training of the image text recognition model.

[0100] In a specific application scenario, in the process of generating an image-text contrast loss item from each image to all texts in a training sample according to a text classification embedding vector and an image classification embedding vector, a logarithmic function, an exponential function, a vector modulus, and a temperature coefficient of contrast loss can be applied to generate an image-text contrast loss item from each image to all texts in a training sample according to the text classification embedding vector and the image classification embedding vector. For example, an image-text contrast loss item generation formula can be used to generate an image-text contrast loss item from each image to all texts in a training sample according to the text classification embedding vector and the image classification embedding vector; the image-text contrast loss item generation formula includes:

[0101] ;

[0102] in, represents the image-text contrast loss term from the image of the i-th training sample to all texts; log represents the logarithmic function; represents the exponential function; Represents the image classification embedding vector of the jth training sample; The temperature coefficient representing the contrast loss, whose value can be determined flexibly, for example, it can be 0.9, 1.0, 1.1, etc.; , u and v represent column vectors, Represents the vector magnitude.

[0103] In a specific application scenario, in the process of generating a text-image contrast loss item from each text in a training sample to all images according to a text classification embedding vector and an image classification embedding vector, a logarithmic function, an exponential function, a vector modulus, and a temperature coefficient of contrast loss can be applied to generate a text-image contrast loss item from each text in a training sample to all images according to the text classification embedding vector and the image classification embedding vector. For example, a text-image contrast loss item generation formula can be used to generate a text-image contrast loss item from each text in a training sample to all images according to the text classification embedding vector and the image classification embedding vector. The text-image contrast loss item generation formula includes:

[0104] ;

[0105] in, Represents the text-image contrast loss term from the text of the i-th training sample to all images; Represents the text classification embedding vector of the jth training sample.

[0106] In a specific application scenario, in the process of generating a target contrast loss item according to an image-text contrast loss item and a text-image contrast loss item, a constant coefficient may be applied to perform a weighted average of the image-text contrast loss item and the text-image contrast loss item to generate a target contrast loss item. For example, a target contrast loss item generation formula may be used to generate a target contrast loss item according to an image-text contrast loss item and a text-image contrast loss item. The target contrast loss item generation formula includes:

[0107] ;

[0108] in, represents the target contrast loss term; Represents a constant coefficient, whose value can be determined flexibly, for example, it can be 0.5, 0.4, 0.6, etc.

[0109] In practical applications, in the process of processing the target contrast loss item, the value of the constraint item and the value of the similarity loss item according to the target loss function to generate the loss value of the image text recognition model, the target contrast loss item, the value of the constraint item and the value of the similarity loss item can be multiplied according to the target loss function to generate the loss value of the image text recognition model. For example, a formula can be generated by the target loss function to process the target contrast loss item, the value of the constraint item and the value of the similarity loss item according to the target loss function to generate the loss value of the image text recognition model; the target loss function generation formula includes:

[0110] ; Where L represents the loss value.

[0111] In this way, at the beginning of training, the constraint term is close to zero, and the target contrast loss term in the loss function L plays a major role. As the training process progresses, the constraint term is gradually added to the loss function L, showing the similarity between the constrained text embedding vector and the image embedding vector.

[0112] It should be noted that the objective loss function generation formula can be expanded as:

[0113] ;

[0114] Or further expanded as:

[0115] .

[0116] The present invention provides a model training method, which comprises the following steps: obtaining an image text recognition model to be trained; obtaining a target number of training samples, wherein the training samples include image text pairs consisting of images and texts; obtaining recognition results of the image text recognition model on the training samples; processing the recognition results according to a target loss function to generate a loss value of the image text recognition model; and updating the image text recognition model according to the loss value; wherein the target loss function includes a similarity loss term, and the similarity loss term is used to constrain the similarity between the image features and the text features of a positive sample pair to approach 1, and to constrain the similarity between the image features and the text features of a negative sample pair to approach 0. The beneficial effect of the present invention is that during the training process of the image text recognition model, a similarity loss term is set in the target loss function for calculating the loss value, so as to constrain the similarity between the image features and the text features of the positive sample pair to be close to 1, and constrain the similarity between the image features and the text features of the negative sample pair to be close to 0 with the help of the similarity loss term, so that the similarities of the positive and negative sample pairs are different, thereby decoupling the similarities of the positive sample pair and the negative sample pair, and achieving targeted and accurate control of the similarity of the positive sample pair and the negative sample pair, so that a high-precision image text recognition model can be trained, and then the image and text can be accurately recognized.

[0117] It should be noted that in one round of training, the image text recognition model needs to be trained multiple times, and each training uses the target loss function of the present invention to generate a loss value to adjust the parameters of the image text recognition model. For ease of understanding, it is assumed that the image text recognition model is trained using a small batch training method.

[0118] Assume that the total training cycle (epoch) is 20 cycles. One training cycle means that all training samples are used once without repetition. In order to avoid the order of samples in the data set affecting the performance of the model and to improve the generalization performance of the model, at the beginning of each training cycle, all samples in the training data set are randomly rearranged. In each training cycle, the disrupted training data set is divided into smaller sets. This small sample set is called a small batch of samples. Depending on the training configuration, the number of samples in each small batch can be set to 128, 256 or 512, etc. The model training algorithm reads a small batch of training samples each time and uses these samples to calculate the loss value, update the model parameters, etc. When a training cycle (epoch) ends, the training algorithm enters the next training cycle (epoch). When the total number of training cycles reaches the preset value, the model training ends. It should be noted that: in special cases, within a training cycle, the total number of samples in the data set cannot be divided into integer batches, and some individual samples are not used (discarded directly) or are used multiple times (to complete the data, for example: adding several used samples to the end of the data set). The impact of these individual cases on the training results can be ignored.

[0119] In the training process of the image text recognition model, the training input (initial parameter setting) is: read data from the training data set D, the number of training samples in a small batch is B=512; the initial training cycle t=1, the total training cycle E=20; the parameters related to the stochastic gradient descent algorithm: the initial learning rate is Lr=0.1, the momentum coefficient β=0.9, the weight decay coefficient is decay=0.0005, the learning rate drops to 0.01 in the 10th training cycle, the learning rate drops to 0.001 in the 15th training cycle, and the learning rate drops to 0.0001 in the 18th training cycle. The training process of the image text recognition model can be as follows:

[0120] Initialize the image encoding module and text encoding module in the image text recognition model. For example, the parameters in the image encoding module can be initialized with the parameters of the known ViT-L / 16 model pre-trained on the ImageNet-21k dataset; the parameters in the text encoding module can be initialized with the parameters of the known BERT_LARGE model pre-trained on the "General Language Understanding Evaluation Benchmark Set";

[0121] When t≤E, perform the following operations: t=t+1, randomly shuffle the order of samples in the training data set D, select a batch of B training samples from the training data set D, calculate the loss value L based on the above training samples and parameter settings, and then use the stochastic gradient descent algorithm to update the model parameters;

[0122] Repeat the steps of selecting a batch of B training samples from the training data set D, calculating the loss value L by applying the target loss function of the present invention according to the training samples and parameter settings, and then updating the model parameters by using the stochastic gradient descent algorithm until all samples of the data set D are used;

[0123] End the current training cycle; output the image encoding module and the text encoding module.

[0124] See also Figure 3 , Figure 3 A flowchart of a retrieval method provided by an embodiment of the present invention.

[0125] A search method provided by an embodiment of the present invention may include the following steps:

[0126] Step S201: Acquire the object to be retrieved, which includes text or image.

[0127] Step S202: Obtain candidate matching objects; when the object to be retrieved is text, the candidate matching object is an image; when the object to be retrieved is an image, the candidate matching object is text.

[0128] In practical applications, the object to be searched and the candidate matching object can be obtained first, the object to be searched is the object to be searched, and the candidate matching object is the object that is preset and can be used as the search result, and when the object to be searched is text, the candidate matching object is an image, and when the object to be searched is an image, the candidate matching object is text. In other words, according to the search method of the present invention, an image corresponding to a text can be output, or text corresponding to an image can be output, etc.

[0129] Step S203: Process the object to be retrieved by using the trained image text recognition model to obtain the features of the object to be retrieved.

[0130] Step S204: Process the candidate matching object through the image text recognition model to obtain the features of the candidate matching object.

[0131] In practical applications, after obtaining the object to be retrieved and the candidate matching objects, the object to be retrieved can be processed by the trained image text recognition model to obtain the features of the object to be retrieved, which can be specifically the corresponding classification embedding vector; the candidate matching objects can be processed by the image text recognition model to obtain the features of the candidate matching objects, which can be specifically the corresponding classification embedding features.

[0132] It should be noted that the training process of the image text recognition model is the same as the above embodiment, including: obtaining a target number of training samples, the training samples include image text pairs consisting of images and texts; obtaining the recognition results of the image text recognition model for the training samples; processing the recognition results according to the target loss function to generate the loss value of the image text recognition model; updating the image text recognition model according to the loss value; wherein the target loss function includes a similarity loss term, and the similarity loss term is used to constrain the similarity between the image features and the text features of the positive sample pair to approach 1, and constrain the similarity between the image features and the text features of the negative sample pair to approach 0. And the text and image types are consistent with the data types in the training data set of the joint training image and text model; that is, in the training data set of the joint training image and text model, the image and the corresponding text constitute an image text pair, and the language type of the text determines the input text language type that can be processed by the text encoding module. In other words, if the text in the training dataset of the joint training image and text model is in Chinese, the text should also be in Chinese; if the text in the training dataset is in English, the text should also be in English; if the text in the training dataset covers multiple languages, the text can contain multiple languages ​​to ensure that the language type of the text is consistent with the text in the training dataset to ensure the accuracy and reliability of image retrieval.

[0133] Step S205: Calculate the similarity value between the feature of the object to be retrieved and the feature of each candidate matching object.

[0134] Step S206: taking a similarity value greater than or equal to a preset threshold as a target value.

[0135] Step S207: Determine the candidate matching object corresponding to the target value as the search result of the object to be searched.

[0136] In practical applications, after obtaining the features of the object to be retrieved and the features of the candidate matching objects, the similarity value between the features of the object to be retrieved and the features of each candidate matching object can be calculated, and the similarity value greater than or equal to a preset threshold is used as the target value; the candidate matching object corresponding to the target value is determined as the retrieval result of the object to be retrieved, so as to determine the retrieval result of the object to be retrieved by the similarity between the features.

[0137] In a specific application scenario, in the process of calculating the similarity value between the feature of the object to be retrieved and the feature of each candidate matching object, the similarity value between the feature of the object to be retrieved and the feature of each candidate matching object can be calculated by a similarity generation formula; the similarity generation formula includes:

[0138] S=<v1,v2> ;

[0139] Wherein, S represents the similarity value between the feature of the object to be retrieved and the feature of the candidate matching object; v1 represents the feature of the object to be retrieved, v2 represents the feature of the candidate matching object, and the operation type represented by < > can refer to the above embodiment.

[0140] From this implementation process, it can be seen that in the process of calculating the similarity value between the features of the object to be retrieved and the features of the candidate matching objects, the present application processes the features of the object to be retrieved and the features of the candidate matching objects through the < > operation. Since the < > operation not only applies the vector modulus of the features of the object to be retrieved and the features of the candidate matching objects, but also applies the transposition of the features of the object to be retrieved, it can more accurately analyze the similarity value between the features of the object to be retrieved and the features of the candidate matching objects, thereby more accurately guiding the retrieval process and obtaining more accurate retrieval results.

[0141] For ease of understanding, the retrieval method of the present invention is now described with an image retrieval process to help users quickly find images that match their descriptions through concise text queries, providing a convenient image search and browsing experience. The image retrieval process implements the following functions: a text is used as a query, and one or more images that are consistent with the text content description are retrieved from the image library. For example, when the input query is "a cat", the query system will return images related to "cat", such as an image of an orange cat napping in the sun, an image of a black and white cat playing on the grass, etc. In addition, when the input query is "beautiful scenery", the query system can return images of magnificent mountains and blue skies, images of gorgeous sunsets, etc. Similarly, when the input query is "city night view", the query system can return images of flashing city lights and high-rise buildings, images of night scenes on busy streets, etc.

[0142] Assume that the image retrieval system is Figure 4 As shown in the figure, the image database Db is the image database to be queried, which includes a large number of unlabeled images whose contents are unknown, that is, the categories of objects in the images are unknown. The image database Db is large in scale and covers images from multiple fields and scenes. Each image in the image database is an independent instance and has not been classified or labeled in advance. For the convenience of description, the total number of images in the image database Db is defined as N. The image retrieval process is as follows:

[0143] Get the query text entered by the user;

[0144] The query text is input into the text encoding module trained by the present invention to obtain the feature embedding vector F T , ,Pick As features of the input query text;

[0145] Each image I in the image database Db to be queried k Input to the image encoding module trained using the present invention, , get the output feature embedding vector F for each image Ik , ;Pick As image I k features; then, the image feature database Db F Including N image features, the set is , image features correspond one to one with images;

[0146] Calculate the similarity between the features of the query text T and each feature in the image feature data

[0147] , ;

[0148] Obtain a similarity threshold Ts, the value of which can be flexibly determined as needed, such as 0.8;

[0149] According to the similarity threshold Ts, the image index set is obtained as , which means returning all s q The set of indices ≥Ts;

[0150] Index collection by image , find the corresponding image from the image database Db and return it.

[0151] The present invention also provides an electronic device and a computer-readable storage medium, both of which have the corresponding effects of a model training method or a retrieval method provided in an embodiment of the present invention. Figure 5 , Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0152] An electronic device provided by an embodiment of the present invention includes a memory 201 and a processor 202. The memory 201 stores a computer program. When the processor 202 executes the computer program, the steps of the model training method or the retrieval method described in any of the above embodiments are implemented.

[0153] See also Figure 6Another electronic device provided in an embodiment of the present invention may also include: an input port 203 connected to the processor 202, used to transmit commands input from the outside to the processor 202; a display unit 204 connected to the processor 202, used to display the processing results of the processor 202 to the outside; a communication module 205 connected to the processor 202, used to realize communication between the electronic device and the outside. The display unit 204 can be a display panel, a laser scanning display, etc.; the communication method adopted by the communication module 205 includes but is not limited to mobile high-definition link technology (Mobile High-Definition Link, MHL), Universal Serial Bus (Universal Serial Bus, USB), High-Definition Multimedia Interface (High-Definition Multimedia Interface, HDMI), wireless connection: wireless fidelity technology (WIreless Fidelity, WiFi), Bluetooth communication technology, low-power Bluetooth communication technology, and communication technology based on IEEE802.11s.

[0154] An embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the model training method or the retrieval method described in any of the above embodiments are implemented.

[0155] The computer-readable storage medium involved in the present invention includes random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM (Compact Disc Read-Only Memory), or any other form of storage medium known in the technical field.

[0156] A computer program product provided by an embodiment of the present invention includes a computer program / instruction, which, when executed by a processor, implements the steps of the model training method or retrieval method described in any of the above embodiments.

[0157] For the description of the relevant parts of an electronic device, a computer program product, and a computer-readable storage medium provided by an embodiment of the present invention, please refer to the detailed description of the corresponding parts in a model training method or a retrieval method provided by an embodiment of the present invention, which will not be repeated here. In addition, the parts of the above-mentioned technical solutions provided by the embodiments of the present invention that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.

[0158] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0159] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model training method, characterized in that: include: Get the image text recognition model to be trained; Obtaining a target number of training samples, wherein the training samples include image-text pairs consisting of an image and a text; Obtaining a recognition result of the image text recognition model on the training sample; Processing the recognition result according to the target loss function to generate a loss value of the image text recognition model; Update the image text recognition model according to the loss value; The target loss function includes a similarity loss term, and the similarity loss term is used to constrain the similarity between the image features and the text features of the positive sample pair to be close to 1, and to constrain the similarity between the image features and the text features of the negative sample pair to be close to 0; The recognition result is processed according to the target loss function to generate the loss value of the image text recognition model, including: Combining the target number of image classification embedding vectors into a column vector, wherein the image classification embedding vector includes a vector obtained by extracting the image in the training sample by the image encoding module in the image text recognition model; Combining the target number of text classification embedding vectors into a row vector, wherein the text classification embedding vector includes a vector obtained after the text encoding module in the image text recognition model extracts the text in the training sample; Forming a similarity matrix based on the column vector and the row vector; Determining a target dimension for elements in the similarity matrix; Generate a first matrix with rows and columns of the target dimension, the first matrix including a matrix with diagonal elements of 1 and other elements of 0; Generate a second matrix with rows and columns of the target dimension, the second matrix including a matrix with diagonal elements of 0 and other elements of 1; Processing the similarity matrix based on the first matrix and the second matrix to generate a value of the similarity loss term; The value of the similarity loss term and the recognition result are processed according to the target loss function to generate a loss value of the image text recognition model.

2. The model training method according to claim 1, characterized in that: The image text recognition model includes an image encoding module and a text encoding module; Obtaining the recognition result of the image text recognition model on the training sample includes: Obtaining an image embedding vector obtained by extracting the image in the training sample by the image encoding module, wherein the image embedding vector includes an image classification embedding vector; Obtaining a text embedding vector obtained by extracting the text in the training sample by the text encoding module, wherein the text embedding vector includes a text classification embedding vector; The image embedding vector and the text embedding vector are used as the recognition result.

3. The model training method according to claim 2, characterized in that: Processing the similarity matrix based on the first matrix and the second matrix to generate a value of the similarity loss term includes: Applying a constant coefficient and an F-norm, performing a matrix position-wise multiplication process on the similarity matrix based on the first matrix and the second matrix, and generating a value of the similarity loss term.

4. The model training method according to claim 3, characterized in that: The objective loss function includes a constraint term of the similarity loss term, the constraint term changes with the number of training times, and gradually maximizes the similarity of positive sample pairs and minimizes the similarity of negative sample pairs.

5. The model training method according to claim 4, characterized in that: Processing the value of the similarity loss term and the recognition result according to the target loss function to generate a loss value of the image text recognition model includes: Determining a cycle number for training the image text recognition model; Determine the constraint item which takes the periodic ordinal number as an independent variable and increases monotonically; Processing the current cycle ordinal number based on the constraint item to generate a value of the constraint item; The value of the constraint item, the value of the similarity loss item and the recognition result are processed according to the target loss function to generate a loss value of the image text recognition model.

6. The model training method according to claim 5, characterized in that: Processing the current cycle ordinal number based on the constraint item to generate the value of the constraint item includes: An activation function is performed on the current cycle ordinal number based on the constraint item to generate a value of the constraint item.

7. The model training method according to claim 6, characterized in that: The value of the constraint item, the value of the similarity loss item and the recognition result are processed according to the target loss function to generate the loss value of the image text recognition model, including: Generate an image-text contrast loss item from each image to all texts in the training sample according to the text classification embedding vector and the image classification embedding vector; Generate a text-image contrast loss term from each text in the training sample to all images according to the text classification embedding vector and the image classification embedding vector; Generate a target contrast loss term according to the image-text contrast loss term and the text-image contrast loss term; The target contrast loss term, the value of the constraint term and the value of the similarity loss term are processed according to the target loss function to generate a loss value of the image text recognition model.

8. The model training method according to claim 7, characterized in that: According to the text classification embedding vector and the image classification embedding vector, an image-text contrast loss item from each image to all texts in the training sample is generated, including: Applying a logarithmic function, an exponential function, a vector modulus and a temperature coefficient of contrast loss, an image-text contrast loss term from each image to all texts in the training sample is generated according to the text classification embedding vector and the image classification embedding vector.

9. The model training method according to claim 8, characterized in that: Generating a text-image contrast loss term from each text in the training sample to all images according to the text classification embedding vector and the image classification embedding vector, including: A logarithmic function, an exponential function, a vector modulus and a temperature coefficient of contrast loss are applied to generate a text-image contrast loss term from each text in the training sample to all images according to the text classification embedding vector and the image classification embedding vector.

10. The model training method according to claim 9, characterized in that: Generating a target contrast loss term according to the image-text contrast loss term and the text-image contrast loss term, including: A constant coefficient is applied to perform weighted averaging on the image-text contrast loss term and the text-image contrast loss term to generate a target contrast loss term.

11. The model training method according to claim 10, characterized in that: The target contrast loss term, the value of the constraint term, and the value of the similarity loss term are processed according to the target loss function to generate a loss value of the image text recognition model, including: The target contrast loss term, the value of the constraint term and the value of the similarity loss term are multiplied according to the target loss function to generate a loss value of the image text recognition model.

12. A search method, characterized in that: include: Acquire an object to be retrieved, wherein the object to be retrieved includes text or an image; Acquire a candidate matching object; when the object to be retrieved is text, the candidate matching object is an image; when the object to be retrieved is an image, the candidate matching object is text; Processing the object to be retrieved by using a trained image text recognition model to obtain features of the object to be retrieved; Processing the candidate matching object by using the image text recognition model to obtain features of the candidate matching object; Calculating the similarity value between the feature of the object to be retrieved and the feature of each candidate matching object; Taking the similarity value greater than or equal to a preset threshold as a target value; Determine the candidate matching object corresponding to the target value as the search result of the object to be searched; The training process of the image text recognition model includes: obtaining a target number of training samples, wherein the training samples include image-text pairs consisting of images and texts; obtaining recognition results of the image text recognition model on the training samples; processing the recognition results according to a target loss function to generate a loss value of the image text recognition model; updating the image text recognition model according to the loss value; wherein the target loss function includes a similarity loss term, and the similarity loss term is used to constrain the similarity between the image features and the text features of the positive sample pair to approach 1, and to constrain the similarity between the image features and the text features of the negative sample pair to approach 0; Among them, the recognition result is processed according to the target loss function to generate the loss value of the image text recognition model, including: the target number of the image classification embedding vectors are formed into a column vector, the image classification embedding vector includes the vector obtained after the image encoding module in the image text recognition model extracts the image in the training sample; the target number of the text classification embedding vectors are formed into a row vector, the text classification embedding vector includes the vector obtained after the text encoding module in the image text recognition model extracts the text in the training sample; based on the column vector and the row vector, a similarity matrix is ​​formed; the target dimension of the elements in the similarity matrix is ​​determined; a first matrix with rows and columns being the target dimension is generated, the first matrix includes a matrix with diagonal 1 and other elements being 0; a second matrix with rows and columns being the target dimension is generated, the second matrix includes a matrix with diagonal 0 and other elements being 1; the similarity matrix is ​​processed based on the first matrix and the second matrix to generate the value of the similarity loss term; the value of the similarity loss term and the recognition result are processed according to the target loss function to generate the loss value of the image text recognition model.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

14. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method according to any one of claims 1 to 12 when executing the computer program.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images