Image-text classification network training method and device, image-text classification method and device and storage medium

Abstract

The invention discloses an image-text classification network training method, an image-text classification method, equipment and a storage medium. The image-text classification network training method comprises the steps of determining an image-text classification network; determining image-text samples of each batch; the image-text sample comprises image data and text information; in the same batch of image-text samples, labeling the image data according to the text information; in the same batch of image-text samples, image data are input into an image encoder to be encoded into image features, text information is input into a text encoder to be encoded into text features, and the image features and the text features are input into a fusion module to calculate the probability of mutual matching between the image data and the text information; calculating a total loss value of mutual matching between the image data and the text information according to the difference between the probability and the label calculation; and updating the image-text classification network according to the total loss value. According to the embodiment of the invention, the image-text classification network can be finely adjusted, and the generalization ability of the image-text classification network is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning, and in particular to a training of a graph-text classification network and a graph-text classification method, device and storage medium thereof. Background Art

[0002] In business scenarios such as video surveillance in rail transit, one or more video data are collected, and a graphic classification network is trained according to the needs of the business scenario. The classification network is used to classify the video data, and image frames that meet the needs are automatically located from the video data.

[0003] Since the labels (i.e. categories) used when training the classification network are fixed, the ability of the classification network is to output fixed categories.

[0004] However, the needs of business scenarios are constantly changing, which exceeds the scope of the labels used in the previous training classification network. If we organize and develop vocabulary based on the new needs of the business scenarios and independently train a new image and text classification network based on the open vocabulary, there will be some duplication of functions between different image and text classification networks, which will significantly increase development and management costs. Summary of the invention

[0005] In view of this, the present invention provides a training of a graph-text classification network and a graph-text classification method, device and storage medium thereof, so as to classify the graph-text classification network by taking into account both fixed labels and open vocabulary.

[0006] A first aspect of the present invention provides a training method for an image-text classification network, comprising:

[0007] Determine a picture and text classification network; the picture and text classification network includes an image encoder, a text encoder and a fusion module;

[0008] Determine each batch of image and text samples; the image and text samples include image data and text information; part of the text information is description information, and part of the text information is category information;

[0009] Among the image and text samples in the same batch, labeling the image data according to the text information;

[0010] In the image and text samples of the same batch, the image data is input into the image encoder to be encoded into image features, the text information is input into the text encoder to be encoded into text features, and the image features and the text features are input into the fusion module to calculate the probability of mutual matching between the image data and the text information;

[0011] Calculating a total loss value of mutual matching between the image data and the text information according to a difference between the probability and the label calculation;

[0012] The image-text classification network is updated according to the total loss value.

[0013] A second aspect of the present invention provides a method for classifying images and texts, comprising:

[0014] Loading a picture-text classification network trained according to the method of the first aspect of claim; the picture-text classification network comprises an image encoder, a text encoder and a fusion module;

[0015] Receiving image data and text information input by a client; the text information is description information or category information;

[0016] Input the image data into the image encoder and encode it into image features, input the text information into the text encoder and encode it into text features, input the image features and the text features into the fusion module to calculate the probability of matching the image data to the text information;

[0017] The image data matching the text information is determined according to the probability.

[0018] A third aspect of the present invention provides a training device for a picture-text classification network, comprising:

[0019] A picture and text classification network determination module, used to determine a picture and text classification network; the picture and text classification network includes an image encoder, a text encoder and a fusion module;

[0020] An image and text sample determination module is used to determine image and text samples of each batch; the image and text samples include image data and text information; part of the text information is description information, and part of the text information is category information;

[0021] A labeling module, used for labeling the image data according to the text information in the same batch of the image and text samples;

[0022] A probability calculation module, for inputting the image data into the image encoder to be encoded into image features, inputting the text information into the text encoder to be encoded into text features, and inputting the image features and the text features into the fusion module to calculate the probability of matching between the image data and the text information;

[0023] A total loss value calculation module, used to calculate the total loss value of the mutual matching between the image data and the text information according to the difference between the probability and the label calculation;

[0024] The image and text classification network updating module is used to update the image and text classification network according to the total loss value.

[0025] A fourth aspect of the present invention provides a graphic and text classification device, comprising:

[0026] An image-text classification network loading module, used to load the image-text classification network trained according to the method of embodiment 1; the image-text classification network includes an image encoder, a text encoder and a fusion module;

[0027] An input receiving module, used to receive image data and text information input by a client; the text information is description information or category information;

[0028] A probability calculation module, used for inputting the image data into the image encoder to be encoded into image features, inputting the text information into the text encoder to be encoded into text features, and inputting the image features and the text features into the fusion module to calculate the probability of matching the image data to the text information;

[0029] A matching determination module is used to determine the image data that matches the text information based on the probability.

[0030] A fifth aspect of the present invention provides an electronic device, the electronic device comprising:

[0031] at least one processor; and

[0032] a memory communicatively connected to the at least one processor; wherein,

[0033] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the training of the image and text classification network as described in the first aspect or the image and text classification method as described in the second aspect.

[0034] The sixth aspect of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the training of the image and text classification network as described in the first aspect or the image and text classification method as described in the second aspect.

[0035] The seventh aspect of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the training of the image and text classification network as described in the first aspect or the image and text classification method as described in the second aspect.

[0036] In this embodiment, a graph-text classification network is determined; the graph-text classification network includes an image encoder, a text encoder and a fusion module; each batch of graph-text samples is determined; the graph-text samples include image data and text information; part of the text information is description information, and part of the text information is category information; in the same batch of graph-text samples, the image data is labeled according to the text information; in the same batch of graph-text samples, the image data is input into the image encoder to be encoded as image features, the text information is input into the text encoder to be encoded as text features, the image features and the text features are input into the fusion module to calculate the probability of mutual matching between the image data and the text information; the total loss value of mutual matching between the image data and the text information is calculated according to the difference between the probability and the label calculation; the graph-text classification network is updated according to the total loss value. This embodiment can fine-tune the graph-text classification network, and while retaining fixed label information, it adds open description information, enhances the generalization ability of the graph-text classification network, and performs well in different tasks. It not only maintains advantages in tasks such as image retrieval, but also shows excellent performance in classification tasks, and can quickly adapt to new demand changes in business scenarios.

[0037] In addition, maintaining an image and text classification network and avoiding training a new image and text classification network can help reduce development and management costs.

[0038] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 It is a flowchart of a training method for a graph-text classification network provided in Example 1 of the present invention.

[0041] Figure 2 It is a structural diagram of a picture and text classification network provided in Example 1 of the present invention.

[0042] Figure 3 This is a flow chart of a method for image and text classification provided in Embodiment 2 of the present invention.

[0043] Figure 4 It is a structural diagram of a training device for a picture and text classification network provided in Embodiment 3 of the present invention.

[0044] Figure 5 It is a structural schematic diagram of a picture and text classification device provided in Embodiment 4 of the present invention.

[0045] Figure 6 It is a structural schematic diagram of an electronic device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0047] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can cover sequential implementations other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0048] Embodiment 1

[0049] See also Figure 1 , shows a flow chart of a training method for a graph-text classification network provided by the first embodiment of the present invention. The method can be executed by a training device for a graph-text classification network. The training of the graph-text classification network and the graph-text classification device can be implemented in the form of hardware and / or software. The training device for the graph-text classification network can be configured in an electronic device. Figure 1 As shown, the method includes:

[0050] Step 101: Determine the image and text classification network.

[0051] In this embodiment, if Figure 2 As shown in the figure, the image and text classification network includes an image encoder ImageEncoder, a text encoder TextEncoder and a fusion module FM (Fusion Module).

[0052] Among them, the ImageEncoder is responsible for encoding the image data to extract features from the image data, the TextEncoder is responsible for encoding the text information to extract features from the text information, and the Fusion Module (FM) is responsible for performing a classification task on the features of the image data and the features of the text information to screen out the image data containing text information.

[0053] In practical applications, the structure of the image-text classification network (including the ImageEncoder, TextEncoder, and Fusion Module (FM)) is not limited to the neural network designed manually, and can also be a neural network optimized by the model quantization method, a neural network searched for image-text characteristics through the NAS (Neural Architecture Search) method, and so on. This embodiment does not limit this.

[0054] Step 102: Determine the image-text samples of each batch.

[0055] In this embodiment, the image-text classification network can be trained by using the image-text samples through multiple rounds of iteration. Among them, the image-text samples include image data and text information, and the image data contains content matching the text information.

[0056] For example, if the text information is "cat", the content in the image data includes "cat"; if the text information is "the cat is sleeping", the content in the image data includes "the cat is sleeping".

[0057] Furthermore, some of the text information is descriptive information, and the descriptive information is open vocabulary, and some of the text information is category information, and the category information is fixed labels.

[0058] During the process of iteratively training the image-text classification network in a certain iteration, m frames of image data marked with descriptive information and n frames of image data marked with category information are randomly sampled respectively to construct a batch of image-text samples.

[0059] Furthermore, 0 < m < batch_size, 0 < n < batch_size, and m + n = batch_size, where batch_size is the size of the batch, batch_size is any positive integer greater than 0, and is generally an exponent of 2, such as 32, 64, 128, etc.

[0060] Then, in each batch of image-text samples, there are both image data marked with descriptive information and image data marked with category information.

[0061] When reading image and text samples, the image data is directly read. When reading the text information corresponding to the image data, the text information is converted into a vector Token using methods such as the Tokenizer of the text encoder.

[0062] If the image and text sample is image data and description information, the description information corresponding to the image data can be converted into a vector Token using methods such as the Tokenizer of the text encoder, denoted as T o , exemplarily, its shape is [1,77].

[0063] If the image and text sample is image data and label information, there are k label information in the label information set C, that is, C = {cls i |i=1,2,…,k}, we can use the Tokenizer of the text encoder to convert the label information corresponding to the image data into a vector Token, denoted by T cls , exemplarily, its shape is [k,77].

[0064] Step 103: In the same batch of image and text samples, label the image data according to the text information.

[0065] In the same batch of image and text samples, the image data can be labeled with Label according to factors such as the type of text information (i.e., description information, category information, etc.) and sorting.

[0066] In one embodiment of the present invention, step 103 may include the following steps:

[0067] Step 1031: If the text information is description information, the image data is annotated with a specified index value as a label.

[0068] For the same image and text sample, if the text information is description information, the image data can be annotated with a specified index value (such as -1) as a label.

[0069] The index value corresponding to the description information is different from the index value corresponding to each category information.

[0070] Step 1032: If the text information is category information, annotate the image data with an index value corresponding to the category information as a label.

[0071] For the same image and text sample, if the text information is category information, the image data can be annotated with the original k index values ​​(C.index) corresponding to the category information as the label Label.

[0072] At this time, for a batch of image samples of size batch_size, after preprocessing, the batch of image samples contains several random frames of image data, denoted as L img , and the vector Token of text information corresponding to the image data (including T o and T cls ), denoted as L txt , and the label corresponding to the image data, denoted as L label .

[0073] Step 1033: sort the image and text samples in the same batch.

[0074] In this embodiment, the image and text samples of the same batch are sorted, wherein the image and text samples whose text information is descriptive information are sorted before the image and text samples whose text information is category information, and vice versa, the image and text samples whose text information is category information are sorted after the image and text samples whose text information is descriptive information, so as to facilitate the subsequent calculation of the total loss value and improve the efficiency of training the image and text classification network.

[0075] In the specific implementation, the label L label The length is Len label .

[0076] In the same batch of image and text samples, all labels L label The label L in the specified index value (such as -1) label Write the first index list L label-o In the example, let the first index list L label-o The length is Len label-o .

[0077] In the same batch of image and text samples, all labels are written into the second index list L tmp In the second index list L tmp The value of is {i│i=0,1,2,…,Len label -1}.

[0078] Create the third index list L index。

[0079] The first index list L label-o The label is written into the third index list L index That is, the first index list L label-0 The value of is assigned from the head to the third index list L index , and the second index list L tmp Except the first index list L label-o The other tags are written into the third index list L index In the L index middle.

[0080] Then, this process can be expressed as: index =L label-o ∪(L tmp \L label-o ).

[0081] According to the third index list L index The order of labels in the same batch of image and text samples (including image data L img , vector L txt With label L label ) to sort.

[0082] Step 1034: If the sorting is completed, the index value of the tag is adjusted according to the sorting order.

[0083] If the image and text samples are sorted, the index value of each label in the image and text samples can be adjusted according to the sorting order of the image and text samples to ensure the uniqueness of the index value of each label.

[0084] In a specific implementation, the first index list L may be counted label-o LengthLen label-o , the index values ​​of the tags (value -1) sorted within the length are increased in sequence. For example, the index values ​​of the tags (value -1) sorted within the length are 0, 1, 2, ..., Len. label-o -1.

[0085] The index values ​​of the tags sorted after the length (values ​​not -1) are accumulated by the length Len. label-o , that is, C.index=C.index+Len label-o .

[0086] In this embodiment, through a reasonable hybrid data emission strategy, the image and text samples of both open descriptive information and fixed label category information can be effectively combined, which not only fully utilizes the advantages of the two types of data, but also improves the generalization ability of the image and text classification network, providing a new solution for complex cross-modal tasks.

[0087] Step 104: In the same batch of image and text samples, the image data is input into the image encoder to be encoded into image features, the text information is input into the text encoder to be encoded into text features, and the image features and text features are input into the fusion module to calculate the probability of mutual matching between the image data and the text information.

[0088] like Figure 2 As shown, for the same batch of image and text samples, the image data L img Input the image encoder ImageEncoder for encoding to obtain the image feature F img .

[0089] For example, the image feature F img is a matrix of shape [batch_size,768].

[0090] The text information L txt Input the text encoder TextEncoder for encoding to obtain the text feature F txt .

[0091] For example, the text feature F txt For the shape of [Len label-o +k,768] matrix.

[0092] The image feature F img With the text feature F txt The input fusion module FM interacts and calculates the image data L img With text information L txt The probability of matching each other.

[0093] In a specific implementation, the fusion module FM comprises a scaling factor.

[0094] On the one hand, the product of the scaling factor, the image feature and the transposed matrix of the text feature is calculated to obtain a target matrix; the target matrix records the probability of matching the image data to the text information.

[0095] The target matrix can be expressed as: Among them, logit i2t is the target matrix, θ is the scaling factor, and T represents the transpose of the matrix.

[0096] On the other hand, the product of the scaling factor, the transposed matrix of the text feature and the image feature is calculated to obtain a reference matrix; the reference matrix records the probability of matching from the text information to the image data.

[0097] The reference matrix can be expressed as: Among them, logit t2i is the reference matrix, θ is the scaling factor, and T represents the transpose of the matrix.

[0098] Step 105: Calculate the total loss value of the mutual matching between the image data and the text information based on the difference between the probability and the label calculation.

[0099] In this embodiment, the probability of the mutual matching between the image data and the text information can be compared with the pre-labeled labels, and the difference between the two can be calculated from one or more dimensions as the total loss value of the mutual matching between the image data and the text information.

[0100] In one embodiment of the present invention, step 105 may include the following steps:

[0101] Step 1051: Calculate the first partial loss value for matching from image data to text information based on the difference between the target matrix and the label.

[0102] In this embodiment, the total loss value of the mutual matching between the image data and the text information includes two parts, one part is the first part loss value from matching the image data to the text information, and the other part is the second part loss value from matching the text information to the image data.

[0103] For the first division loss value from matching image data to text information, the target matrix logit can be considered i2t With label L label If it is a multi-classification task between two images, then the first division loss value from matching image data to text information can be calculated using methods such as cross entropy.

[0104] Exemplarily, the first partial loss value is expressed as:

[0105]

[0106] Among them, loss i2t is the loss value of the first division, Z j,i is the target matrix logit i2t The probability of the jth row and ith column in y i Indicates the i-th label L label , batch_size is the batch size, Len label-o +k is the target matrix logit i2t The dimension (length) of .

[0107] Step 1052: Calculate a second partial loss value for matching the image data to the text information based on the difference between the reference matrix and the label.

[0108] For the second division loss value from matching text information to image data, the reference matrix logit can be considered t2i With label L label It is a multi-classification and multi-label fusion task. Then, the cross entropy and other methods can be used to calculate the second division loss value from matching text information to image data.

[0109] In a specific implementation, the second division loss value of matching the text information to the image data is further divided into two parts according to different types of text information.

[0110] Accordingly, depending on the type of text information, the reference matrix logit t2iDivided into the first interval matrix logit corresponding to the descriptive information t2i_1 The second interval matrix logit corresponding to the label information t2i_2 .

[0111] Part of it uses cross entropy and other methods based on the first interval matrix logit t2i_1 The label L corresponding to the description information label The difference between them calculates the first sub-loss value matching from image data to text information.

[0112] Exemplarily, the first sub-loss value is expressed as:

[0113]

[0114] Among them, loss t2i_1 is the first sub-loss value, Z j,i is the first interval matrix logit t2i_1 The probability of the jth row and ith column in y i is the label corresponding to the i-th description information, Len label-o is the number (length) of tags belonging to the description information.

[0115] In the other part, the cross entropy and other methods are used according to the second interval matrix logit t2i_2 The label L corresponding to the category information label The difference between them calculates the second sub-loss value matching from image data to text information.

[0116] Exemplarily, the second sub-loss value is expressed as:

[0117]

[0118] t = batch_size - Len label-o

[0119] Among them, loss t2i_2 is the second sub-loss value, logit t2i_2 is the second interval matrix, σ is the activation function, batch_size is the batch size, Len label-o +k is the dimension (length) of the target matrix, Y T [:,i] represents the transposed matrix of the one-hot encoding of the label with index value i.

[0120] Among them, the one-hot encoding can be expressed as:

[0121]

[0122] Among them, i represents the index value, j represents the index of the category information, and Y∈R [t，k] .

[0123] The first sub-loss value and the second sub-loss value are fused into a second partial loss value that matches the image data to the text information in a linear or nonlinear manner.

[0124] Step 1053: merge the first partial loss value and the second partial loss value into a total loss value that matches the image data and the text information.

[0125] In this embodiment, the first partial loss value and the second partial loss value (including the first sub-loss value and the second sub-loss value) can be fused into a total loss value that matches the image data and the text information in a linear or nonlinear manner.

[0126] Exemplarily, the total loss value is expressed as:

[0127] loss=loss i2t +loss t2i_1 +loss t2i_2

[0128] Among them, loss is the total loss value, loss i2t is the loss value of the first division, loss t2i_1 is the first sub-loss value, loss t2i_2 is the second sub-loss value.

[0129] In this embodiment, on the one hand, the first partial loss value from matching image data to text information is calculated, and on the other hand, the second partial loss value from matching image data to text information is calculated based on the type of text information (i.e., description information, label information). The two are merged into the total loss value of the mutual matching between image data and text information, so that the image and text classification network can learn the processing capabilities of two different text information, achieving more comprehensive and efficient learning.

[0130] Step 106: Update the image and text classification network according to the total loss value.

[0131] In this embodiment, the total loss value can be substituted into optimization algorithms such as SGD (stochastic gradient descent) and Adam (Adaptive momentum) to calculate the update amplitude of the parameters in the image and text classification network, and the parameters in the image and text classification network are updated according to the update amplitude.

[0132] In addition, training conditions can be set in advance as conditions for stopping training the image and text classification network, for example, the number of iterations reaches a certain threshold, the total loss value is less than a certain threshold, the change in the total loss value in multiple iterative training is less than a certain threshold, and so on. In each round of iterative training, it is judged whether the corresponding parameters (such as the number of iterations, total loss value, etc.) meet the training conditions.

[0133] If the training conditions are met, the image-text classification network can be considered to have completed training.

[0134] If the training conditions are not met, the next round of iterative training can be entered, and steps 104 to 106 are executed again, and the iterative training is repeated in this way until the image-text classification network is trained.

[0135] In this embodiment, a graph-text classification network is determined; the graph-text classification network includes an image encoder, a text encoder and a fusion module; each batch of graph-text samples is determined; the graph-text samples include image data and text information; part of the text information is description information, and part of the text information is category information; in the same batch of graph-text samples, the image data is labeled according to the text information; in the same batch of graph-text samples, the image data is input into the image encoder to be encoded as image features, the text information is input into the text encoder to be encoded as text features, the image features and the text features are input into the fusion module to calculate the probability of mutual matching between the image data and the text information; the total loss value of mutual matching between the image data and the text information is calculated according to the difference between the probability and the label calculation; the graph-text classification network is updated according to the total loss value. This embodiment can fine-tune the graph-text classification network, and while retaining fixed label information, it adds open description information, enhances the generalization ability of the graph-text classification network, and performs well in different tasks. It not only maintains advantages in tasks such as image retrieval, but also shows excellent performance in classification tasks, and can quickly adapt to new demand changes in business scenarios.

[0136] In addition, maintaining an image and text classification network and avoiding training a new image and text classification network can help reduce development and management costs.

[0137] Embodiment 2

[0138] See also Figure 3 , shows a flowchart of a method for image and text classification provided by the first embodiment of the present invention, the method can be executed by an image and text classification device, the training of the image and text classification network and the image and text classification device can be implemented in the form of hardware and / or software, and the image and text classification device can be configured in an electronic device. Figure 3 As shown, the method includes:

[0139] Step 301: Load the image and text classification network.

[0140] In this embodiment, a picture-text classification network trained according to the method of the first embodiment may be loaded, wherein the picture-text classification network includes an image encoder ImageEncoder, a text encoder TextEncoder and a fusion module FM.

[0141] Step 302: Receive image data and text information input by the client.

[0142] In this embodiment, the client may provide controls for inputting image data and text information on a UI (User Interface), and the user may operate the relevant controls on the UI of the client to determine the image data and text information.

[0143] In different business scenarios, the form of image data is different. For example, image data is movies, surveillance videos, illustrations, and so on.

[0144] Generally, image data is stored on a server. When the server provides image and text classification services, the range of image data is set by default. Users can modify the range of image data by operating on the client UI.

[0145] In addition, the text information is descriptive information or category information. For descriptive information, a text control can be provided on the client UI, allowing the user to enter the descriptive information in the text control. For category information, a selection control (such as a drop-down menu, etc.) can be provided on the client UI, allowing the user to select category information in the selection control.

[0146] Step 303: input the image data into the image encoder to be encoded into image features, input the text information into the text encoder to be encoded into text features, and input the image features and text features into a fusion module to calculate the probability of matching the image data to the text information.

[0147] In this embodiment, the image data L img Input the image encoder ImageEncoder for encoding to obtain the image feature F img .

[0148] For example, the image feature F img is a matrix of shape [batch_size,768].

[0149] The text information L txt Input the text encoder TextEncoder for encoding to obtain the text feature F txt .

[0150] For example, the text feature F txt For the shape of [Len label-0 +k,768] matrix.

[0151] The image feature F img With the text feature F txt The input fusion module Head interacts and calculates the image data L img Matched to text information L txt probability.

[0152] In a specific implementation, the fusion module Head includes a scaling factor, then the product of the scaling factor, the transposed matrix of the image feature and the text feature can be calculated to obtain a target matrix; the target matrix records the probability of matching the image data to the text information.

[0153] The target matrix can be expressed as: Among them, logit i2t is the target matrix, θ is the scaling factor, and t represents the transpose of the matrix.

[0154] Step 304: Determine the image data that matches the text information based on the probability.

[0155] In this embodiment, a threshold method, a sorting method, or the like may be used to determine the image data that matches the text information based on probability.

[0156] For example, an image data with the highest probability and greater than a certain threshold is selected as the image data matching the text information and provided to the user for browsing.

[0157] For another example, N (N is a positive integer) image data with the highest probability are selected as image data matching the text information and provided to the user for browsing.

[0158] In this embodiment, a graphic and text classification network is loaded; the graphic and text classification network includes an image encoder, a text encoder and a fusion module; image data and text information input by a client are received; the text information is descriptive information or category information; the image data is input into the image encoder and encoded into image features, the text information is input into the text encoder and encoded into text features, the image features and text features are input into the fusion module to calculate the probability of matching the image data to the text information; the image data matching the text information is determined based on the probability. The graphic and text classification network of this embodiment has strong generalization ability and performs well in different tasks. It not only maintains advantages in tasks such as image retrieval, but also shows excellent performance in classification tasks, and can quickly adapt to new demand changes in business scenarios.

[0159] In addition, maintaining an image and text classification network and avoiding training a new image and text classification network can help reduce development and management costs.

[0160] Embodiment 3

[0161] See also Figure 4, shows a schematic diagram of the structure of a training device for a picture and text classification network provided by Embodiment 3 of the present invention. Figure 4 As shown, the device comprises:

[0162] An image-text classification network determination module 401 is used to determine an image-text classification network; the image-text classification network includes an image encoder, a text encoder and a fusion module;

[0163] The image and text sample determination module 402 is used to determine the image and text samples of each batch; the image and text samples include image data and text information; part of the text information is description information, and part of the text information is category information;

[0164] The labeling module 403 is used to label the image data according to the text information in the same batch of the image and text samples;

[0165] A probability calculation module 404 is used to input the image data into the image encoder to be encoded into image features, input the text information into the text encoder to be encoded into text features, and input the image features and the text features into the fusion module to calculate the probability of matching between the image data and the text information;

[0166] A total loss value calculation module 405, used to calculate the total loss value of the mutual matching between the image data and the text information according to the difference between the probability and the label calculation;

[0167] The image and text classification network updating module 406 is used to update the image and text classification network according to the total loss value.

[0168] In one embodiment of the present invention, the label marking module 403 includes:

[0169] A description and annotation module, configured to annotate the image data with a specified index value as a label if the text information is description information;

[0170] A category labeling module, configured to label the image data with an index value corresponding to the category information as a label if the text information is category information;

[0171] An image and text sample sorting module is used to sort the image and text samples of the same batch; wherein the image and text samples whose text information is descriptive information are sorted before the image and text samples whose text information is category information;

[0172] The index value adjustment module is used to adjust the index value of the tag according to the order of sorting if the sorting is completed.

[0173] In one embodiment of the present invention, the image and text sample sorting module includes:

[0174] A first index list processing module, used for writing the labels belonging to the specified index value in the same batch of the image and text samples into a first index list;

[0175] A second index list processing module, used for writing all the labels in the same batch of image and text samples into a second index list;

[0176] A third index list creation module, used to create a third index list;

[0177] A third index list processing module, configured to write the tags in the first index list into the third index list, and to write the tags in the second index list other than the tags in the first index list into the third index list;

[0178] An index sorting module, used to sort the image and text samples of the same batch according to the order of the labels in the third index list;

[0179] The index value adjustment module comprises:

[0180] A length statistics module, used to count the length of the first index list;

[0181] An index value incrementing module, used for sequentially incrementing the index values ​​of the tags sorted within the length;

[0182] The index value accumulation module is used to accumulate the length to the index values ​​of the tags sorted after the length.

[0183] In one embodiment of the present invention, the fusion module includes a scaling factor; the probability calculation module 404 includes:

[0184] A target matrix calculation module, used for calculating the product of the scaling factor, the image feature and the transposed matrix of the text feature to obtain a target matrix; the target matrix records the probability of matching the image data to the text information;

[0185] The reference matrix calculation module is used to calculate the product of the scaling factor, the text feature and the transposed matrix of the image feature to obtain a reference matrix; the reference matrix records the probability of matching the text information to the image data.

[0186] In one embodiment of the present invention, the total loss value calculation module 405 includes:

[0187] A first partial loss value calculation module, used for calculating a first partial loss value matching from the image data to the text information according to a difference between the target matrix and the label;

[0188] A second partial loss value calculation module, used for calculating a second partial loss value matching from the image data to the text information according to a difference between the reference matrix and the label;

[0189] A total loss value fusion module is used to fuse the first partial loss value and the second partial loss value into a total loss value that matches the image data and the text information.

[0190] In one embodiment of the present invention, the second division loss value calculation module includes:

[0191] A matrix division module, used for dividing the reference matrix into a first interval matrix corresponding to the description information and a second interval matrix corresponding to the label information;

[0192] A first sub-loss value calculation module, used for calculating a first sub-loss value for matching from the image data to the text information according to a difference between the first interval matrix and the label corresponding to the description information;

[0193] A second sub-loss value calculation module, used for calculating a second sub-loss value for matching from the image data to the text information according to a difference between the second interval matrix and the label corresponding to the category information;

[0194] A partial loss value fusion module is used to fuse the first sub-loss value and the second sub-loss value into a second partial loss value that matches the image data to the text information.

[0195] In one embodiment of the present invention, the first partial loss value is expressed as:

[0196]

[0197] Among them, loss i2t is the loss value of the first division, Z j,i is the probability of the jth row and ith column in the target matrix, y i Represents the i-th label, batch_size is the batch size, Len label-o + k is the dimension of the target matrix;

[0198] The first sub-loss value is expressed as:

[0199]

[0200] Among them, losst2i_1 is the first sub-loss value, Z j,i is the probability of the jth row and ith column in the first interval matrix, y i is the label corresponding to the i-th description information, Len label-o is the number of the tags belonging to the description information;

[0201] The second sub-loss value is expressed as:

[0202]

[0203] t = batch_size - Len label-o

[0204] Among them, loss t2i_2 is the second sub-loss value, logit t2i_2 is the second interval matrix, σ is the activation function, batch_size is the batch size, Len label-o +k is the dimension of the target matrix, Y T [:,i] represents the transposed matrix of the one-hot encoding of the label with index value i;

[0205] The total loss value is expressed as:

[0206] loss=loss i2t +loss t2i_1 +loss t2i_2

[0207] Among them, loss is the total loss value, loss i2t is the loss value of the first division, loss t2i_1 is the first sub-loss value, loss t2i_2 is the second sub-loss value.

[0208] The training device for the image-text classification network provided in the embodiment of the present invention can execute the training method for the image-text classification network provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the training method for the image-text classification network.

[0209] Embodiment 4

[0210] See also Figure 5 , shows a schematic diagram of the structure of a graphic and text classification device provided by Embodiment 4 of the present invention. Figure 5 As shown, the device comprises:

[0211] An image-text classification network loading module 501 is used to load an image-text classification network trained according to the method of the first embodiment; the image-text classification network includes an image encoder, a text encoder and a fusion module;

[0212] The input receiving module 502 is used to receive image data and text information input by the client; the text information is description information or category information;

[0213] A probability calculation module 503 is used to input the image data into the image encoder to be encoded into image features, input the text information into the text encoder to be encoded into text features, and input the image features and the text features into the fusion module to calculate the probability of matching the image data to the text information;

[0214] The matching determination module 504 is used to determine the image data matching the text information according to the probability.

[0215] In one embodiment of the present invention, the fusion module includes a scaling factor; the probability calculation module 503 includes:

[0216] The target matrix calculation module is used to calculate the product of the scaling factor, the image feature and the transposed matrix of the text feature to obtain a target matrix; the target matrix records the probability of matching the image data to the text information.

[0217] The image-text classification device provided in the embodiment of the present invention can execute the image-text classification method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the image-text classification method.

[0218] Embodiment 5

[0219] See also Figure 6 , shows a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0220] like Figure 6As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0221] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0222] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 performs the various methods and processes described above, such as the training of the image and text classification network and its image and text classification method.

[0223] In some embodiments, the training of the image and text classification network and the image and text classification method thereof can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the training of the image and text classification network and the image and text classification method thereof described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the training of the image and text classification network and the image and text classification method thereof by any other appropriate means (for example, by means of firmware).

[0224] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0225] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0226] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0227] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0228] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0229] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0230] Embodiment 6

[0231] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the training of the image and text classification network and the image and text classification method provided by any embodiment of the present invention.

[0232] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0233] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0234] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A training method for an image-text classification network, characterized in that: include: Determine a picture and text classification network; the picture and text classification network includes an image encoder, a text encoder and a fusion module; Determine each batch of image and text samples; the image and text samples include image data and text information; part of the text information is description information, and part of the text information is category information; Among the image and text samples in the same batch, labeling the image data according to the text information; In the image and text samples of the same batch, the image data is input into the image encoder to be encoded into image features, the text information is input into the text encoder to be encoded into text features, and the image features and the text features are input into the fusion module to calculate the probability of mutual matching between the image data and the text information; Calculating a total loss value of mutual matching between the image data and the text information according to a difference between the probability and the label calculation; The image-text classification network is updated according to the total loss value.

2. The method according to claim 1, characterized in that The step of labeling the image data in the same batch of the image and text samples according to the text information includes: If the text information is description information, marking the image data with a specified index value as a label; If the text information is category information, annotating the image data with an index value corresponding to the category information as a label; Sorting the image and text samples of the same batch; wherein the image and text samples whose text information is descriptive information are sorted before the image and text samples whose text information is category information; If the sorting is completed, the index value of the tag is adjusted according to the sorting order.

3. The method according to claim 2, characterized in that The sorting of the image and text samples in the same batch includes: In the image and text samples of the same batch, the tags belonging to the specified index value are written into the first index list; In the image and text samples of the same batch, all the tags are written into the second index list; Create a third index list; Writing the tags in the first index list into the third index list, and writing the tags in the second index list other than the tags in the first index list into the third index list; Sorting the image and text samples of the same batch according to the order of the labels in the third index list; The step of adjusting the index value of the tag according to the sorting order includes: Counting the length of the first index list; Increasing the index values ​​of the tags sorted within the length in sequence; The length is accumulated for the index values ​​of the tags sorted after the length.

4. The method according to any one of claims 1 to 3, characterized in that The fusion module includes a scaling factor; the inputting the image feature and the text feature into the fusion module to calculate the probability of the image data and the text information matching each other includes: Calculating the product of the scaling factor, the image feature and the transposed matrix of the text feature to obtain a target matrix; the target matrix records the probability of matching the image data to the text information; The product of the scaling factor, the text feature and the transposed matrix of the image feature is calculated to obtain a reference matrix; the reference matrix records the probability of matching the text information to the image data.

5. The method according to claim 4, characterized in that The calculating the total loss value of the mutual matching between the image data and the text information according to the difference between the probability and the label calculation includes: Calculating a first partial loss value for matching the image data to the text information according to a difference between the target matrix and the label; Calculating a second partial loss value for matching the image data to the text information according to a difference between the reference matrix and the label; The first partial loss value and the second partial loss value are merged into a total loss value that matches the image data and the text information.

6. The method according to claim 5, characterized in that The calculating the second partial loss value from matching the image data to the text information according to the difference between the reference matrix and the label comprises: Dividing the reference matrix into a first interval matrix corresponding to the description information and a second interval matrix corresponding to the label information; Calculating a first sub-loss value for matching from the image data to the text information according to a difference between the first interval matrix and the label corresponding to the description information; Calculate a second sub-loss value for matching the image data to the text information according to a difference between the second interval matrix and the label corresponding to the category information; The first sub-loss value and the second sub-loss value are merged into a second partial loss value that matches the image data to the text information.

7. The method according to claim 6, characterized in that The first division loss value is expressed as: Among them, loss i2t is the loss value of the first division, Z j,i is the probability of the jth row and ith column in the target matrix, y i Represents the i-th label, batch_size is the batch size, Len label-o + k is the dimension of the target matrix; The first sub-loss value is expressed as: Among them, loss t2i_1 is the first sub-loss value, Z j,i is the probability of the jth row and ith column in the first interval matrix, y i is the label corresponding to the i-th description information, Len label-o is the number of the tags belonging to the description information; The second sub-loss value is expressed as: t=batch_size-Len label-o Among them, loss t2i_2 is the second sub-loss value, logit t2i_2 is the second interval matrix, σ is the activation function, batch_size is the batch size, Len label-o +k is the dimension of the target matrix, Y T [:,i] represents the transposed matrix of the one-hot encoding of the label with index value i; The total loss value is expressed as: loss=loss i2t +loss t2i_1 +loss t2i_2 Among them, loss is the total loss value, loss i2t is the loss value of the first division, loss t2i_1 is the first sub-loss value, loss t2i_2 is the second sub-loss value.

8. A method for classifying images and texts, characterized in that: include: Loading a picture-text classification network trained by the method according to any one of claims 1 to 7; the picture-text classification network comprises an image encoder, a text encoder and a fusion module; Receiving image data and text information input by a client; the text information is description information or category information; Input the image data into the image encoder and encode it into image features, input the text information into the text encoder and encode it into text features, input the image features and the text features into the fusion module to calculate the probability of matching the image data to the text information; The image data matching the text information is determined according to the probability.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the training method of the image-text classification network as described in any one of claims 1 to 7 or the image-text classification method as described in claim 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the training method of the image-text classification network according to any one of claims 1 to 7 or the image-text classification method according to claim 8 is implemented.

Images