Method, device, electronic device, medium and program product for determining image matching of copywriting

Abstract

The present disclosure relates to a method, device, electronic device, medium and program product for determining the image matching of a text, the method comprising the following steps: obtaining a target text; extracting text features of the target text; respectively determining the similarity between the image features of each image to be selected in the image set to be selected and the text features of the target text; and determining the image matching of the target text in the image set to be selected based on the similarity. By applying the technical solution provided by the embodiments of the present disclosure, by determining the similarity between the image features of the image to be selected and the text features of the target text, the image matching of the target text can be determined simply and quickly in the image set to be selected, thereby improving the efficiency of determining the image matching of the text, and without relying on whether similar texts can be accurately found, the image matching of the target text can be directly determined in the image set to be selected, thereby improving the matching accuracy of the image matching.

Description

Technical Field

[0001] The present disclosure relates to the field of computer application technology, and in particular to a method, device, electronic device, medium and program product for determining an image matching a document. Background Art

[0002] With the rapid development of computer technology and Internet technology, information publishing platforms often use deep learning technology to intelligently determine the pictures for news reports, literary reports and other documents to save labor costs.

[0003] At present, a commonly used method for determining the image matching of a copy is to first find similar copywriting to the target copywriting to be matched with an image, then search for images similar to the images matching the similar copywriting in a massive image library, and use the similar images found as the images matching the target copywriting. That is, the matching image is determined by using the similarity between texts and images as a bridge.

[0004] This method is relatively cumbersome. It is necessary to first find text similar to the target text before further finding images similar to the accompanying pictures of the similar text. It is inefficient. Moreover, if there is a deviation when searching for text similar to the target text, the images similar to the accompanying pictures of the similar text will not be able to accurately match the target text, and the matching accuracy of the accompanying pictures is low. Summary of the invention

[0005] The purpose of the present disclosure is to provide a method, device, electronic device, medium and program product for determining the matching images of a copy, so as to improve the efficiency of determining the matching images of the copy and improve the matching accuracy of the matching images.

[0006] In order to solve the above technical problems, the present disclosure provides the following technical solutions:

[0007] According to a first aspect of an embodiment of the present disclosure, a method for determining an image for a copy is provided, comprising:

[0008] Get the target copy;

[0009] Extracting text features of the target copy;

[0010] Determine the similarity between the image feature of each candidate image in the candidate image set and the text feature of the target copy;

[0011] According to the similarity, a picture matching the target text is determined in the set of images to be selected.

[0012] In a specific implementation of the present disclosure, the image set to be selected is obtained by the following steps:

[0013] Determining the target category to which the target document belongs based on the text features of the target document;

[0014] The material library is searched for material images belonging to the target category, and the found material images are added as candidate images to the candidate image set.

[0015] In a specific implementation of the present disclosure, the category to which each material image in the material library belongs is determined by the following steps:

[0016] Extracting image features of each material image in the material library;

[0017] Based on the image features of each material image, the category to which each material image belongs is determined.

[0018] In a specific implementation of the present disclosure, the extracting text features of the target document includes:

[0019] Extracting text features of the target copy through the text classification branch in the picture matching network model;

[0020] Correspondingly, the image features of each image to be selected in the image set to be selected are: extracted by the image classification branch in the image matching network model;

[0021] Accordingly, the step of respectively determining the similarity between the image feature of each image to be selected in the image set to be selected and the text feature of the target text includes:

[0022] The similarity between the image features of each candidate image in the candidate image set and the text features of the target text is determined respectively through the image-text similarity matching branch in the image matching network model.

[0023] In a specific embodiment of the present disclosure, the image matching network model is obtained by pre-training through the following steps:

[0024] Obtain a training data set, wherein the training data set includes a plurality of training data pairs, each training data pair includes a training image and a training text, and each training image and each training text has a corresponding category label;

[0025] Using the training data set and the category labels corresponding to the training images and the training texts in each training data pair, iteratively training the pre-built initial model of the picture matching network includes a text classification branch, an image classification branch and a picture-text similarity matching branch;

[0026] After the iterative training is completed, the image matching network model is obtained.

[0027] In a specific embodiment of the present disclosure, the pre-built image matching network initial model is iteratively trained by using the training data set and the category labels corresponding to the training images and training texts in each training data pair, including:

[0028] Determine each training data pair in the training data set as the current training data pair in turn, input the training image in the current training data pair into the image classification branch of the pre-constructed image matching network initial model, and determine the image classification loss corresponding to the training image in the current training data pair based on the output result of the image classification branch and the class label corresponding to the training image in the current training data pair;

[0029] Input the training text in the current training data pair into the text classification branch of the initial model of the image matching network, and determine the text classification loss corresponding to the training text in the current training data pair based on the output result of the text classification branch and the category label corresponding to the training text in the current training data pair;

[0030] Determining the modality contrast loss corresponding to the current training data pair through the image-text similarity matching branch of the initial model of the image matching network;

[0031] Determine a model loss based on an image classification loss corresponding to a training image in the current training data pair, a text classification loss corresponding to a training text in the current training data pair, and a modality contrast loss corresponding to the current training data pair;

[0032] Based on the model loss, the parameters of the image classification branch, the text classification branch and the image-text similarity matching branch included in the initial model of the picture matching network are adjusted.

[0033] In a specific implementation of the present disclosure, determining the image classification loss corresponding to the training image in the current training data pair based on the output result of the image classification branch and the class label corresponding to the training image in the current training data pair includes:

[0034] Extracting image features of the training image in the current training data pair using the image feature extraction module of the image classification branch;

[0035] Inputting the image features of the training image in the current training data pair into the image classification module of the image classification branch to obtain a first classification probability that the training image in the current training data pair belongs to each category;

[0036] Based on the first classification probability and the class label corresponding to the training image in the current training data pair, an image classification loss corresponding to the training image in the current training data pair is determined.

[0037] In a specific implementation of the present disclosure, determining the text classification loss corresponding to the training text in the current training data pair based on the output result of the text classification branch and the category label corresponding to the training text in the current training data pair includes:

[0038] Extracting text features of the training text in the current training data pair using the text feature extraction module of the text classification branch;

[0039] Inputting the text features of the training text in the current training data pair into the text classification module of the text classification branch to obtain a second classification probability that the training text in the current training data pair belongs to each category;

[0040] Based on the second classification probability and the class label corresponding to the training text in the current training data pair, a text classification loss corresponding to the training text in the current training data pair is determined.

[0041] In a specific implementation of the present disclosure, determining the modality contrast loss corresponding to the current training data pair through the image-text similarity matching branch of the initial model of the image matching network includes:

[0042] Inputting the image features corresponding to the training image and the text features corresponding to the training text in the current training data pair into the image-text similarity matching branch of the initial model of the image matching network;

[0043] Obtaining the image-text similarity matrix corresponding to the current training data pair using the similarity calculation module of the image-text similarity matching branch;

[0044] The loss calculation module of the image-text similarity matching branch is used to perform difference calculation on the image-text similarity matrix and a preset image-text matching matrix to determine the modality contrast loss corresponding to the current training data pair.

[0045] In a specific implementation of the present disclosure, the image-text matching matrix is ​​a diagonal matrix; the loss calculation module using the image-text similarity matching branch performs difference calculation on the image-text similarity matrix and a preset image-text matching matrix to determine the modality contrast loss corresponding to the current training data pair, including:

[0046] Utilizing the loss calculation module of the image-text similarity matching branch, respectively performing difference calculations on the rows corresponding to the image-text similarity matrix and the preset image-text matching matrix, and obtaining row-based differences based on the difference values ​​of all rows;

[0047] Calculating the difference between the columns corresponding to the image-text similarity matrix and the image-text matching matrix respectively, and obtaining the column-based difference based on the difference values ​​of all the columns;

[0048] Based on the row-based difference and the column-based difference, a modality contrast loss corresponding to the current training data pair is determined.

[0049] In a specific implementation of the present disclosure, the following steps are performed to determine whether the iterative training of the initial model of the image matching network is completed:

[0050] If the number of iterations reaches a preset threshold or the model loss of the initial model of the picture matching network stops decreasing, it is determined that the iterative training of the initial model of the picture matching network is completed.

[0051] In a specific implementation of the present disclosure, determining the image matching the target text in the set of candidate images according to the similarity includes:

[0052] A first set number of candidate images with the greatest similarity in the candidate image set are determined as the accompanying images of the target copy.

[0053] According to a second aspect of an embodiment of the present disclosure, a device for determining a picture matching a copy is provided, comprising:

[0054] A target copy obtaining unit, used for obtaining a target copy;

[0055] A text feature extraction unit, used to extract text features of the target document;

[0056] A similarity determination unit, used to respectively determine the similarity between the image feature of each candidate image in the candidate image set and the text feature of the target copy;

[0057] The picture matching determination unit is used to determine the picture matching of the target text in the set of candidate images according to the similarity.

[0058] In a specific implementation of the present disclosure, it further includes a unit for obtaining a set of images to be selected, which is used to obtain the set of images to be selected by the following steps:

[0059] Determining the target category to which the target document belongs based on the text features of the target document;

[0060] The material library is searched for material images belonging to the target category, and the found material images are added as candidate images to the candidate image set.

[0061] In a specific implementation of the present disclosure, an image category determination unit is further included, which is used to determine the category to which each material image in the material library belongs through the following steps:

[0062] Extracting image features of each material image in the material library;

[0063] Based on the image features of each material image, the category to which each material image belongs is determined.

[0064] In a specific implementation of the present disclosure, the text feature extraction unit is used to:

[0065] Extracting text features of the target copy through the text classification branch in the picture matching network model;

[0066] Correspondingly, the image features of each image to be selected in the image set to be selected are: extracted by the image classification branch in the image matching network model;

[0067] Accordingly, the similarity determination unit is used to:

[0068] The similarity between the image features of each candidate image in the candidate image set and the text features of the target text is determined respectively through the image-text similarity matching branch in the image matching network model.

[0069] In a specific embodiment of the present disclosure, a model training unit is further included, which is used to pre-train and obtain the image matching network model through the following steps:

[0070] Obtain a training data set, wherein the training data set includes a plurality of training data pairs, each training data pair includes a training image and a training text, and each training image and each training text has a corresponding category label;

[0071] Using the training data set and the category labels corresponding to the training images and the training texts in each training data pair, iteratively training the pre-built initial model of the picture matching network includes a text classification branch, an image classification branch and a picture-text similarity matching branch;

[0072] After the iterative training is completed, the image matching network model is obtained.

[0073] In a specific embodiment of the present disclosure, the model training unit is used to:

[0074] Determine each training data pair in the training data set as the current training data pair in turn, input the training image in the current training data pair into the image classification branch of the pre-constructed image matching network initial model, and determine the image classification loss corresponding to the training image in the current training data pair based on the output result of the image classification branch and the class label corresponding to the training image in the current training data pair;

[0075] Input the training text in the current training data pair into the text classification branch of the initial model of the image matching network, and determine the text classification loss corresponding to the training text in the current training data pair based on the output result of the text classification branch and the category label corresponding to the training text in the current training data pair;

[0076] Determining the modality contrast loss corresponding to the current training data pair through the image-text similarity matching branch of the initial model of the image matching network;

[0077] Determine a model loss based on an image classification loss corresponding to a training image in the current training data pair, a text classification loss corresponding to a training text in the current training data pair, and a modality contrast loss corresponding to the current training data pair;

[0078] Based on the model loss, the parameters of the image classification branch, the text classification branch and the image-text similarity matching branch included in the initial model of the picture matching network are adjusted.

[0079] In a specific embodiment of the present disclosure, the model training unit is used to:

[0080] Extracting image features of the training image in the current training data pair using the image feature extraction module of the image classification branch;

[0081] Inputting the image features of the training image in the current training data pair into the image classification module of the image classification branch to obtain a first classification probability that the training image in the current training data pair belongs to each category;

[0082] Based on the first classification probability and the class label corresponding to the training image in the current training data pair, an image classification loss corresponding to the training image in the current training data pair is determined.

[0083] In a specific embodiment of the present disclosure, the model training unit is used to:

[0084] Extracting text features of the training text in the current training data pair using the text feature extraction module of the text classification branch;

[0085] Inputting the text features of the training text in the current training data pair into the text classification module of the text classification branch to obtain a second classification probability that the training text in the training data pair belongs to each category;

[0086] Based on the second classification probability and the class label corresponding to the training text in the current training data pair, a text classification loss corresponding to the training text in the current training data pair is determined.

[0087] In a specific embodiment of the present disclosure, the model training unit is used to:

[0088] Inputting the image features corresponding to the training image and the text features corresponding to the training text in the current training data pair into the image-text similarity matching branch of the initial model of the image matching network;

[0089] Obtaining the image-text similarity matrix corresponding to the current training data pair using the similarity calculation module of the image-text similarity matching branch;

[0090] The loss calculation module of the image-text similarity matching branch is used to perform difference calculation on the image-text similarity matrix and a preset image-text matching matrix to determine the modality contrast loss corresponding to the current training data pair.

[0091] In a specific implementation of the present disclosure, the image-text matching matrix is ​​a diagonal matrix; the model training unit is used to:

[0092] Utilizing the loss calculation module of the image-text similarity matching branch, respectively performing difference calculations on the rows corresponding to the image-text similarity matrix and the preset image-text matching matrix, and obtaining row-based differences based on the difference values ​​of all rows;

[0093] Calculating the difference between the columns corresponding to the image-text similarity matrix and the image-text matching matrix respectively, and obtaining the column-based difference based on the difference values ​​of all the columns;

[0094] Based on the row-based difference and the column-based difference, a modality contrast loss corresponding to the current training data pair is determined.

[0095] In a specific implementation of the present disclosure, the model training unit is used to determine whether the iterative training of the initial model of the image matching network is completed through the following steps:

[0096] If the number of iterations reaches a preset threshold or the model loss of the initial model of the picture matching network stops decreasing, it is determined that the iterative training of the initial model of the picture matching network is completed.

[0097] In a specific implementation of the present disclosure, the image matching determination unit is used to:

[0098] A first set number of candidate images with the greatest similarity in the candidate image set are determined as the accompanying images of the target copy.

[0099] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0100] processor;

[0101] a memory for storing instructions executable by the processor;

[0102] The processor is configured to execute the instructions to implement the method for determining the image matching for the text as described in the first aspect.

[0103] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the method for determining an image matching for a document described in the first aspect.

[0104] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, which includes computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that an electronic device having the processor executes the method for determining the image matching of the copy described in the first aspect.

[0105] By applying the technical solution provided by the embodiment of the present disclosure, after obtaining the target text, the text features of the target text are first extracted, and then the similarity between the image features of each image to be selected in the image set to be selected and the text features of the target text is determined respectively, and then the matching picture of the target text is determined in the image set to be selected based on the similarity. By determining the similarity between the image features of the image to be selected and the text features of the target text, the matching picture of the target text can be determined simply and quickly in the image set to be selected, thereby improving the efficiency of determining the matching picture of the text, and there is no need to rely on whether similar texts can be accurately found, and the matching picture of the target text is directly determined in the image set to be selected, thereby improving the matching accuracy of the matching picture.

[0106] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0108] Figure 1 It is a flowchart of an implementation method of a method for determining an image matching for a copy in an embodiment of the present disclosure;

[0109] Figure 2 A schematic diagram of the structure of the initial network model in the embodiment of the present disclosure;

[0110] Figure 3 A schematic diagram of the training process of the initial network model in the embodiment of the present disclosure;

[0111] Figure 4 Schematic diagram of the process of matching pictures of target text in the embodiment of the present disclosure;

[0112] Figure 5 It is a structural schematic diagram of a device for determining a picture matching a copy in an embodiment of the present disclosure;

[0113] Figure 6 It is a structural schematic diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0114] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0115] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0116] The core of the present invention is to provide a method for determining the image matching of a text, which can be applied to any scenario where the image matching of a text needs to be determined. For example, an information publishing platform has a news report to be published, and the text alone cannot attract the attention of users. It is necessary to determine the image matching the text of the news report. Figure 1 Publish together to better attract users and help users better understand the news report.

[0117] See also Figure 1 FIG. 1 is a flowchart of a method for determining a text image provided by an embodiment of the present disclosure. The method may include the following steps:

[0118] S110: Obtain target copy.

[0119] In the embodiment of the present disclosure, the target text may be any text that is currently to be assigned with an image, such as a news report or a literary report.

[0120] After obtaining the target copy, you can proceed to the next steps.

[0121] S120: Extracting text features of the target copy.

[0122] After obtaining the target text, the text features of the target text can be extracted. Specifically, the text of the target text, including the title and content, can be segmented and encoded first, such as one-hot encoding (one-bit effective encoding or one-hot encoding). One-hot encoding is to achieve the mapping of text → 0-N and other numbers through a text dictionary, where N is the total number of characters in the entire dictionary library. For example, "Beijing" is mapped to "1002". Then, the text features of the target text can be extracted based on the encoding results.

[0123] S130: Determine the similarity between the image feature of each candidate image in the candidate image set and the text feature of the target copy.

[0124] In the embodiment of the present disclosure, a set of to-be-selected images may be obtained in advance, and the set of to-be-selected images may include a number of to-be-selected images, and image features of each to-be-selected image may be extracted in advance.

[0125] After obtaining the target text and extracting the text features of the target text, the similarity between the image features of each candidate image in the candidate image set and the text features of the target text can be determined, thereby respectively determining the similarity between the image features of each candidate image and the text features of the target text.

[0126] When determining the similarity, the feature dimensions of the image features of the selected image and the feature dimensions of the text features of the target copy may be kept consistent through compression or the like.

[0127] S140: Determine the image matching the target copy in the image set to be selected based on the similarity.

[0128] After respectively determining the similarity between the image features of each candidate image in the candidate image set and the text features of the target text, the image matching the target text can be further determined in the candidate image set according to the magnitude of the similarity. For any candidate image in the candidate image set, the greater the similarity between the image features of the candidate image and the text features of the target text, the more it can be considered that the candidate image matches the target text, and the more suitable the candidate image is as the matching image of the target text; conversely, the smaller the similarity between the image features of the candidate image and the text features of the target text, the less it can be considered that the candidate image matches the target text, and the less suitable the candidate image is as the matching image of the target text.

[0129] After the target text image is determined in the image set to be selected based on the similarity, the target text image can be further output, and the relevant personnel can manually determine which images to retain. Alternatively, if the target text image cannot be determined in the image set to be selected based on the similarity, a prompt message that there is no suitable image can be output to facilitate the relevant personnel to manually determine it in time.

[0130] By applying the method provided by the embodiment of the present disclosure, after obtaining the target text, the text features of the target text are first extracted, and then the similarity between the image features of each image to be selected in the image set to be selected and the text features of the target text is determined respectively, and then the matching picture of the target text is determined in the image set to be selected based on the similarity. By determining the similarity between the image features of the image to be selected and the text features of the target text, the matching picture of the target text can be determined simply and quickly in the image set to be selected, thereby improving the efficiency of determining the matching picture of the text, and it is not necessary to rely on whether similar texts can be accurately found, and the matching picture of the target text can be directly determined in the image set to be selected, thereby improving the matching accuracy of the matching picture.

[0131] In one embodiment of the present disclosure, the image set to be selected can be obtained by the following steps:

[0132] Step 1: Based on the text features of the target copy, determine the target category to which the target copy belongs;

[0133] Step 2: Search for material images belonging to the target category in the material library, and add the found material images as candidate images to the candidate image set.

[0134] For the convenience of description, the above two steps are combined for explanation.

[0135] In the disclosed embodiment, a material library can be pre-established by image acquisition or image transformation, and the material library includes a number of material images belonging to various categories, such as material images belonging to sports categories, movie categories, entertainment categories, and funny categories. The same material image can belong to one or more categories. Specifically, the category to which each material image belongs can be determined by manual recognition, and the category to which each material image in the material library belongs can also be determined by the following steps:

[0136] Extract image features of each material image in the material library;

[0137] Based on the image features of each material image, the category to which each material image belongs is determined.

[0138] That is, for each material image in the material library, the image features of the material image can be extracted first, such as by using an image feature extraction module to extract the image features of the material image, and then based on the image features of the material image, the category to which the material image belongs can be determined, such as by using an image classification module based on the image features of the material image to obtain the classification probability that the material image belongs to each category, and the category with the highest classification probability is determined as the category to which the material image belongs.

[0139] Every time a material image is added to the material library, the image features of the material image can be extracted, and based on the image features of the material image, the category to which the material image belongs is determined, and the category to which the material image belongs is recorded in the material library.

[0140] Based on the image features of each material image, the category to which each material image belongs can be accurately determined, providing a basic guarantee for the subsequent accurate determination of the accompanying images.

[0141] After obtaining the target text and extracting the text features of the target text, the target category to which the target text belongs can be determined based on the text features of the target text. Specifically, the target category to which the target text belongs can be determined by a text classification module.

[0142] After determining the target category to which the target copy belongs, the material library may be searched for material images belonging to the target category based on the target category, and then the searched material images are added as candidate images to the candidate image set.

[0143] This will make the category of the candidate images in the candidate image set the same as the target category of the target copy, and the possibility of matching the candidate images and the target copy is high. For example, if the target category of the target copy is "sports category", then the category of each candidate image in the candidate image set is also "sports category".

[0144] By implementing coarse filtering through target categories, the images to be selected in the image set can be streamlined, and material images with low relevance can be screened out, which can improve the execution efficiency of the subsequent steps of separately determining the similarity between the image features of each image to be selected in the image set and the text features of the target copy.

[0145] Of course, in order to ensure that the image set to be selected is large and complete, all material images in the material library can be added to the image set to be selected as images to be selected, so as to ensure that the image matching the target copy can be determined in the image set to be selected.

[0146] In one embodiment of the present disclosure, determining the image matching the target text in the image set to be selected according to the similarity may include the following steps:

[0147] A predetermined number of candidate images with the greatest similarity in the candidate image set are determined as the accompanying images for the target copy.

[0148] The set number can be set and adjusted according to the actual situation. For example, if it is set to 3, the first three candidate images with the greatest similarity will be determined as the matching images of the target copy. In this way, it can be ensured that the image features of the matching images of the determined target copy are similar to the text features of the target copy, thereby improving the accuracy of the determined matching images.

[0149] In one embodiment of the present disclosure, extracting text features of a target document may include the following steps:

[0150] Extract the text features of the target copy through the text classification branch in the picture matching network model;

[0151] Correspondingly, the image features of each candidate image in the candidate image set are: extracted by the image classification branch in the image matching network model;

[0152] Accordingly, respectively determining the similarity between the image feature of each candidate image in the candidate image set and the text feature of the target copy may include the following steps:

[0153] The similarity between the image features of each candidate image in the candidate image set and the text features of the target text is determined through the image-text similarity matching branch in the image matching network model.

[0154] In an embodiment of the present disclosure, a picture matching network model may be pre-trained and obtained, and the picture matching network model may include a text classification branch, an image classification branch, and a picture-text similarity matching branch.

[0155] The image matching network model can be pre-trained by following the steps below:

[0156] The first step is to obtain a training data set, which includes multiple training data pairs. Each training data pair includes a training image and a training text. Each training image and each training text has a corresponding class label.

[0157] The second step: using the training data set and the class labels corresponding to the training images and training texts in each training data pair, iteratively train the pre-built initial model of the picture matching network. The initial model of the picture matching network includes a text classification branch, an image classification branch, and a picture-text similarity matching branch.

[0158] The third step: After the iterative training is completed, the image matching network model is obtained.

[0159] For easier understanding, the above three steps are explained together.

[0160] In the disclosed embodiment, a training data set may be obtained first. The training data set includes a plurality of training data pairs, each training data pair including a training image and a training text. The training data pairs may be obtained by data collection or the like. For each training data pair, the training image in the training data pair may be one or more accompanying images of the training text in the training data pair, so as to improve the accuracy of model training. For example, a news report is collected by data collection, the text of the news report may be used as the training text, the accompanying image of the news report may be used as the training image, and the training text and the training image may constitute a training data pair. If the news report has multiple accompanying images, each accompanying image may be used as a training image to constitute multiple training data pairs with the training text, and multiple accompanying images may be used as training images to constitute one training data pair with the training text.

[0161] Each training image and each training text has a corresponding category label, such as sports category, movie category, entertainment category, funny category, etc.

[0162] After obtaining the training data set, the pre-built initial model of the picture matching network can be iteratively trained using the training data set and the category labels corresponding to the training images and training texts in each training data pair. The initial model of the picture matching network includes a text classification branch, an image classification branch, and a picture-text similarity matching branch. During the iterative training process, all parameters of the text classification branch, image classification branch, and picture-text similarity matching branch included in the initial model of the picture matching network are involved in the training.

[0163] You can use the SGD (Stochastic Gradient Descent) optimizer, set the initial learning rate to 0.01, and set the batch size to 64*the number of GPU (graphics processing unit) cards.

[0164] During the iterative training process, the parameters of the initial model of the picture matching network can be continuously adjusted to make the initial model of the picture matching network continue to converge. The following steps can be used to determine whether the iterative training of the initial model of the picture matching network is completed:

[0165] If the number of iterations reaches a preset threshold or the model loss of the initial model of the picture matching network stops decreasing, it can be determined that the iterative training of the initial model of the picture matching network is completed.

[0166] The number threshold can be set according to actual conditions, and the embodiments of the present disclosure do not limit this. By determining whether the number of iterations or the model loss of the initial model of the image matching network decreases, it is possible to accurately determine whether the iterative training of the initial model of the image matching network is completed.

[0167] After the iterative training is completed, the initial model of the image matching network after the training can be used as the image matching network model and applied to the actual scene. Through iterative training, the matching accuracy of the image matching network model can be improved.

[0168] Specifically, the text features of the target text can be extracted through the text classification branch in the picture matching network model, the image features of each candidate image in the candidate image set can be extracted through the image classification branch in the picture matching network model, and the similarity between the image features of each candidate image in the candidate image set and the text features of the target text can be determined respectively through the image-text similarity matching branch in the picture matching network model.

[0169] The structure of the initial model of the picture matching network is the same as that of the picture matching network model.

[0170] The picture matching network model can be used to accurately extract text features and image features, and can accurately determine the similarity between the image features of the selected image and the text features of the target copy.

[0171] In one embodiment of the present disclosure, using the training data set and the category labels corresponding to the training images and training texts in each training data pair, iteratively training the pre-built image matching network initial model may include the following steps:

[0172] Step 1: Determine each training data pair in the training data set as the current training data pair in turn, input the training image in the current training data pair into the image classification branch of the pre-built image matching network initial model, and determine the image classification loss corresponding to the training image in the current training data pair based on the output result of the image classification branch and the class label corresponding to the training image in the current training data pair;

[0173] Step 2: Input the training text in the current training data pair into the text classification branch of the initial model of the image matching network, and determine the text classification loss corresponding to the training text in the current training data pair based on the output result of the text classification branch and the class label corresponding to the training text in the current training data pair;

[0174] Step 3: Determine the modality contrast loss corresponding to the current training data pair through the image similarity matching branch of the initial model of the image matching network;

[0175] Step 4: Determine the model loss based on the image classification loss corresponding to the training image in the current training data pair, the text classification loss corresponding to the training text in the current training data pair, and the modality contrast loss corresponding to the current training data pair;

[0176] Step 5: Based on the model loss, adjust the parameters of the image classification branch, text classification branch, and image-text similarity matching branch of the initial model of the image matching network.

[0177] For the convenience of description, the above steps are combined for explanation.

[0178] The obtained training data set includes multiple training data pairs, each training data pair includes a training image and a training text, and each training image and each training text has a corresponding category label.

[0179] Each training data pair in the training data set can be determined as the current training data pair in turn, and the training image in the current training data pair can be input into the image classification branch of the pre-built initial model of the image matching network. After the training image is processed by the image classification branch, the output result of the image classification branch can be obtained. The output result of the image classification branch can include the classification probability that the training image belongs to each category. Based on the output result of the image classification branch and the category label corresponding to the training image in the current training data pair, the image classification loss corresponding to the training image in the current training data pair can be determined. The current training data pair is the training data pair targeted by the current operation, that is, each training data pair in the training data set is operated according to the operation of the current training data pair.

[0180] In a specific embodiment of the present disclosure, Figure 2 As shown, the image classification branch may include an image feature extraction module and an image classification module. Based on the output result of the image classification branch and the class label corresponding to the training image in the current training data pair, determining the image classification loss corresponding to the training image in the current training data pair may include the following steps:

[0181] The first step: using the image feature extraction module of the image classification branch to extract the image features of the training image in the current training data pair;

[0182] The second step: inputting the image features of the training image in the current training data pair into the image classification module of the image classification branch to obtain the first classification probability that the training image in the current training data pair belongs to each category;

[0183] The third step: based on the first classification probability and the class label corresponding to the training image in the current training data pair, determine the image classification loss corresponding to the training image in the current training data pair.

[0184] In the disclosed embodiment, the image feature extraction module included in the image classification branch can use a feature extraction network such as a convolutional network, such as Resent-50D, Inception-V3, etc. The training image in the current training data pair is input into the image feature extraction module of the image classification branch, and the image feature extraction module can be used to extract the deep content expression features of the training image, that is, the image feature X, and the feature dimension can be D.

[0185] After extracting the image features of the training image in the current training data pair, the image features of the training image can be further input into the image classification module of the image classification branch. The training image is classified by the image classification module to obtain the first classification probability that the training image belongs to each category. The image classification module can include multiple fully connected layers.

[0186] After obtaining the first classification probability that the training image in the current training data pair belongs to each category, the image classification loss corresponding to the training image in the current training data pair can be determined based on the first classification probability and the category label corresponding to the training image in the current training data pair. Specifically, the first classification probability and the category label corresponding to the training image in the current training data pair can be used as inputs of the first cross entropy loss function, and the first cross entropy loss function is used to determine the first cross entropy loss. The first cross entropy loss is the image classification loss corresponding to the training image in the current training data pair, which can be recorded as L1, such as Figure 2 shown.

[0187] like Figure 3 As shown, there are multiple frames of training images. Resent-50D can be used to extract image features from each frame of the image, and then multi-frame feature interaction can be performed to obtain the final image features of the training image. Based on the image features of the training image, the first classification probability of the training image belonging to each category can be obtained, and then based on the first classification probability and the category label corresponding to the training image, the image classification loss L1 corresponding to the training image can be determined.

[0188] The first classification probability that the training image belongs to each category is accurately obtained through the image feature extraction module and the image classification module, and then the image classification loss corresponding to the training image can be accurately determined by using the first classification probability and the category label corresponding to the training image.

[0189] While inputting the training image in the current training data pair into the image classification branch of the initial model of the picture matching network and determining the image classification loss corresponding to the training image in the current training data pair, the training text in the current training data pair can be input into the text classification branch of the initial model of the picture matching network. After the training text is processed by the text classification branch, the output result of the text classification branch can be obtained. The output result of the text classification branch can include the classification probability that the training text belongs to each category. Based on the output result of the text classification branch and the category label corresponding to the training text in the current training data pair, the text classification loss corresponding to the training text in the current training data pair can be determined.

[0190] In a specific embodiment of the present disclosure, Figure 2As shown, the text classification branch includes a text feature extraction module and a text classification module. Based on the output result of the text classification branch and the class label corresponding to the training text in the current training data pair, determining the text classification loss corresponding to the training text in the current training data pair may include the following steps:

[0191] The first step: using the text feature extraction module of the text classification branch to extract the text features of the training text in the current training data pair;

[0192] The second step: inputting the text features of the training text in the current training data pair into the text classification module of the text classification branch to obtain the second classification probability that the training text in the current training data pair belongs to each category;

[0193] The third step: based on the second classification probability and the class label corresponding to the training text in the current training data pair, determine the text classification loss corresponding to the training text in the current training data pair.

[0194] In the disclosed embodiment, the text feature extraction module included in the text classification branch can use a feature extractor for natural language processing, such as Bert, etc. The training text in the current training data pair can be first segmented and encoded, such as one-hot encoding (one-bit effective encoding or one-hot encoding). Then the encoded training text is input into the text feature extraction module of the text classification branch, and the deep content expression features of the training text, that is, the text features T, can be extracted by the text feature extraction module, and the feature dimension can be D.

[0195] After extracting the image features of the training text in the current training data pair, the text features of the training text can be further input into the text classification module of the text classification branch. The training text is classified by the text classification module to obtain the second classification probability that the training text belongs to each category. The text classification module can include multiple fully connected layers.

[0196] After obtaining the second classification probability that the training text in the current training data pair belongs to each category, the text classification loss corresponding to the training text in the current training data pair can be determined based on the second classification probability and the category label corresponding to the training text in the current training data pair. Specifically, the second classification probability and the category label corresponding to the training text in the current training data pair can be used as inputs of the second cross entropy loss function, and the second cross entropy loss function is used to determine the second cross entropy loss. The second cross entropy loss is the text classification loss corresponding to the training text in the current training data pair, which can be recorded as L2, such as Figure 2 shown.

[0197] like Figure 3As shown, the training text is "flowers", and text features can be extracted for the training text. Then, based on the text features of the training text, the second classification probability that the training text belongs to each category can be obtained. Then, based on the second classification probability and the category label corresponding to the training text, the text classification loss L2 corresponding to the training text can be determined.

[0198] The second classification probability that the training text belongs to each category is accurately obtained through the text feature extraction module and the text classification module. Then, the text classification loss corresponding to the training text can be accurately determined by using the second classification probability and the category label corresponding to the training text.

[0199] After determining the image classification loss corresponding to the training image and the text classification loss corresponding to the training text in the current training data pair, the modality contrast loss corresponding to the current training data pair can be further determined through the image-text similarity matching branch of the initial model of the image matching network.

[0200] In a specific embodiment of the present disclosure, Figure 2 As shown, the image-text similarity matching branch may include a similarity calculation module and a loss calculation module. Determining the modality contrast loss corresponding to the current training data pair through the image-text similarity matching branch of the initial model of the image matching network may include the following steps:

[0201] The first step: input the image features corresponding to the training image and the text features corresponding to the training text in the current training data pair into the image-text similarity matching branch of the initial model of the image matching network;

[0202] The second step: using the similarity calculation module of the image-text similarity matching branch to obtain the image-text similarity matrix corresponding to the current training data pair;

[0203] The third step: Use the loss calculation module of the image-text similarity matching branch to calculate the difference between the image-text similarity matrix and the preset image-text matching matrix to determine the modal contrast loss corresponding to the current training data pair.

[0204] In the disclosed embodiment, the image features corresponding to the training image and the text features corresponding to the training text in the current training data pair can be obtained first. Specifically, the image features of the training image can be extracted using the image feature extraction module of the image classification branch, and the text features of the training text can be extracted using the text feature extraction module of the text classification branch.

[0205] The image features corresponding to the training image and the text features corresponding to the training text in the current training data pair can be input into the image-text similarity matching branch of the initial model of the image matching network.

[0206] First, the similarity calculation module of the image-text similarity matching branch is used to calculate and process, and the image-text similarity matrix corresponding to the current training data pair, such as the cos similarity matrix, is obtained by using the similarity calculation module. Then, the loss calculation module of the image-text similarity matching branch is used to calculate the difference between the image-text similarity matrix and the preset image-text matching matrix, and determine the modal contrast loss corresponding to the current training data pair.

[0207] The difference between the image-text similarity matrix and the image-text matching matrix is ​​calculated. Specifically, KL divergence calculation or square difference distance calculation can be performed. KL divergence is Kullback-Leibler divergence, which can also be called relative entropy or information divergence. It is used to measure the asymmetry of the difference between two probability distributions.

[0208] Specifically, the image-text matching matrix can be a diagonal matrix, that is, a matrix in which the diagonals are all 1 and the others are all 0. The loss calculation module of the image-text similarity matching branch can be used to first perform difference calculations on the rows corresponding to the image-text similarity matrix and the preset image-text matching matrix, and obtain the row-based differences based on the difference values ​​of all rows. At the same time, the difference calculations are performed on the columns corresponding to the image-text similarity matrix and the image-text matching matrix, and obtain the column-based differences based on the difference values ​​of all columns. Based on the row-based differences and the column-based differences, the modal contrast loss corresponding to the current training data pair is determined.

[0209] That is, after using the similarity calculation module of the image-text similarity matching branch to obtain the image-text similarity matrix corresponding to the current training data pair, the loss calculation module of the image-text similarity matching branch can be used to calculate the differences between the rows corresponding to the image-text similarity matrix and the image-text matching matrix, and obtain the difference values ​​of all rows. Based on the difference values ​​of all rows, the row-based difference can be obtained. For example, when the difference calculation is a KL divergence calculation, a row-based KL divergence can be obtained, denoted as KL1, and the KL divergence values ​​of all rows can be added and averaged to obtain the row-based KL divergence. For another example, when the difference calculation is a square difference distance calculation, the square difference distance values ​​of all rows can be added and averaged to obtain the row-based distance.

[0210] At the same time, the loss calculation module of the image-text similarity matching branch can be used to calculate the differences between the columns corresponding to the image-text similarity matrix and the image-text matching matrix, and obtain the difference values ​​of all columns. Based on the difference values ​​of all columns, the column-based difference can be obtained. For example, when the difference calculation is KL divergence calculation, the column-based KL divergence can be obtained, denoted as KL2, and the KL divergence values ​​of all columns can be summed and averaged to obtain the column-based KL divergence. For another example, when the difference calculation is square difference distance calculation, the square difference distance values ​​of all columns can be summed and averaged to obtain the column-based distance.

[0211] After respectively obtaining the row-based difference and the column-based difference, the modal contrast loss corresponding to the current training data pair can be determined based on the row-based difference and the column-based difference. For example, the weighted sum of the row-based difference and the column-based difference can be determined as the modal contrast loss corresponding to the current training data pair. For example, if the difference calculation is a KL divergence calculation, the weighted sum of the row-based KL divergence and the column-based KL divergence can be determined as the modal contrast loss corresponding to the current training data pair. For another example, if the difference calculation is a squared difference distance calculation, the weighted sum of the row-based distance and the column-based distance can be determined as the modal contrast loss corresponding to the current training data pair.

[0212] The weight can be set and adjusted according to the actual situation. For example, if it is set to 0.5, then, taking the difference calculation as KL divergence calculation as an example, the modal contrast loss L3 = KL1*0.5+KL2*0.5. Figure 2 shown.

[0213] like Figure 3 As shown, after obtaining the image features of the training images and the text features of the training texts, the image-text similarity matrix can be obtained. By performing KL divergence calculation on the image-text similarity matrix and the image-text matching matrix, the modal contrast loss L3 can be determined.

[0214] Through the above method, the modal contrast loss corresponding to each training data pair can be accurately determined, thus providing a basic guarantee for the smooth progress of subsequent model training.

[0215] The image-text similarity matching branch is used to determine the modal contrast loss corresponding to the current training data pair, making it possible to represent the image features of the training image and the text features of the training text in one space.

[0216] At this point, the image classification loss corresponding to the training image in the current training data pair, the text classification loss corresponding to the training text, and the modality contrast loss corresponding to the current training data pair have been determined. Based on these losses, the model loss can be determined, that is, the total loss of the initial model of the image matching network, which can be recorded as Loss, Loss = L1 + L2 + L3. Alternatively, the model loss Loss can be the weighted sum of L1, L2 and L3.

[0217] Based on the model loss, the parameters of the image classification branch, text classification branch, and image-text similarity matching branch included in the initial model of the picture matching network can be adjusted. Specifically, the model parameters can be adjusted with the goal of minimizing the model loss so that the initial model of the picture matching network continues to converge.

[0218] In the process of iteratively training the initial model of the image matching network using the training data set and the category labels corresponding to the training images and training texts in each training data pair, the model loss can be accurately determined based on the image classification loss, text classification loss, and modality contrast loss. Therefore, based on the model loss, the model parameters can be effectively adjusted to improve the matching accuracy of the model.

[0219] When the number of iterations reaches the set threshold or the model loss stops decreasing, it can be determined that the iterative training of the initial model of the image matching network is completed and the image matching network model is obtained.

[0220] Then, in the process of determining the matching pictures for the target copy, Figure 4 As shown, the text features of the target text can be extracted by the text feature extraction module of the text classification branch in the picture matching network model, the target category to which the target text belongs can be determined by the text classification module of the text classification branch, the image features of each material image in the material library can be extracted by the image feature extraction module of the image classification branch in the picture matching network model, the category to which each material image belongs can be determined by the image classification module of the image classification branch, and according to the target category to which the target text belongs, the material library is searched for material images belonging to the target category to obtain a set of images to be selected, the image features of the images to be selected in the set of images to be selected can call the image features of the corresponding material images in the material library, and the similarity between the image features of each image to be selected in the set of images to be selected and the text features of the target text can be determined by the image-text similarity matching branch in the picture matching network model, and then according to the similarity, the picture matching for the target text is determined in the set of images to be selected.

[0221] The disclosed embodiment performs a coarse filtering based on the target category to which the target text belongs, screens out a large number of material images with low relevance, retains the candidate images with strong relevance to the target text, and then implements similarity matching of text features and image features through a similarity calculation module, so that the image matching of the determined target text is more accurate, thereby maximizing the matching accuracy of the text and the image while saving manpower matching costs and machine operating costs.

[0222] Corresponding to the above method embodiment, the embodiment of the present disclosure further provides a device for determining a picture matching for a copy. The device for determining a picture matching for a copy described below and the method for determining a picture matching for a copy described above can refer to each other.

[0223] See also Figure 5 As shown, the device may include the following units:

[0224] A target copy obtaining unit 510 is used to obtain a target copy;

[0225] A text feature extraction unit 520, used to extract text features of the target document;

[0226] A similarity determination unit 530 is used to determine the similarity between the image feature of each candidate image in the candidate image set and the text feature of the target copy;

[0227] The picture matching determination unit 540 is used to determine the picture matching of the target copy in the image set to be selected according to the similarity.

[0228] By using the device provided by the embodiment of the present disclosure, after obtaining the target text, the text features of the target text are first extracted, and then the similarity between the image features of each image to be selected in the image set to be selected and the text features of the target text is determined respectively, and then the matching picture of the target text is determined in the image set to be selected based on the similarity. By determining the similarity between the image features of the image to be selected and the text features of the target text, the matching picture of the target text can be determined in the image set to be selected simply and quickly, thereby improving the efficiency of determining the matching picture of the text, and there is no need to rely on whether similar texts can be accurately found, and the matching picture of the target text is directly determined in the image set to be selected, thereby improving the matching accuracy of the matching picture.

[0229] In a specific embodiment of the present disclosure, a unit for obtaining a set of images to be selected is further included, which is used to obtain the set of images to be selected by the following steps:

[0230] Based on the text features of the target copy, determine the target category to which the target copy belongs;

[0231] Search for material images belonging to the target category in the material library, and add the found material images as candidate images to the candidate image set.

[0232] In a specific implementation of the present disclosure, an image category determination unit is further included, which is used to determine the category to which each material image in the material library belongs through the following steps:

[0233] Extract image features of each material image in the material library;

[0234] Based on the image features of each material image, the category to which each material image belongs is determined.

[0235] In a specific implementation of the present disclosure, the text feature extraction unit 520 is used to:

[0236] Extract the text features of the target copy through the text classification branch in the picture matching network model;

[0237] Correspondingly, the image features of each candidate image in the candidate image set are: extracted by the image classification branch in the image matching network model;

[0238] Accordingly, the similarity determination unit 530 is used to:

[0239] The similarity between the image features of each candidate image in the candidate image set and the text features of the target text is determined through the image-text similarity matching branch in the image matching network model.

[0240] In a specific embodiment of the present disclosure, a model training unit is further included, which is used to pre-train and obtain a graph matching network model through the following steps:

[0241] Obtain a training data set, the training data set includes multiple training data pairs, each training data pair includes a training image and a training text, and each training image and each training text has a corresponding category label;

[0242] The pre-built initial model of the picture matching network is iteratively trained using the training data set and the category labels corresponding to the training images and training texts in each training data pair. The initial model of the picture matching network includes a text classification branch, an image classification branch, and an image-text similarity matching branch.

[0243] After the iterative training is completed, the image matching network model is obtained.

[0244] In a specific embodiment of the present disclosure, the model training unit is used to:

[0245] Determine each training data pair in the training data set as the current training data pair in turn, input the training image in the current training data pair into the image classification branch of the pre-built initial model of the image matching network, and determine the image classification loss corresponding to the training image in the current training data pair based on the output result of the image classification branch and the class label corresponding to the training image in the current training data pair;

[0246] Input the training text in the current training data pair into the text classification branch of the initial model of the image matching network, and determine the text classification loss corresponding to the training text in the current training data pair based on the output result of the text classification branch and the class label corresponding to the training text in the current training data pair;

[0247] Determine the modality contrast loss corresponding to the current training data pair through the image-text similarity matching branch of the initial model of the image matching network;

[0248] Determine a model loss based on an image classification loss corresponding to a training image in a current training data pair, a text classification loss corresponding to a training text in a current training data pair, and a modality contrast loss corresponding to the current training data pair;

[0249] Based on the model loss, adjust the parameters of the image classification branch, text classification branch, and image-text similarity matching branch of the initial model of the picture matching network.

[0250] In a specific embodiment of the present disclosure, the model training unit is used to:

[0251] Extracting image features of the training image in the current training data pair using the image feature extraction module of the image classification branch;

[0252] Inputting the image features of the training image in the current training data pair into the image classification module of the image classification branch to obtain the first classification probability that the training image in the current training data pair belongs to each category;

[0253] An image classification loss corresponding to the training image in the current training data pair is determined based on the first classification probability and the class label corresponding to the training image in the current training data pair.

[0254] In a specific embodiment of the present disclosure, the model training unit is used to:

[0255] Using the text feature extraction module of the text classification branch to extract text features of the training text in the current training data pair;

[0256] Inputting the text features of the training text in the current training data pair into the text classification module of the text classification branch to obtain the second classification probability that the training text in the current training data pair belongs to each category;

[0257] Based on the second classification probability and the class label corresponding to the training text in the current training data pair, a text classification loss corresponding to the training text in the current training data pair is determined.

[0258] In a specific embodiment of the present disclosure, the model training unit is used to:

[0259] Input the image features corresponding to the training image and the text features corresponding to the training text in the current training data pair into the image-text similarity matching branch of the initial model of the image matching network;

[0260] The similarity calculation module of the image-text similarity matching branch is used to obtain the image-text similarity matrix corresponding to the current training data pair;

[0261] The loss calculation module of the image-text similarity matching branch is used to calculate the difference between the image-text similarity matrix and the preset image-text matching matrix to determine the modal contrast loss corresponding to the current training data pair.

[0262] In a specific implementation of the present disclosure, the image-text matching matrix is ​​a diagonal matrix; the model training unit is used to:

[0263] The loss calculation module of the image-text similarity matching branch is used to calculate the difference between the rows corresponding to the image-text similarity matrix and the preset image-text matching matrix, and the row-based difference is obtained based on the difference values ​​of all rows;

[0264] The difference between the columns corresponding to the image-text similarity matrix and the image-text matching matrix is ​​calculated respectively, and the column-based difference is obtained based on the difference values ​​of all columns;

[0265] Based on the row-based difference and column-based difference, the modality contrast loss corresponding to the current training data pair is determined.

[0266] In a specific implementation of the present disclosure, the model training unit is used to determine whether the iterative training of the initial model of the image matching network is completed through the following steps:

[0267] If the number of iterations reaches a preset threshold or the model loss of the initial model of the picture matching network stops decreasing, it is determined that the iterative training of the initial model of the picture matching network is completed.

[0268] In a specific implementation of the present disclosure, the image matching determination unit 540 is used to:

[0269] A predetermined number of candidate images with the greatest similarity in the candidate image set are determined as the accompanying images for the target copy.

[0270] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0271] Corresponding to the above method embodiment, the present disclosure also provides an electronic device, including:

[0272] processor;

[0273] a memory for storing processor-executable instructions;

[0274] The processor is configured to execute instructions to implement the above-mentioned method for determining the image matching of the copy.

[0275] like Figure 6 , which is a schematic diagram of the composition structure of an electronic device, the electronic device may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through the communication bus 13.

[0276] In the embodiment of the present disclosure, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices.

[0277] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in an embodiment of a method for determining a picture matching a text.

[0278] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present disclosure, the memory 11 at least stores programs for implementing the following functions:

[0279] Get the target copy;

[0280] Extract text features of the target copy;

[0281] Determine the similarity between the image features of each candidate image in the candidate image set and the text features of the target copy;

[0282] According to the similarity, the image matching the target copy is determined in the set of candidate images.

[0283] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function (such as a feature extraction function and a similarity determination function), etc.; the data storage area may store data created during use, such as feature data, similarity data, etc.

[0284] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0285] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0286] Of course, it should be noted that Figure 6The structure shown does not constitute a limitation on the electronic device in the embodiment of the present disclosure. In actual applications, the electronic device may include Figure 6 More or fewer components than shown, or combinations of certain components.

[0287] Corresponding to the above method embodiment, the embodiment of the present disclosure further provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can execute the above-mentioned method for determining the image matching of the copy.

[0288] In addition, it should be noted that: the embodiment of the present disclosure also provides a computer program product or a computer program, which may include computer instructions, and the computer instructions may be stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor can execute the computer instructions so that the electronic device executes the description of the method for determining the illustration of the text in the corresponding embodiment described above, so it will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in the present disclosure, please refer to the description of the method embodiment of the present disclosure.

[0289] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0290] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for determining a picture for a copy, characterized in that: include: Get the target copy; Extracting text features of the target copy; The text features are extracted through the text classification branch of the picture matching network model; Determine respectively the similarity between the image features of each image to be selected in the image set and the text features of the target copy; the image features of each image to be selected are determined by the image classification branch of the image matching network model; the similarity between the image features of each image to be selected and the text features of the target copy is determined by the image-text similarity matching branch of the image matching network model; Determining, according to the similarity, a picture for the target text in the set of images to be selected; The picture matching network model is obtained in the following way: Determine each training data pair in the training data set as the current training data pair in turn, input the training image in the current training data pair into the image classification branch of the pre-built image matching network initial model, and determine the image classification loss corresponding to the training image in the current training data pair based on the output result of the image classification branch and the class label corresponding to the training image in the current training data pair; Input the training text in the current training data pair into the text classification branch of the initial model of the image matching network, and determine the text classification loss corresponding to the training text in the current training data pair based on the output result of the text classification branch and the category label corresponding to the training text in the current training data pair; Determining the modality contrast loss corresponding to the current training data pair through the image-text similarity matching branch of the initial model of the image matching network; Determine a model loss based on an image classification loss corresponding to a training image in the current training data pair, a text classification loss corresponding to a training text in the current training data pair, and a modality contrast loss corresponding to the current training data pair; Based on the model loss, the parameters of the image classification branch, the text classification branch and the image-text similarity matching branch included in the initial model of the picture matching network are adjusted.

2. The method for determining the image matching of a copy according to claim 1, characterized in that: The image set to be selected is obtained by following the steps: Determining the target category to which the target document belongs based on the text features of the target document; The material library is searched for material images belonging to the target category, and the found material images are added as candidate images to the candidate image set.

3. The method for determining the image matching of a copy according to claim 2, characterized in that: The category to which each material image in the material library belongs is determined by the following steps: Extracting image features of each material image in the material library; Based on the image features of each material image, the category to which each material image belongs is determined.

4. The method for determining the image matching of a copy according to claim 1, characterized in that: The extracting text features of the target document includes: Extracting text features of the target copy through the text classification branch in the picture matching network model; Correspondingly, the image features of each image to be selected in the image set to be selected are: extracted by the image classification branch in the image matching network model; Accordingly, the step of respectively determining the similarity between the image feature of each image to be selected in the image set to be selected and the text feature of the target text includes: The similarity between the image features of each candidate image in the candidate image set and the text features of the target text is determined respectively through the image-text similarity matching branch in the image matching network model.

5. The method for determining the image matching of a copy according to claim 4, characterized in that: The image matching network model is obtained by pre-training through the following steps: Obtain a training data set, wherein the training data set includes a plurality of training data pairs, each training data pair includes a training image and a training text, and each training image and each training text has a corresponding category label; Using the training data set and the category labels corresponding to the training images and the training texts in each training data pair, iteratively training the pre-built initial model of the picture matching network includes a text classification branch, an image classification branch and a picture-text similarity matching branch; After the iterative training is completed, the image matching network model is obtained.

6. The method for determining the image matching of a copy according to claim 1, characterized in that: The determining, based on the output result of the image classification branch and the class label corresponding to the training image in the current training data pair, the image classification loss corresponding to the training image in the current training data pair comprises: Extracting image features of the training image in the current training data pair using the image feature extraction module of the image classification branch; Inputting the image features of the training image in the current training data pair into the image classification module of the image classification branch to obtain a first classification probability that the training image in the current training data pair belongs to each category; Based on the first classification probability and the class label corresponding to the training image in the current training data pair, an image classification loss corresponding to the training image in the current training data pair is determined.

7. The method for determining the image matching of a copy according to claim 1, characterized in that: The determining, based on the output result of the text classification branch and the category label corresponding to the training text in the current training data pair, the text classification loss corresponding to the training text in the current training data pair comprises: Extracting text features of the training text in the current training data pair using the text feature extraction module of the text classification branch; Inputting the text features of the training text in the current training data pair into the text classification module of the text classification branch to obtain a second classification probability that the training text in the current training data pair belongs to each category; Based on the second classification probability and the class label corresponding to the training text in the current training data pair, a text classification loss corresponding to the training text in the current training data pair is determined.

8. The method for determining the image matching of a copy according to claim 1, characterized in that: The determining the modality contrast loss corresponding to the current training data pair through the image-text similarity matching branch of the initial model of the image matching network includes: Inputting the image features corresponding to the training image and the text features corresponding to the training text in the current training data pair into the image-text similarity matching branch of the initial model of the image matching network; Obtaining the image-text similarity matrix corresponding to the current training data pair using the similarity calculation module of the image-text similarity matching branch; The loss calculation module of the image-text similarity matching branch is used to perform difference calculation on the image-text similarity matrix and a preset image-text matching matrix to determine the modality contrast loss corresponding to the current training data pair.

9. The method for determining the image matching of a copy according to claim 8, characterized in that: The image-text matching matrix is ​​a diagonal matrix; the loss calculation module using the image-text similarity matching branch performs difference calculation on the image-text similarity matrix and a preset image-text matching matrix to determine the modality contrast loss corresponding to the current training data pair, including: Utilizing the loss calculation module of the image-text similarity matching branch, respectively performing difference calculations on the rows corresponding to the image-text similarity matrix and the preset image-text matching matrix, and obtaining row-based differences based on the difference values ​​of all rows; Calculating the difference between the columns corresponding to the image-text similarity matrix and the image-text matching matrix respectively, and obtaining the column-based difference based on the difference values ​​of all the columns; Based on the row-based difference and the column-based difference, a modality contrast loss corresponding to the current training data pair is determined.

10. The method for determining the image matching of a copy according to claim 5, characterized in that: The following steps are used to determine whether the iterative training of the initial model of the image matching network is completed: If the number of iterations reaches a preset threshold or the model loss of the initial model of the picture matching network stops decreasing, it is determined that the iterative training of the initial model of the picture matching network is completed.

11. The method for determining the image matching of a copy according to any one of claims 1 to 10, characterized in that: Determining the image matching the target text in the set of candidate images according to the similarity includes: A first set number of candidate images with the greatest similarity in the candidate image set are determined as the accompanying images of the target copy.

12. A device for determining a picture for a copy, characterized in that: include: A target copy obtaining unit, used for obtaining a target copy; A text feature extraction unit, used to extract text features of the target document; The text features are extracted through the text classification branch of the picture matching network model; A similarity determination unit, used to respectively determine the similarity between the image features of each image to be selected in the image set and the text features of the target copy; the image features of each image to be selected are determined by the image classification branch of the image matching network model; the similarity between the image features of each image to be selected and the text features of the target copy is determined by the image-text similarity matching branch of the image matching network model; A picture matching determination unit, configured to determine a picture matching the target text in the image set to be selected according to the similarity; Wherein, the picture matching network model is obtained through the model training subunit; The model training subunit is used to: Determine each training data pair in the training data set as the current training data pair in turn, input the training image in the current training data pair into the image classification branch of the pre-built image matching network initial model, and determine the image classification loss corresponding to the training image in the current training data pair based on the output result of the image classification branch and the class label corresponding to the training image in the current training data pair; Input the training text in the current training data pair into the text classification branch of the initial model of the image matching network, and determine the text classification loss corresponding to the training text in the current training data pair based on the output result of the text classification branch and the category label corresponding to the training text in the current training data pair; Determining the modality contrast loss corresponding to the current training data pair through the image-text similarity matching branch of the initial model of the image matching network; Determine a model loss based on an image classification loss corresponding to a training image in the current training data pair, a text classification loss corresponding to a training text in the current training data pair, and a modality contrast loss corresponding to the current training data pair; Based on the model loss, the parameters of the image classification branch, the text classification branch and the image-text similarity matching branch included in the initial model of the picture matching network are adjusted.

13. The device for determining the image matching of a copy according to claim 12, characterized in that: The system further comprises a unit for obtaining a set of images to be selected, which is used to obtain the set of images to be selected by: Determining the target category to which the target document belongs based on the text features of the target document; The material library is searched for material images belonging to the target category, and the found material images are added as candidate images to the candidate image set.

14. The device for determining the image matching of a copy according to claim 13, characterized in that: The invention also includes an image category determination unit, which is used to determine the category to which each material image in the material library belongs through the following steps: Extracting image features of each material image in the material library; Based on the image features of each material image, the category to which each material image belongs is determined.

15. The device for determining the image matching of a copy according to claim 12, characterized in that: The text feature extraction unit is used to: Extracting text features of the target copy through the text classification branch in the picture matching network model; Correspondingly, the image features of each image to be selected in the image set to be selected are: extracted by the image classification branch in the image matching network model; Accordingly, the similarity determination unit is used to: The similarity between the image features of each candidate image in the candidate image set and the text features of the target text is determined respectively through the image-text similarity matching branch in the image matching network model.

16. The device for determining the image matching of a copy according to claim 15, characterized in that: It also includes a model training unit, which is used to pre-train and obtain the image matching network model through the following steps: Obtain a training data set, wherein the training data set includes a plurality of training data pairs, each training data pair includes a training image and a training text, and each training image and each training text has a corresponding category label; Using the training data set and the category labels corresponding to the training images and the training texts in each training data pair, iteratively training the pre-built initial model of the picture matching network includes a text classification branch, an image classification branch and a picture-text similarity matching branch; After the iterative training is completed, the image matching network model is obtained.

17. The device for determining the image matching of a copy according to claim 12, characterized in that: The model training unit is used to: Extracting image features of the training image in the current training data pair using the image feature extraction module of the image classification branch; Inputting the image features of the training image in the current training data pair into the image classification module of the image classification branch to obtain a first classification probability that the training image in the current training data pair belongs to each category; Based on the first classification probability and the class label corresponding to the training image in the current training data pair, an image classification loss corresponding to the training image in the current training data pair is determined.

18. The device for determining the image matching of a copy according to claim 12, characterized in that: The model training unit is used to: Extracting text features of the training text in the current training data pair using the text feature extraction module of the text classification branch; Inputting the text features of the training text in the current training data pair into the text classification module of the text classification branch to obtain a second classification probability that the training text in the current training data pair belongs to each category; Based on the second classification probability and the class label corresponding to the training text in the current training data pair, a text classification loss corresponding to the training text in the current training data pair is determined.

19. The device for determining the image matching of a copy according to claim 12, characterized in that: The model training unit is used to: Inputting the image features corresponding to the training image and the text features corresponding to the training text in the current training data pair into the image-text similarity matching branch of the initial model of the image matching network; Obtaining the image-text similarity matrix corresponding to the current training data pair using the similarity calculation module of the image-text similarity matching branch; The loss calculation module of the image-text similarity matching branch is used to perform difference calculation on the image-text similarity matrix and a preset image-text matching matrix to determine the modality contrast loss corresponding to the current training data pair.

20. The device for determining the image matching of a copy according to claim 19, characterized in that: The image-text matching matrix is ​​a diagonal matrix; the model training unit is used to: Utilizing the loss calculation module of the image-text similarity matching branch, respectively performing difference calculations on the rows corresponding to the image-text similarity matrix and the preset image-text matching matrix, and obtaining row-based differences based on the difference values ​​of all rows; Calculating the difference between the columns corresponding to the image-text similarity matrix and the image-text matching matrix respectively, and obtaining the column-based difference based on the difference values ​​of all the columns; Based on the row-based difference and the column-based difference, a modality contrast loss corresponding to the current training data pair is determined.

21. The device for determining the image matching of a copy according to claim 16, characterized in that: The model training unit is used to determine whether the iterative training of the initial model of the image matching network is completed through the following steps: If the number of iterations reaches a preset threshold or the model loss of the initial model of the picture matching network stops decreasing, it is determined that the iterative training of the initial model of the picture matching network is completed.

22. The device for determining the image matching of a text according to any one of claims 12 to 21, characterized in that: The picture matching determination unit is used to: A first set number of candidate images with the greatest similarity in the candidate image set are determined as the accompanying images of the target copy.

23. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for determining the image matching of a text as described in any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for determining the image matching of a copy as described in any one of claims 1 to 11.

25. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that an electronic device having the processor executes the method for determining the image matching of a copy as described in any one of claims 1 to 11.

Images