Image title text determination method, device, electronic device and storage medium

Abstract

The embodiments of the present disclosure provide a method, device, electronic device and storage medium for determining image title text. The method comprises: obtaining text features and image visual features corresponding to a target image; inputting the text features and image visual features into a target title text extraction model; processing the text features and image visual features based on the target title text extraction model to determine a target image title text corresponding to the target image. The embodiments of the present disclosure make full use of the cross-modal information interaction of image features of different granularity and text features to determine whether an image has an editorial title and accurately extract and sort the editorial titles in the image, thereby improving the quality and relevance of image selection.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of image processing technology, and in particular, to a method, device, electronic device, and storage medium for determining image title text. Background Art

[0002] With the rapid development of the Internet and the widespread use of social media in recent years, people are becoming more and more accustomed to uploading what they see and hear anytime and anywhere, and the proportion of multimodal data in the business is also increasing day by day. In the search business of Dianping, each user-uploaded review or note in the current search interface displays one picture, but the user actually provides more than one picture for the review or note. Therefore, how to select one picture from these pictures to display in the search interface to optimize the user's search experience has become a major direction for the current business.

[0003] In order to select high-quality and valuable pictures, the following two solutions are currently used in the display picture optimization scenario:

[0004] 1. Based on image quality. Currently, most image selection scenarios are based on the evaluation of the image quality itself. After scoring the image quality, the images are selected and displayed after being sorted by the scores.

[0005] 2. Optimization based on image and text understanding. The current image understanding method is based on the multimodal fusion method of image and text. By scoring the similarity between text and images, the image with the highest similarity is selected for display.

[0006] The above two solutions select display images based on the visual aesthetic quality of the image itself and the relevance of the image and text, respectively, and optimize the user's experience when searching by improving the quality and information content of the display images. After investigation, it was found that the click-through rate and positive feedback (such as likes, favorites, etc.) of the content with user-edited titles in the display images are better than those without user-edited titles. How to accurately extract user-edited titles in images is a problem that needs to be solved urgently. Summary of the invention

[0007] The embodiments of the present disclosure provide a method, device, electronic device and storage medium for determining image title text, which are used to fully utilize the cross-modal information interaction of different granularity features of images and text features, determine whether an image has an editorial title, and accurately extract and sort the editorial titles in the image, thereby improving the quality and relevance of image selection.

[0008] According to a first aspect of an embodiment of the present disclosure, a method for determining an image title text is provided, comprising:

[0009] Obtain text features and image visual features corresponding to the target image;

[0010] Inputting the text features and the image visual features into a target title text extraction model;

[0011] The text features and the image visual features are processed based on the target title text extraction model to determine the target image title text corresponding to the target image.

[0012] Optionally, acquiring text features and image visual features corresponding to the target image includes:

[0013] Based on character recognition technology, recognizing image text in the target image;

[0014] Based on the area of ​​the image text in the target image, the target image is cropped to generate a text area image and a non-text area image;

[0015] Based on the pre-trained feature extraction model, feature extraction processing is performed on the text area image and the non-text area image respectively to obtain the text features and the image visual features.

[0016] Optionally, before inputting the text features and the image visual features into the target title text extraction model, the method further includes:

[0017] Acquire a sample image; the sample image is an image containing text, and the sample image corresponds to an initial title text label;

[0018] Obtaining sample text features and sample image visual features corresponding to the sample image;

[0019] Inputting the sample text features and the sample image visual features into a title text extraction model to be trained; the title text extraction model to be trained comprises: a coding layer and a prediction layer;

[0020] Calling the encoding layer to fuse the sample text features and the sample image visual features to generate image fusion features;

[0021] Calling the prediction layer to perform prediction processing on the image fusion features to obtain a predicted title text label;

[0022] Based on the initial title text label and the predicted title text label, a loss value corresponding to the title text extraction model to be trained is calculated;

[0023] When the loss value is within a preset range, the trained title text extraction model to be trained is used as the target title text extraction model.

[0024] Optionally, the prediction layer includes: a first title prediction layer, a second title prediction layer and a text order prediction layer,

[0025] The calling of the prediction layer to perform prediction processing on the image fusion feature to obtain a predicted title text label includes:

[0026] Calling the first title prediction layer to perform prediction processing on the image fusion feature to generate a first prediction label for predicting whether the image contains edited title text;

[0027] Calling the second title prediction layer to perform prediction processing on the image fusion feature to generate a second prediction label for predicting whether the text constitutes a user-edited title;

[0028] Calling the text order prediction layer to perform prediction processing on the image fusion feature to generate a third prediction label for predicting the order of text in the title text edited by the user;

[0029] The predicted title text label is determined according to the first predicted label, the second predicted label and the third predicted label.

[0030] Optionally, the target title text extraction model includes: an encoding layer and a prediction layer,

[0031] The step of processing the text features and the image visual features based on the target title text extraction model to determine the target image title text corresponding to the target image includes:

[0032] Calling the encoding layer to fuse the text features and the image visual features to generate target image fusion features;

[0033] The prediction layer is called to perform title prediction processing on the target image fusion features to obtain a target image title text corresponding to the target image.

[0034] According to a second aspect of an embodiment of the present disclosure, there is provided a device for determining an image title text, comprising:

[0035] An image feature acquisition module is used to acquire text features and image visual features corresponding to the target image;

[0036] An image feature input module, used for inputting the text features and the image visual features into a target title text extraction model;

[0037] The image title determination module is used to process the text features and the image visual features based on the target title text extraction model to determine the target image title text corresponding to the target image.

[0038] Optionally, the image feature acquisition module includes:

[0039] An image text recognition unit, used for recognizing the image text in the target image based on character recognition technology;

[0040] A region image generating unit, configured to crop the target image based on the region of the image text within the target image, to generate a text region image and a non-text region image;

[0041] The image feature acquisition unit is used to perform feature extraction processing on the text area image and the non-text area image respectively based on a pre-trained feature extraction model to obtain the text feature and the image visual feature.

[0042] Optionally, the device further comprises:

[0043] A sample image acquisition module is used to acquire a sample image; the sample image is an image containing text, and the sample image corresponds to an initial title text label;

[0044] A sample image feature acquisition module, used to acquire sample text features and sample image visual features corresponding to the sample image;

[0045] A sample image feature input module is used to input the sample text features and the sample image visual features into a title text extraction model to be trained; the title text extraction model to be trained includes: a coding layer and a prediction layer;

[0046] An image fusion feature generation module, used for calling the encoding layer to perform fusion processing on the sample text feature and the sample image visual feature to generate an image fusion feature;

[0047] A predicted title label acquisition module is used to call the prediction layer to perform prediction processing on the image fusion feature to obtain a predicted title text label;

[0048] A loss value calculation module, used to calculate the loss value corresponding to the title text extraction model to be trained based on the initial title text label and the predicted title text label;

[0049] The target title extraction model acquisition module is used to use the trained title text extraction model to be trained as the target title text extraction model when the loss value is within a preset range.

[0050] Optionally, the prediction layer includes: a first title prediction layer, a second title prediction layer and a text order prediction layer,

[0051] The predicted title tag acquisition module includes:

[0052] A first prediction label generating unit, configured to call the first title prediction layer to perform prediction processing on the image fusion feature, and generate a first prediction label for predicting whether the image contains an edited title text;

[0053] A second prediction label generating unit, used for calling the second title prediction layer to perform prediction processing on the image fusion feature, and generating a second prediction label for predicting whether the text constitutes a user-edited title;

[0054] A third prediction label generating unit, used for calling the text order prediction layer to perform prediction processing on the image fusion feature, and generating a third prediction label for predicting the order of the text in the title text edited by the user;

[0055] The predicted title label determination unit is used to determine the predicted title text label according to the first predicted label, the second predicted label and the third predicted label.

[0056] Optionally, the target title text extraction model includes: an encoding layer and a prediction layer,

[0057] The image title determination module comprises:

[0058] A target fusion feature generating unit, used for calling the coding layer to perform fusion processing on the text feature and the image visual feature to generate a target image fusion feature;

[0059] The target image title acquisition unit is used to call the prediction layer to perform title prediction processing on the target image fusion feature to obtain the target image title text corresponding to the target image.

[0060] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0061] A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-mentioned methods for determining the image title text is implemented.

[0062] According to a fourth aspect of an embodiment of the present disclosure, a readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute any of the above-mentioned methods for determining image title text.

[0063] The embodiments of the present disclosure provide a method, device, electronic device and storage medium for determining image title text, which obtains text features and image visual features corresponding to a target image, inputs the text features and image visual features into a target title text extraction model, processes the text features and image visual features based on the target title text extraction model, and determines the target image title text corresponding to the target image. The embodiments of the present disclosure make full use of the cross-modal information interaction of image features of different granularity and text features to determine whether an image has an editorial title and accurately extract and sort the editorial titles in the image, thereby improving the quality and relevance of image selection. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the description of the embodiments of the present disclosure will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0065] Figure 1 A flowchart of a method for determining image title text provided by an embodiment of the present disclosure;

[0066] Figure 2 A flowchart of another method for determining image title text provided by an embodiment of the present disclosure;

[0067] Figure 3 A schematic diagram of the structure of a device for determining image title text provided by an embodiment of the present disclosure;

[0068] Figure 4 A schematic diagram of the structure of another apparatus for determining image title text provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present disclosure.

[0070] Embodiment 1

[0071] Reference Figure 1 , which shows a flowchart of a method for determining an image title text according to an embodiment of the present disclosure, such as Figure 1 As shown, the image title text determination method may include the following steps:

[0072] Step 101: Obtain text features and image visual features corresponding to a target image.

[0073] The embodiments of the present disclosure can be applied to scenarios in which the title text edited by the user in the image is extracted by utilizing different granularity features of the image and text cross-modal information interaction.

[0074] The target image refers to an image that needs to be edited and text extracted. In this example, the target image can be a dish image, an object image, or the like.

[0075] Text features refer to features formed by text in the target image. In this example, the text features may include: image text recognized by character recognition technology, and features such as a text box and position corresponding to the image text.

[0076] Image visual features refer to the features formed by other area images in the target image except the image text.

[0077] When extracting the image text edited by the user in the target image, the text features and image visual features corresponding to the target image can be obtained. The specific acquisition method will be described in detail in the following embodiment 2, and will not be repeated in this embodiment.

[0078] After the text features and image visual features corresponding to the target image are acquired, step 102 is executed.

[0079] Step 102: Input the text features and the image visual features into a target title text extraction model.

[0080] The target title text extraction model refers to a pre-trained model for extracting text from an image. The training process of the target title text extraction model will be described in detail in the following embodiment 2, and will not be described in detail in this embodiment.

[0081] After obtaining the text features and image visual features corresponding to the target image, the obtained text features and image visual features may be input into the target title text extraction model.

[0082] After the text features and image visual features corresponding to the target image are input into the target title text extraction model, step 103 is performed.

[0083] Step 103: Process the text features and the image visual features based on the target title text extraction model to determine the target image title text corresponding to the target image.

[0084] After the text features and image visual features corresponding to the target image are input into the target title text extraction model, the text features and image visual features may be processed based on the target title text extraction model to determine the target image title text corresponding to the target image.

[0085] The embodiments of the present disclosure make full use of cross-modal information interaction of image features of different granularity and text features to determine whether an image has an editorial title and accurately extract and sort the editorial titles in the image. In the subsequent process, high-quality image selection can be performed based on the extracted title text, which brings certain improvements in image selection quality and relevance.

[0086] The method for determining the image title text provided by the embodiment of the present disclosure obtains the text features and image visual features corresponding to the target image, inputs the text features and image visual features into the target title text extraction model, processes the text features and image visual features based on the target title text extraction model, and determines the target image title text corresponding to the target image. The embodiment of the present disclosure makes full use of the cross-modal information interaction of image features of different granularity and text features to determine whether the image has an editorial title and accurately extracts and sorts the editorial titles in the image, thereby improving the image selection in terms of quality and relevance.

[0087] Embodiment 2

[0088] Reference Figure 2 , shows a flowchart of another method for determining image title text provided by an embodiment of the present disclosure, such as Figure 2 As shown, the image title text determination method may include the following steps:

[0089] Step 201: Acquire a sample image; the sample image is an image containing text, and the sample image corresponds to an initial title text label.

[0090] The embodiments of the present disclosure can be applied to scenarios in which the title text edited by the user in the image is extracted by utilizing different granularity features of the image and text cross-modal information interaction.

[0091] Sample images refer to images used for text extraction model training. In this example, sample images refer to images containing text, and each sample image corresponds to an initial title text label.

[0092] When training a title text extraction model, you can obtain sample images containing text.

[0093] After the sample image is acquired, step 202 is executed.

[0094] Step 202: Obtain sample text features and sample image visual features corresponding to the sample image.

[0095] Sample text features refer to the features formed by the text in the sample image.

[0096] The visual features of the sample image refer to the features formed by the images in other regions of the sample image except the text.

[0097] After the sample image is obtained, the sample text features and the sample image visual features corresponding to the sample image may be obtained.

[0098] In this embodiment, the sample text features and the sample image visual features can be used as input features of the title text extraction model to be trained. Each input feature is formed by five embedded features, namely: image feature, text feature, bounding box position information feature, segment feature and position feature, where:

[0099] Image features: First, map the image features to the same space as the text features to obtain the corresponding image features.

[0100] Text features: All OCR text features are averaged to obtain the total text features of the image. The final text features are composed of the total text features and the OCR text features.

[0101] Bounding box position information feature: For each bounding box recognized by OCR, the information is sorted to obtain a five-dimensional vector, which represents the horizontal coordinate of the center point of the bounding box, the vertical coordinate of the center point, the length, width, and area. All values ​​are between 0 and 1, which is the ratio relative to the whole image. At the same time, the information of the bounding box of the whole image is added. Finally, the bounding box position information encoder E is applied t Map all bounding box information features to the same dimension as other features.

[0102] Segment features: Similar to the VQA task in BERT, the entire image and different OCR regions are defined as different types of segments, thereby separating the entire image and different OCR regions.

[0103] Positional features: Like BERT, the sequence position embedding is added to each input element.

[0104] Among the above five embedded features, text features, bounding box position information features, segment features and position features constitute the sample text features.

[0105] After the sample text features and the sample image visual features corresponding to the sample image are acquired, step 203 is executed.

[0106] Step 203: input the sample text features and the sample image visual features into a title text extraction model to be trained; the title text extraction model to be trained includes: a coding layer and a prediction layer.

[0107] After obtaining the sample text features and sample image visual features corresponding to the sample image, the sample text features and sample image visual features can be input into the title text extraction model to be trained. The title text extraction model to be trained may include: an encoding layer and a prediction layer.

[0108] After the sample feature and the sample image visual feature are input into the title text extraction model to be trained, step 204 is performed.

[0109] Step 204: calling the encoding layer to fuse the sample text features and the sample image visual features to generate image fusion features.

[0110] After the sample feature characteristics and the sample image visual features are input into the title text extraction model to be trained, the encoding layer can be called to fuse the sample text features and the sample image visual features to generate image fusion features.

[0111] In this example, the encoding layer is formed by two layers of Transformer Encoder Layer network. After the sample feature and sample image visual features are input into the title text extraction model to be trained, the two layers of Transformer Encoder Layer network can be called to fuse the input features. The input information is exchanged and learned through the attention mechanism in the Transformer, so that image features and text features, coarse-grained features and fine-grained features can all transmit and communicate information.

[0112] After the encoding layer is called to fuse the sample text features and the sample image visual features to generate image fusion features, step 205 is executed.

[0113] Step 205: Call the prediction layer to perform prediction processing on the image fusion features to obtain a predicted title text label.

[0114] After calling the encoding layer to fuse the sample text features and the sample image visual features to generate the image fusion features, the prediction layer can be called to perform prediction processing on the image fusion features to obtain the predicted title text label.

[0115] In this example, the prediction layer can be divided into three subtasks, using three different prediction networks to predict the results: whether there is a user-edited title in the image, whether the OCR text constitutes a user-edited title, and the order of the OCR text in the user-edited title. The first two are binary classification tasks, and the third is a multi-classification task. All prediction networks are implemented by fully connected layers. The process of generating predicted title text labels can be described in detail in conjunction with the following specific implementation methods.

[0116] In a specific implementation of the embodiment of the present disclosure, the prediction layer includes: a first title prediction layer, a second title prediction layer and a text order prediction layer, and the above step 205 may include:

[0117] Sub-step A1: calling the first title prediction layer to perform prediction processing on the image fusion features to generate a first prediction label for predicting whether the image contains edited title text.

[0118] In an embodiment of the present disclosure, the prediction layers may include: a first title prediction layer, a second title prediction layer, and a text order prediction layer.

[0119] After acquiring the image fusion features, the first title prediction layer can be called to perform prediction processing on the image fusion features to generate a first prediction label for predicting whether the image contains edited title text, that is, the first prediction label can be used to indicate whether the target image contains text edited by the user.

[0120] Sub-step A2: calling the second title prediction layer to perform prediction processing on the image fusion features to generate a second prediction label for predicting whether the text constitutes a user-edited title.

[0121] After acquiring the image fusion features, the second title prediction layer can be called to perform prediction processing on the image fusion features to generate a second prediction label for predicting whether the text constitutes a user-edited title, that is, the second prediction label can be used to indicate whether the text in the target image is a title text edited by the user.

[0122] Sub-step A3: calling the text order prediction layer to perform prediction processing on the image fusion features, and generating a third prediction label for predicting the order of text in the title text edited by the user.

[0123] After acquiring the image fusion features, the text order prediction layer can be called to perform prediction processing on the image fusion features to generate a third prediction label for predicting the order of the text in the user editing the title text, that is, the third prediction label can be used to indicate the order of the text in the extracted target image in the user editing the title text.

[0124] Sub-step A4: Determine the predicted title text label according to the first predicted label, the second predicted label and the third predicted label.

[0125] After obtaining the first prediction label, the second prediction label and the third prediction label through the above steps, the predicted title text label corresponding to the target image can be determined based on the first prediction label, the second prediction label and the third prediction label, that is, the first prediction label, the second prediction label and the third prediction label are collectively used as the predicted title text label.

[0126] After calling the prediction layer to perform prediction processing on the image fusion features to obtain the predicted title text label, step 206 is executed.

[0127] Step 206: Based on the initial title text label and the predicted title text label, calculate the loss value corresponding to the title text extraction model to be trained.

[0128] After calling the prediction layer to predict the image fusion features to obtain the predicted title text label, the loss value corresponding to the title text extraction model to be trained can be calculated based on the initial title text label and the predicted title text label. In this example, since the current scheme divides the task into three subtasks, for these three prediction tasks, this scheme uses the cross-entropy function as the loss function, and uses the label and the prediction result of the corresponding prediction head as the function input to obtain the corresponding loss function value to update the parameters.

[0129] In addition, considering that Task 2 determines whether the OCR text constitutes a user-edited title and Task 3 determines the order of the OCR text in the user-edited title, there is a certain correlation, that is, if Task 2 determines that the OCR text does not constitute a user-edited title, then it does not have a corresponding order in the final edited title, that is, Task 3 should judge it as other. Therefore, in order to strengthen the relevance of its judgment, this task introduces the judgment result of Task 2 into the loss function of Task 3.

[0130] Specifically, at the initial stage of training, Task 3 calculates the cross entropy loss of all OCR multimodal features that constitute the final user edited title (i.e., data with non-zero true labels in Task 3), namely where p i is the true label, q i After the training reaches a certain stage, the judgment result of Task 2 is introduced, that is, data x is randomly selected from each batch of data, and the real label is replaced by the prediction result of Model 2. Then, the loss function of Task 3 is calculated for the data with non-zero labels, that is, where r j is the predicted label of the model in task 2.

[0131] After the loss value of the title text extraction model to be trained is calculated, step 207 is executed.

[0132] Step 207: When the loss value is within a preset range, the trained title text extraction model to be trained is used as the target title text extraction model.

[0133] After calculating the loss value of the title text extraction model to be trained, it can be determined whether the loss value is within a preset range. If the loss value is within the preset range, the trained title text extraction model to be trained can be used as the target title text extraction model.

[0134] When the loss value is outside the preset range, other sample images can be obtained to continue the training process of steps 203 to 206 on the trained model until the loss value is within the preset range.

[0135] Step 208: Acquire text features and image visual features corresponding to the target image.

[0136] The target image refers to an image that needs to be edited and text extracted. In this example, the target image can be a dish image, an object image, or the like.

[0137] Text features refer to features formed by text in the target image. In this example, the text features may include: image text recognized by character recognition technology, and features such as a text box and position corresponding to the image text.

[0138] Image visual features refer to the features formed by other area images in the target image except the image text.

[0139] When extracting the image text edited by the user in the target image, the text features and image visual features corresponding to the target image may be obtained. Specifically, it may be described in detail in combination with the following specific implementation method.

[0140] In a specific implementation of the embodiment of the present disclosure, the above step 208 may include:

[0141] Sub-step B1: Based on character recognition technology, recognize the image text in the target image.

[0142] In this embodiment, after the target image is obtained, a character recognition technology (such as OCR (Optical Character Recognition)) may be used to recognize the image text in the target image.

[0143] After the image text in the target image is recognized based on the character recognition technology, sub-step B2 is performed.

[0144] Sub-step B2: based on the area of ​​the image text in the target image, cropping the target image to generate a text area image and a non-text area image.

[0145] After identifying the image text in the target image, the area of ​​the image text in the target image can be obtained, and the target image can be cropped based on the area to obtain a text area image (i.e., an area image containing text) and a non-text area image (i.e., an area image not containing text).

[0146] After the target image is cropped based on the area of ​​the image text in the target image to generate the text area image and the non-text area image, sub-step B3 is performed.

[0147] Sub-step B3: performing feature extraction processing on the text area image and the non-text area image respectively based on a pre-trained feature extraction model to obtain the text features and the image visual features.

[0148] After cropping the target image based on the area of ​​the image text in the target image to generate a text area image and a non-text area image, feature extraction processing can be performed on the text area image and the non-text area image respectively based on the pre-trained feature extraction model to obtain text features and image visual features.

[0149] After obtaining the text features and image visual features corresponding to the target image, step 209 is executed.

[0150] Step 209: Input the text features and the image visual features into a target title text extraction model.

[0151] The target title text extraction model refers to a pre-trained model for extracting text from an image.

[0152] After obtaining the text features and image visual features corresponding to the target image, the obtained text features and image visual features may be input into the target title text extraction model.

[0153] After the text features and image visual features corresponding to the target image are input into the target title text extraction model, step 210 is performed.

[0154] Step 210: Process the text features and the image visual features based on the target title text extraction model to determine the target image title text corresponding to the target image.

[0155] After the text features and image visual features corresponding to the target image are input into the target title text extraction model, the text features and image visual features may be processed based on the target title text extraction model to determine the target image title text corresponding to the target image.

[0156] The process of determining the target image title text may be described in detail in conjunction with the following specific implementation method.

[0157] In a specific implementation of the embodiment of the present disclosure, the above step 210 may include:

[0158] Sub-step C1: calling the encoding layer to fuse the text features and the image visual features to generate target image fusion features.

[0159] In this embodiment, the target title text extraction model may include: an encoding layer and a prediction layer.

[0160] After obtaining the text features and image visual features corresponding to the target image, the encoding layer can be called to fuse the text features and the image visual features to generate the target image fusion features.

[0161] After generating the target image fusion features, sub-step C2 is performed.

[0162] Sub-step C2: calling the prediction layer to perform title prediction processing on the target image fusion features to obtain the target image title text corresponding to the target image.

[0163] After generating the target image fusion features, the prediction layer can be called to perform title prediction processing on the target image fusion features to obtain the target image title text corresponding to the target image. This process is similar to the process of sub-steps A1 to A2 above, and will not be repeated in this embodiment.

[0164] The embodiments of the present disclosure make full use of cross-modal information interaction of image features of different granularity and text features to determine whether an image has an editorial title and accurately extract and sort the editorial titles in the image, thereby improving the quality and relevance of image selection.

[0165] The method for determining the image title text provided by the embodiment of the present disclosure obtains the text features and image visual features corresponding to the target image, inputs the text features and image visual features into the target title text extraction model, processes the text features and image visual features based on the target title text extraction model, and determines the target image title text corresponding to the target image. The embodiment of the present disclosure makes full use of the cross-modal information interaction of image features of different granularity and text features to determine whether the image has an editorial title and accurately extracts and sorts the editorial titles in the image, thereby improving the image selection in terms of quality and relevance.

[0166] Embodiment 3

[0167] Reference Figure 3 , which shows a schematic diagram of the structure of an apparatus for determining image title text provided by an embodiment of the present disclosure, such as Figure 3 As shown, the image title text determination device 300 may include the following modules:

[0168] An image feature acquisition module 310 is used to acquire text features and image visual features corresponding to a target image;

[0169] An image feature input module 320, used to input the text features and the image visual features into a target title text extraction model;

[0170] The image title determination module 330 is used to process the text features and the image visual features based on the target title text extraction model to determine the target image title text corresponding to the target image.

[0171] The image title text determination device provided by the embodiment of the present disclosure obtains the text features and image visual features corresponding to the target image, inputs the text features and image visual features into the target title text extraction model, processes the text features and image visual features based on the target title text extraction model, and determines the target image title text corresponding to the target image. The embodiment of the present disclosure makes full use of the cross-modal information interaction of image features of different granularity and text features to determine whether the image has an editorial title and accurately extracts and sorts the editorial titles in the image, thereby improving the image selection quality and relevance.

[0172] Embodiment 4

[0173] Reference Figure 4 , which shows a schematic diagram of the structure of another apparatus for determining image title text provided by an embodiment of the present disclosure, such as Figure 4 As shown, the image title text determination device 400 may include the following modules:

[0174] The sample image acquisition module 410 is used to acquire a sample image; the sample image is an image containing text, and the sample image corresponds to an initial title text label;

[0175] A sample image feature acquisition module 420 is used to acquire sample text features and sample image visual features corresponding to the sample image;

[0176] The sample image feature input module 430 is used to input the sample text feature and the sample image visual feature into the title text extraction model to be trained; the title text extraction model to be trained includes: an encoding layer and a prediction layer;

[0177] An image fusion feature generation module 440 is used to call the encoding layer to perform fusion processing on the sample text feature and the sample image visual feature to generate an image fusion feature;

[0178] A predicted title label acquisition module 450 is used to call the prediction layer to perform prediction processing on the image fusion feature to obtain a predicted title text label;

[0179] A loss value calculation module 460, configured to calculate a loss value corresponding to the title text extraction model to be trained based on the initial title text label and the predicted title text label;

[0180] A target title extraction model acquisition module 470 is used to use the trained title text extraction model to be trained as the target title text extraction model when the loss value is within a preset range;

[0181] An image feature acquisition module 480 is used to acquire text features and image visual features corresponding to a target image;

[0182] An image feature input module 490, used to input the text features and the image visual features into a target title text extraction model;

[0183] The image title determination module 4100 is used to process the text features and the image visual features based on the target title text extraction model to determine the target image title text corresponding to the target image.

[0184] Optionally, the image feature acquisition module includes:

[0185] An image text recognition unit, used for recognizing the image text in the target image based on character recognition technology;

[0186] A region image generating unit, configured to crop the target image based on the region of the image text within the target image, to generate a text region image and a non-text region image;

[0187] The image feature acquisition unit is used to perform feature extraction processing on the text area image and the non-text area image respectively based on a pre-trained feature extraction model to obtain the text feature and the image visual feature.

[0188] Optionally, the prediction layer includes: a first title prediction layer, a second title prediction layer and a text order prediction layer,

[0189] The predicted title tag acquisition module includes:

[0190] A first prediction label generating unit, configured to call the first title prediction layer to perform prediction processing on the image fusion feature, and generate a first prediction label for predicting whether the image contains an edited title text;

[0191] A second prediction label generating unit, used for calling the second title prediction layer to perform prediction processing on the image fusion feature, and generating a second prediction label for predicting whether the text constitutes a user-edited title;

[0192] A third prediction label generating unit, used for calling the text order prediction layer to perform prediction processing on the image fusion feature, and generating a third prediction label for predicting the order of the text in the title text edited by the user;

[0193] The predicted title label determination unit is used to determine the predicted title text label according to the first predicted label, the second predicted label and the third predicted label.

[0194] Optionally, the target title text extraction model includes: an encoding layer and a prediction layer,

[0195] The image title determination module comprises:

[0196] A target fusion feature generating unit, used for calling the coding layer to perform fusion processing on the text feature and the image visual feature to generate a target image fusion feature;

[0197] The target image title acquisition unit is used to call the prediction layer to perform title prediction processing on the target image fusion feature to obtain the target image title text corresponding to the target image.

[0198] The image title text determination device provided by the embodiment of the present disclosure obtains the text features and image visual features corresponding to the target image, inputs the text features and image visual features into the target title text extraction model, processes the text features and image visual features based on the target title text extraction model, and determines the target image title text corresponding to the target image. The embodiment of the present disclosure makes full use of the cross-modal information interaction of image features of different granularity and text features to determine whether the image has an editorial title and accurately extracts and sorts the editorial titles in the image, thereby improving the image selection quality and relevance.

[0199] An embodiment of the present disclosure further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image title text determination method of the aforementioned embodiment when executing the program.

[0200] An embodiment of the present disclosure further provides a readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the image title text determination method of the aforementioned embodiment.

[0201] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0202] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems may also be used together with the teachings based thereon. According to the above description, it is apparent that the structure required for constructing such systems is. In addition, the embodiments of the present disclosure are not directed to any particular programming language either. It should be understood that the contents of the embodiments of the present disclosure described herein may be realized using various programming languages, and the description of the specific languages ​​above is intended to disclose the best mode of implementation of the embodiments of the present disclosure.

[0203] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present disclosure can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0204] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present disclosure, the various features of the embodiments of the present disclosure are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed embodiments of the present disclosure require more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, the inventive aspects lie in less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the embodiments of the present disclosure.

[0205] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0206] The various component embodiments of the embodiments of the present disclosure may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components of the dynamic picture generation device according to the embodiments of the present disclosure. The embodiments of the present disclosure may also be implemented as a device or apparatus program for executing part or all of the methods described herein. Such a program implementing an embodiment of the present disclosure may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0207] It should be noted that the above embodiments illustrate rather than limit the embodiments of the present disclosure, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The embodiments of the present disclosure may be implemented by means of hardware including several different elements and by means of appropriately programmed computers. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0208] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0209] The above description is only a preferred embodiment of the embodiments of the present disclosure and is not intended to limit the embodiments of the present disclosure. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the embodiments of the present disclosure should be included in the protection scope of the embodiments of the present disclosure.

[0210] The above is only a specific implementation of the embodiment of the present disclosure, but the protection scope of the embodiment of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed by the embodiment of the present disclosure, which should be included in the protection scope of the embodiment of the present disclosure. Therefore, the protection scope of the embodiment of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for determining image title text, characterized in that: include: Obtain text features and image visual features corresponding to the target image; Inputting the text features and the image visual features into a target title text extraction model; Processing the text features and the image visual features based on the target title text extraction model to determine the target image title text corresponding to the target image; Before inputting the text features and the image visual features into the target title text extraction model, the method further includes: Acquire a sample image; the sample image is an image containing text, and the sample image corresponds to an initial title text label; Obtaining sample text features and sample image visual features corresponding to the sample image; Inputting the sample text features and the sample image visual features into a title text extraction model to be trained; the title text extraction model to be trained comprises: a coding layer and a prediction layer; Calling the encoding layer to fuse the sample text features and the sample image visual features to generate image fusion features; Calling the prediction layer to perform prediction processing on the image fusion feature to obtain a predicted title text label; based on the initial title text label and the predicted title text label, calculating a loss value corresponding to the title text extraction model to be trained; When the loss value is within a preset range, the trained title text extraction model to be trained is used as the target title text extraction model; The prediction layer includes: a first title prediction layer, a second title prediction layer and a text order prediction layer, and calling the prediction layer to perform prediction processing on the image fusion feature to obtain a predicted title text label includes: Calling the first title prediction layer to perform prediction processing on the image fusion feature to generate a first prediction label for predicting whether the image contains edited title text; Calling the second title prediction layer to perform prediction processing on the image fusion feature to generate a second prediction label for predicting whether the text constitutes a user-edited title; Calling the text order prediction layer to perform prediction processing on the image fusion feature to generate a third prediction label for predicting the order of text in the title text edited by the user; The predicted title text label is determined according to the first predicted label, the second predicted label and the third predicted label.

2. The method according to claim 1, characterized in that The step of obtaining text features and image visual features corresponding to the target image includes: Based on character recognition technology, recognizing image text in the target image; Based on the area of ​​the image text in the target image, the target image is cropped to generate a text area image and a non-text area image; Based on the pre-trained feature extraction model, feature extraction processing is performed on the text area image and the non-text area image respectively to obtain the text features and the image visual features.

3. The method according to claim 1, characterized in that The target title text extraction model comprises: a coding layer and a prediction layer, and the processing of the text features and the image visual features based on the target title text extraction model to determine the target image title text corresponding to the target image comprises: calling the coding layer to perform fusion processing on the text features and the image visual features to generate target image fusion features; The prediction layer is called to perform title prediction processing on the target image fusion features to obtain a target image title text corresponding to the target image.

4. A device for determining image title text, characterized in that: include: An image feature acquisition module is used to acquire text features and image visual features corresponding to the target image; An image feature input module, used for inputting the text features and the image visual features into a target title text extraction model; An image title determination module, used for processing the text features and the image visual features based on the target title text extraction model to determine the target image title text corresponding to the target image; A sample image acquisition module is used to acquire a sample image; the sample image is an image containing text, and the sample image corresponds to an initial title text label; A sample image feature acquisition module, used to acquire sample text features and sample image visual features corresponding to the sample image; A sample image feature input module is used to input the sample text features and the sample image visual features into a title text extraction model to be trained; the title text extraction model to be trained includes: a coding layer and a prediction layer; An image fusion feature generation module, used for calling the encoding layer to perform fusion processing on the sample text feature and the sample image visual feature to generate an image fusion feature; A predicted title label acquisition module is used to call the prediction layer to perform prediction processing on the image fusion feature to obtain a predicted title text label; A loss value calculation module, used to calculate the loss value corresponding to the title text extraction model to be trained based on the initial title text label and the predicted title text label; A target title extraction model acquisition module is used to use the trained title text extraction model to be trained as the target title text extraction model when the loss value is within a preset range; The prediction layer includes: a first title prediction layer, a second title prediction layer and a text order prediction layer, and the predicted title tag acquisition module includes: A first prediction label generating unit, configured to call the first title prediction layer to perform prediction processing on the image fusion feature, and generate a first prediction label for predicting whether the image contains an edited title text; A second prediction label generating unit, used for calling the second title prediction layer to perform prediction processing on the image fusion feature, and generating a second prediction label for predicting whether the text constitutes a user-edited title; A third prediction label generating unit is used to call the text order prediction layer to perform prediction processing on the image fusion feature to generate a third prediction label for predicting the order of text in the user editing the title text; a predicted title label determining unit is used to determine the predicted title text label according to the first prediction label, the second prediction label and the third prediction label.

5. The device according to claim 4, characterized in that The image feature acquisition module includes: an image text recognition unit, which is used to recognize the image text in the target image based on the character recognition technology; a region image generation unit, which is used to crop the target image based on the region of the image text in the target image to generate a text region image and a non-text region image; The image feature acquisition unit is used to perform feature extraction processing on the text area image and the non-text area image respectively based on a pre-trained feature extraction model to obtain the text feature and the image visual feature.

6. The device according to claim 4, characterized in that The target title text extraction model includes: a coding layer and a prediction layer, and the image title determination module includes: A target fusion feature generating unit, used for calling the encoding layer to perform fusion processing on the text feature and the image visual feature to generate a target image fusion feature; The target image title acquisition unit is used to call the prediction layer to perform title prediction processing on the target image fusion feature to obtain the target image title text corresponding to the target image.

7. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image title text determination method according to any one of claims 1 to 3 when executing the program.

8. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image title text determination method as described in any one of method claims 1 to 3.

Images