Image semantic understanding method, device, equipment and storage medium

Abstract

Embodiments of the present invention disclose a method, apparatus, device, and storage medium for image semantic understanding. The method comprises: acquiring target image information; inputting the target image information into a pre-trained semantic understanding model to obtain a semantic classification result output by the semantic understanding model, wherein the semantic understanding model is trained based on associated image information and text information; and determining a semantic understanding result based on the semantic classification result. The method provided by the embodiment of the present invention classifies the target image information by directly training a semantic understanding model based on the acquired associated image information and text information, thereby simplifying the amount of annotation required during model training and enabling a semantic understanding model with accurate classification to be trained with minimal annotation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to an image semantic understanding method, apparatus, device, and storage medium. Background Art

[0002] Image semantic understanding is a valuable research area, and it's required in many scenarios, such as image tagging and recommending information using images as input. To understand image semantics, a common approach is to annotate images, such as labeling them as "fashionable" or describing "damaged goods." This annotated textual image data is then used to train an image semantic understanding model to understand the semantics of product images.

[0003] During the implementation of the present invention, the inventors discovered that the prior art has at least the following technical problems: Training an accurate semantic understanding model requires a large amount of annotation, resulting in high annotation complexity. When the amount of annotation is insufficient or the annotation is unreasonable, the accuracy of the trained semantic understanding model is low. Summary of the Invention

[0004] Embodiments of the present invention provide an image semantic understanding method, apparatus, device, and storage medium to improve the accuracy of a semantic understanding model.

[0005] In a first aspect, an embodiment of the present invention provides a method for image semantic understanding, comprising:

[0006] Obtain target image information;

[0007] Input the target image information into a pre-trained semantic understanding model to obtain the semantic classification result output by the semantic understanding model, which is trained based on the associated image information and text information;

[0008] The semantic understanding result is determined based on the semantic classification result.

[0009] In a second aspect, an embodiment of the present invention further provides an image semantic understanding device, comprising:

[0010] A target image acquisition module is used to acquire target image information;

[0011] The model classification module is used to input the target image information into a pre-trained semantic understanding model to obtain the semantic classification results output by the semantic understanding model. The semantic understanding model is trained based on the associated image information and text information;

[0012] The semantic understanding module is used to determine the semantic understanding result based on the semantic classification result.

[0013] In a third aspect, an embodiment of the present invention further provides a computer device, comprising:

[0014] one or more processors;

[0015] a storage device for storing one or more programs;

[0016] When one or more programs are executed by one or more processors, the one or more processors implement the image semantic understanding method provided by any embodiment of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image semantic understanding method provided by any embodiment of the present invention.

[0018] The image semantic understanding method provided by the embodiments of the present invention obtains candidate fence areas for a fence to be generated; obtains target image information; inputs the target image information into a pre-trained semantic understanding model to obtain a semantic classification result output by the semantic understanding model, which is trained based on associated image and text information; and determines a semantic understanding result based on the semantic classification result. By classifying the target image information directly based on the semantic understanding model trained with the acquired associated image and text information, the amount of annotation required during model training is simplified, enabling the training of a semantic understanding model with accurate classification accuracy with minimal annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 1 is a flow chart of an image semantic understanding method provided by an embodiment of the present invention;

[0020] Figure 2 1 is a flow chart of another method for image semantic understanding provided by an embodiment of the present invention;

[0021] Figure 3a 1 is a flow chart of another method for image semantic understanding provided by an embodiment of the present invention;

[0022] Figure 3b This is a schematic diagram of a process for classifying product images provided by an embodiment of the present invention;

[0023] Figure 4 1 is a schematic structural diagram of an image semantic understanding device provided by an embodiment of the present invention;

[0024] Figure 5 It is a structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0026] Figure 1 This is a flow chart of an image semantic understanding method provided by an embodiment of the present invention. This embodiment is applicable to situations when semantic understanding and classification of images are performed, and is particularly applicable to situations when semantic understanding and classification of pictures of objects are performed. The method can be executed by an image semantic understanding device, which can be implemented in software and / or hardware. For example, the image semantic understanding device can be configured in a computer device. Figure 1 As shown, the method includes:

[0027] S110: Acquire target image information.

[0028] In this embodiment, the target image information may be image information that needs to be sentimentally classified. Alternatively, the target image information may be original image information directly obtained from a webpage or application, or may be image information that is processed from the original image information.

[0029] When the original image information contains a lot of data noise, directly using the obtained original image information as the semantic understanding result of the target image information may lead to inaccurate image semantic understanding results. Therefore, to improve the accuracy of image semantic understanding, it is necessary to perform image preprocessing on the original image information to obtain the target image information. Optionally, image preprocessing operations such as data noise removal and image resizing can be performed on the original image information to obtain the target image information. The image resizing can be set based on the trained semantic understanding model, and the optimal size is consistent with the sample image size used during semantic understanding model training.

[0030] S120. Input the target image information into a pre-trained semantic understanding model to obtain a semantic classification result output by the semantic understanding model. The semantic understanding model is trained based on associated image information and text information.

[0031] After acquiring the target image information, the target image information is input into a pre-trained semantic understanding model to obtain a semantic classification result output by the semantic understanding model. Optionally, the semantic classification result can be a semantic label of the target image information, such as positive, neutral, negative, etc.

[0032] In this embodiment, in order to reduce the amount of sample data annotation while ensuring the classification accuracy of the semantic understanding model, training sample data can be generated based on the associated graphic and text information obtained. The associated graphic and text information can be obtained from channels such as web pages and applications.

[0033] For example, the associated image and text information can be obtained from an internet shopping platform. Generally, after purchasing items through an internet shopping platform, users will review the items. The review content may include image information and text information. Optionally, the image information and text information in the same review information can be used as the associated image and text information. After obtaining the associated image and text information, training sample data is generated based on the associated image and text information, and the semantic understanding model is trained using the training sample data to obtain a trained semantic understanding model.

[0034] In one embodiment, the semantic understanding model may include an encoding part and a classification part for semantic understanding of the target image information. Optionally, the semantic understanding model includes an image encoding sub-model and a semantic classification sub-model, and the target image information is input into a pre-trained semantic understanding model to obtain the semantic classification result output by the semantic understanding model, including: inputting the target image information into the image encoding sub-model to obtain the target image semantic vector output by the image encoding sub-model; inputting the target image semantic vector into the semantic classification sub-model to obtain the semantic classification result output by the semantic classification sub-model. Among them, the image encoding sub-module is used to encode the target image information to obtain the target image semantic vector; the semantic classification model is used to determine the semantic classification category of the target graphic information based on the target image semantic vector. Taking the target image information as a product picture as an example, the product picture is input into the image encoding sub-model to obtain the picture semantic coding, and then the picture semantic coding is input into the semantic classification sub-model to predict the sentiment or label classification of the product picture.

[0035] S130: Determine a semantic understanding result based on the semantic classification result.

[0036] In this embodiment, the semantic classification result can be directly used as the semantic understanding result. Assuming the semantic classification result is positive, the positive result is used as the semantic understanding result. This can be understood as the semantic understanding result of the target image information being positive. Assuming the target image information is a product image, it can be understood as the semantic understanding of the product being positive. When semantic understanding is applied in a recommendation scenario, the product corresponding to the target image information with a positive label is a product worth recommending.

[0037] The image semantic understanding method provided by the embodiments of the present invention obtains candidate fence areas for a fence to be generated; obtains target image information; inputs the target image information into a pre-trained semantic understanding model to obtain a semantic classification result output by the semantic understanding model, which is trained based on associated image and text information; and determines a semantic understanding result based on the semantic classification result. By classifying the target image information directly based on the semantic understanding model trained with the acquired associated image and text information, the amount of annotation required during model training is simplified, enabling the training of a semantic understanding model with accurate classification accuracy with minimal annotation.

[0038] Figure 2 This is a flow chart of another semantic understanding method provided by an embodiment of the present invention. Based on the above embodiment, this embodiment further optimizes the training of the semantic understanding model. Figure 2 As shown, the method includes:

[0039] S210 : Generate positive sample data and negative sample data based on the associated graphic and text information in the evaluation information, and generate training sample data based on the positive sample data and the negative sample data.

[0040] In this embodiment, there is no need to manually label all sample information. Instead, training sample data can be generated directly based on the acquired associated image and text information. In general, the associated image and text information is used as positive sample data, and the unrelated image and text information is used as negative sample data. The positive and negative sample data are combined to obtain the training sample data.

[0041] In the context of online shopping platforms, training sample data can be constructed based on the massive amount of text and image review information on these platforms. With the development of mobile internet and communication technologies, reviews on e-commerce platforms now include not only text but also rich images. The coexistence of text and images is a current trend in review modules. The text and images in product reviews are semantically related, with the text often reflecting the semantic information of the image. Based on this, training sample data can be constructed using the text and image data of product reviews to train semantic understanding models. However, reviews may contain unrelated images and text. Therefore, after obtaining the related text and image information, this information can be preprocessed and training sample data generated based on the preprocessed related text and image information.

[0042] In one embodiment, positive sample data and negative sample data are generated based on the associated graphic and text information in the evaluation information, including: obtaining the associated graphic and text information in the evaluation information as sample basic information; performing data normalization processing on the sample basic information to obtain positive sample data; and randomly combining the image information and text information in the evaluation information to obtain negative sample data.

[0043] Optionally, the associated graphic and text information obtained from the evaluation is directly used as the sample basic information, and then the sample basic information is processed to generate training sample data. The processing of the sample basic information can be: data filtering and normalization processing of the obtained information. Among them, data filtering can be understood as filtering out meaningless or junk text content in the information. Normalization processing can be understood as filtering out redundant information in the information, normalizing images, etc. Data filtering can be achieved through a constructed junk text detection tool. Optionally, the junk text detection tool can be constructed based on a text binary classification model. The junk text detection tool can filter out graphic and text samples in which there is no item information in the information. Data normalization processing can also be achieved through a pre-built text simplification model. For example, regarding the original review "First of all, the merchant is very honest. The fruit is very delicious, especially sweet, especially sweet. I say it three times because it's important. The baby and the elderly at home all like to eat it. A very honest merchant. It's great." First, repeated word detection is performed to detect the repeated words "especially sweet, especially sweet, especially sweet." Then, the similarity between the sentences in the original review is calculated, and sentences with repeated semantics are removed. Then, meaningless sentences are filtered out, and finally, the original review is simplified to "The fruit is very delicious and especially sweet." Normalization of images can be performed to remove image noise, unify the image size, and other processing operations. Among them, removing image noise can filter out noise data through at least one filtering method such as median filtering, mean filtering, and Gaussian filtering.

[0044] After normalizing the basic sample information, standardized positive sample data with high correlation between text and images can be obtained. Then, through certain random combinations, the text set in the positive sample data with high correlation between text and images is randomly scattered, and the image set is also scattered. Then, text and image pairs are randomly selected as negative sample data. In order to ensure the balance of difficulty of negative samples, two types of negative samples can be constructed: completely random negative samples and constrained random negative samples. Completely random negative samples can be negative samples obtained by randomly combining the associated text and image information after breaking them up. Constrained random negative samples can be negative samples obtained by randomly combining text and images under set constraints, such as setting a random combination of text and images under a certain category as a negative sample.

[0045] S220: Based on the training sample data, the pre-built semantic understanding model is trained to obtain a trained semantic understanding model.

[0046] After obtaining the training sample data, the semantic understanding model is trained based on the training sample data to obtain a trained semantic understanding model.

[0047] In one embodiment of the present invention, the image encoding submodel in the semantic understanding model is a twin-tower model. The twin-tower model primarily encodes two objects into fixed-length vectors and then calculates the relationship between the two objects by calculating the similarity between the two vectors. In this embodiment, the training sample data includes image information and text information. Therefore, to calculate the encoding of image and text information, an image encoder and a text encoder are required.

[0048] On the basis of the above, the image coding sub-model includes a text encoder and an image encoder. Based on the training sample data, the pre-built semantic understanding model is trained to obtain the trained semantic understanding model, including: for the image-text information pairs in the training sample data, the sample text information of the image-text information pairs is input into the text encoder to obtain the target text coding information output by the text encoder, and the sample image information of the image-text information pairs is input into the image encoder to obtain the target image coding information output by the image encoder; based on the target text coding information and the target image coding information, the target loss value is determined, and the image coding sub-model is trained with the target loss value reaching the convergence condition as the goal to obtain the trained image coding sub-model; based on the trained image coding sub-model, the semantic classification sub-model is trained to obtain the trained semantic classification sub-model.

[0049] In this embodiment, the image encoding submodel and semantic classification submodel in the semantic understanding model can be trained independently or simultaneously. The overall training strategy for the image encoding submodel is as follows: first, the text and image information in the input training sample data are encoded separately to obtain a text semantic vector as the target text encoding information, and an image semantic vector as the target image encoding information. The semantic similarity between the target text encoding information and the target image encoding information is then calculated as the image-text matching score. A target loss value is then calculated based on the image-text matching score. Fixed-size batches of data are input to the image-text matching model for backpropagation training, with model parameters updated until the model converges. The text encoder can utilize a Transformer model, which uses a multi-head self-attention mechanism to encode and calculate text, capturing global semantic information. Furthermore, because it lacks temporal dependencies, it can be fully parallelized and computationally efficient. The image encoder can utilize a Swig-Transformer model. Calculating the target loss value based on the image-text matching score can be accomplished by calculating the cross entropy between the image-text matching score and the ground truth standard.

[0050] When obtaining associated image and text information from evaluation information, the image and text information may contain multiple images. For example, a user may upload multiple images and add a string of text to their evaluation. Therefore, when the image and text information pair contains multiple sample image information, it is necessary to obtain a target image encoding information based on these multiple sample image information.

[0051] In one embodiment, the image-text information pair includes at least two sample image information, and the sample image information of the image-text information pair is input into an image encoder to obtain target image encoding information output by the image encoder, including: superimposing the sample image information to obtain superimposed image information; and inputting the superimposed image information into the image encoder to obtain the target image encoding information output by the image encoder. Optionally, multiple sample image information can be superimposed to obtain a pair of superimposed image information, and the superimposed image information can be input into the image encoder to obtain the target image encoding information.

[0052] In another embodiment, the image-text information pair includes at least two sample image information, and the sample image information of the image-text information pair is input into an image encoder to obtain image coding information output by the image encoder, including: inputting each sample image information into the image encoder separately to obtain image coding information of each sample image information output by the image encoder; summing the image coding information of each sample image information to obtain target image coding information. It is also possible to calculate the image coding information of each sample image information separately through the image encoder, and then sum the image coding information of all sample image information to obtain the target image coding information. Preferably, the image coding information of each sample image information is calculated first, and then the sum is obtained to obtain the target image coding information. Calculating the image coding information of each sample image information separately avoids feature loss caused by image superposition, making the obtained target image coding information more accurate.

[0053] After training the image encoding sub-model, the semantic classification sub-model is trained based on the trained image encoding sub-model. In this embodiment, the text encoder and the image encoder are respectively connected to the semantic classification sub-model, and the semantic classification sub-model is trained based on the trained image encoding sub-model to obtain the trained semantic classification sub-model. This includes: determining the text encoding information of the text information in the training sample data based on the text encoder; constructing semantic classification model training samples based on the text encoding information and the text information labels; and training the pre-constructed semantic classification sub-model based on the semantic classification model training samples to obtain the trained semantic classification sub-model.

[0054] The trained image encoding sub-model can achieve semantic understanding of image and text information. Image sentiment classification can be achieved based on the semantic understanding of the image encoding sub-model. In this embodiment, the training sample data for the semantic classification sub-model can be the positive sample data obtained above. The semantic classification sub-model is a basic classification model used to classify text information into positive, neutral, and negative categories.

[0055] When training the semantic classification sub-model, the text semantic vectors of the text information in the associated image-text information can be used as input to predict the sentiment label of the text information. The text semantic vectors of the text information can be obtained using the text encoder in the image encoding sub-model. The text meaning vectors and text labels output by the text encoder can be used to construct training samples for the semantic classification model to train the semantic classification sub-model. After the semantic classification sub-model is trained using text information, it can be transferred to image information to achieve sentiment classification of the image information.

[0056] S230: Acquire target image information.

[0057] S240. Input the target image information into a pre-trained semantic understanding model to obtain a semantic classification result output by the semantic understanding model, where the semantic understanding model is trained based on associated image information and text information.

[0058] S250: Determine a semantic understanding result based on the semantic classification result.

[0059] The method provided by the embodiment of the present invention generates positive sample data and negative sample data based on the associated graphic and text information in the evaluation information, and generates training sample data based on the positive sample data and the negative sample data; based on the training sample data, a pre-constructed semantic understanding model is trained to obtain a trained semantic understanding model, which simplifies the amount of annotation when constructing the training samples and realizes that a semantic understanding model with accurate classification can be trained with less annotation.

[0060] Based on the above solution, a preferred embodiment is provided, taking a product image as an example to illustrate the semantic understanding of the image. Figure 3a This is a flow chart of another method for image semantic understanding provided by an embodiment of the present invention. Figure 3aAs shown, a dual-tower image-text matching model is first trained using image-text pairs from reviews as input. This dual-tower image-text matching model consists of two parts: one tower is a text encoder, and the other is an image encoder. The text encoder then encodes the sentiment-labeled reviews to obtain text semantic vectors. A sentiment or label classification model is then trained using the text semantic vectors as input (x) and the corresponding sentiment or label as input (y). Since the classification model uses semantic vectors as input, it can be used to classify sentiment and labels based on the semantic vectors of product images.

[0061] Generally speaking, the semantic understanding of an image includes the following steps:

[0062] 1. Build an image-text matching model to achieve semantic understanding of images

[0063] 1) Data preprocessing

[0064] There is a semantic correlation between text and images in product reviews, and the text often reflects the semantic information of the image. Using the image and text data of product reviews as input, we can build an image-text matching model to understand the semantics of product images. We can also preprocess the image and text data of product reviews to construct training sample data for model training.

[0065] The data preprocessing stage is designed to build training data for the image-text matching model. E-commerce platforms have a massive amount of image-text review data, but also contain a lot of noisy data, such as junk reviews and images, or redundant review content. Data can be normalized using the following two methods:

[0066] First, data filtering uses a pre-built spam detection model to filter out meaningless or spam text and images. The spam detection model is a simple binary text classification model used to determine whether a text is spam. Common examples of spam include: storytelling: "Have you ever wondered why Andy inherited a mansion in Wei Wei's neighborhood? Because Wei Wei is Wei Guoqiang's son..."; meaningless word-filling: "Fanfou is posting an advertisement: Brother, brother, brother, brother, brother, brother..."; and advertising: "[Zhonghai Property] Dear owner of Shop XX, Block XX, Zhonghai Jincheng International Garden Commercial Street, Foshan: Zhonghai Jincheng Management Office kindly reminds you..." The spam detection model can filter out images and text samples that contain no product information at all.

[0067] Second, data normalization uses a text reduction model to filter out redundant information in the text, such as repeated and meaningless words, while retaining as much product-related dimensional information as possible. For example, the original review "First of all, the merchant is very trustworthy. The fruit is very delicious and very sweet. I say it three times because it's important. My baby and the elderly at home love it. Super trustworthy merchant. Great!" can be simplified to "The fruit is very delicious and very sweet." Furthermore, image processing techniques are used to normalize the images, including removing noise and unifying the image size.

[0068] Through the above two steps, we can obtain relatively standard positive sample data with high correlation between product images and text. Then, we randomly combine the images and text in the positive sample data to obtain negative sample data.

[0069] 2) Model training

[0070] Through data processing, we obtained training samples for image-text matching: (text, image, label), where the text is a preprocessed evaluation text; the image can be one or multiple images, or even a short video; the label indicates whether the image and text match, with a match value of 1 and a mismatch value of 0.

[0071] The Twin Towers model can be used as an image-text matching model. This model first encodes the input text and image separately. The text encoder uses the Transformer model, while the image encoder uses the Swig-Transformer model. The Swig-Transformer model is an improved version of the Transformer model for images. Unlike text, which is a quasi-one-dimensional vector, images are two-dimensional pixel matrices, resulting in longer sequences and higher computational complexity. Furthermore, the original Transformer model performs poorly for instance segmentation in images and is poor at learning product boundaries in images. The Swin-Transformer model makes the following improvements: 1) It encodes text hierarchically, consisting of four stages; 2) It introduces patch partitioning and merging operations to merge local blocks, reducing the transformer input length and improving model efficiency; 3) It alternates between W-MSA and SW-MSA for feature extraction at each stage. W-MSA divides the image into fixed-size windows and performs multi-head self-attention calculations only within the window, while SW-MSA adjusts the position of the window divisions to perform self-attention, addressing the issue of information loss between different windows in W-MSA. These improvements not only accelerate model computation but also improve the model's semantic understanding of images.

[0072] After obtaining the text semantic vector and the image semantic vector, we calculate their semantic similarity as the image-text matching score. Usually, people choose cosine similarity as a measure of semantic similarity. Here, we choose a radian method as a measure of semantic similarity. The calculation is as follows:

[0073]

[0074] where v a , v b Semantic vectors representing text and images are used. After obtaining the semantic similarity between the text and image, the cross entropy between it and the true label is calculated using the above formula as the loss function for model optimization. Fixed-size batches of data are fed into the image-text matching model for backpropagation training, updating model parameters until the model converges.

[0075] When evaluating multiple images, there are two approaches: one is to stack the images and then input them into the image encoding model to obtain the image semantic vector; the other is to encode the images separately and then sum them to obtain the overall semantic vector. We choose the latter as the output of semantic understanding for multiple images.

[0076] 2. Transfer Learning

[0077] The trained image-text matching model can encode images and text into the same semantic space. This allows the transfer of text classification capabilities to image classification, enabling automated image sentiment analysis and image tagging.

[0078] A text sentiment classification model (also known as a semantic classification submodel) is a basic classification model that categorizes text into positive, neutral, and negative categories. The simplest approach is to train a sentiment classification model based on e-commerce review star ratings. However, due to inaccurate star ratings, the model's accuracy is often low. Therefore, manual annotation can be used to label review texts as positive, neutral, or negative, and then train a sentiment classification model.

[0079] During the training phase, the text sentiment classification model uses the semantic vectors of reviews as input to predict the sentiment label of the text. The semantic vectors are encoded by the text encoder of the image-text matching model. Once the sentiment classification model has been trained with the text semantic vectors as input, it can use the semantic vectors of images as input to identify the sentiment of the image. Finally, the text sentiment model is transferred to product images to perform sentiment analysis of the images. Similarly, a multi-classification model for text labels can be trained using the massive amount of semantic labels and review-related data from the mall to achieve automatic image labeling.

[0080] 3. Model Application

[0081] Through the above process, a sentiment and label classification model for product images was constructed, which consists of an image encoder Swin-Transformer and an image-text classifier. Figure 3b This is a flow chart of a product image classification process provided by an embodiment of the present invention. Figure 3b As shown in the figure, the image of the product to be predicted is used as input. Data preprocessing is first performed, mainly to remove data noise and convert the icon to a fixed size. Then, the image is input into the image encoder Swin-Transformer model to obtain the image semantic vector. Finally, the image semantic vector is input into the classifier to predict the image's sentiment or label classification.

[0082] Based on the massive amount of text and image review information from e-commerce platforms, the embodiments of the present invention achieve automatic semantic understanding of product images using a text-image matching model. Transformer and Swin-Transformer are used to semantically encode text and images, respectively, resulting in better encoding results. The semantic vectors of images can be input as image features into downstream tasks such as product recommendations. Using automatic image semantic understanding technology, product images and text are mapped to the same semantic space. Then, using a classification model for review content, automatic sentiment classification and labeling of images are achieved.

[0083] Figure 4 This is a schematic diagram of the structure of an image semantic understanding device provided by an embodiment of the present invention. The image semantic understanding device can be implemented in software and / or hardware. For example, the image semantic understanding device can be configured in a computer device. Figure 4 As shown, the apparatus includes a target image acquisition module 410, a model classification module 420, and a semantic understanding module 430, wherein:

[0084] The target image acquisition module 410 is used to acquire target image information;

[0085] A model classification module 420 is configured to input target image information into a pre-trained semantic understanding model and obtain a semantic classification result output by the semantic understanding model, where the semantic understanding model is trained based on associated image information and text information;

[0086] The semantic understanding module 430 is used to determine a semantic understanding result based on the semantic classification result.

[0087] The image semantic understanding device provided by the embodiments of the present invention uses a target image acquisition module to obtain a candidate fence area for a fence to be generated; then obtains target image information; a model classification module inputs the target image information into a pre-trained semantic understanding model to obtain a semantic classification result output by the semantic understanding model, which is trained based on associated image information and text information; and the semantic understanding module determines a semantic understanding result based on the semantic classification result. By classifying the target image information directly based on the semantic understanding model trained using the acquired associated image information and text information, the amount of annotation required during model training is simplified, enabling a semantic understanding model with accurate classification to be trained with minimal annotation.

[0088] Optionally, based on the above solution, the semantic understanding model includes an image encoding sub-model and a semantic classification sub-model, and the model classification module 420 is specifically used to:

[0089] Input the target image information into the image coding sub-model to obtain the target image semantic vector output by the image coding sub-model;

[0090] The target image semantic vector is input into the semantic classification sub-model to obtain the semantic classification result output by the semantic classification sub-model.

[0091] Optionally, based on the above solution, the device includes a model training module, including:

[0092] The sample construction submodule is used to generate positive sample data and negative sample data based on the image and text information associated with the evaluation information, and generate training sample data based on the positive sample data and the negative sample data;

[0093] The model training submodule is used to train the pre-built semantic understanding model based on the training sample data to obtain the trained semantic understanding model.

[0094] Optionally, based on the above solution, the sample construction submodule is specifically used to:

[0095] Obtaining the associated graphic and text information in the evaluation information as sample basic information;

[0096] Perform data normalization on the basic information of the sample to obtain positive sample data;

[0097] The image information and text information in the evaluation information are randomly combined to obtain negative sample data.

[0098] Optionally, based on the above solution, the image encoding sub-model includes a text encoder and an image encoder, and the model training sub-module includes:

[0099] A coding information acquisition unit is configured to input sample text information of the image-text information pair in the training sample data into a text encoder to obtain target text coding information output by the text encoder, and input sample image information of the image-text information pair into an image encoder to obtain target image coding information output by the image encoder;

[0100] a loss value calculation unit, configured to determine a target loss value based on target text encoding information and target image encoding information, and train the image encoding sub-model with the target loss value reaching a convergence condition as a goal, thereby obtaining a trained image encoding sub-model;

[0101] The sub-model training unit is used to train the semantic classification sub-model based on the trained image encoding sub-model to obtain the trained semantic classification sub-model.

[0102] Optionally, based on the above solution, the image-text information pair includes at least two sample image information, and the coding information acquisition unit is specifically configured to:

[0103] Superimposing the image information of each sample to obtain superimposed image information;

[0104] The superimposed image information is input into the image encoder to obtain target image coding information output by the image encoder.

[0105] Optionally, based on the above solution, the image-text information pair includes at least two sample image information, and the coding information acquisition unit is specifically configured to:

[0106] Inputting each sample image information into an image encoder respectively to obtain image coding information of each sample image information output by the image encoder;

[0107] The image coding information of each sample image information is summed to obtain the target image coding information.

[0108] Optionally, based on the above solution, the text encoder and the image encoder are respectively connected to the semantic classification sub-model, and the sub-model training unit is specifically used to:

[0109] Determining text encoding information of the text information based on a text encoder for the text information in the training sample data;

[0110] Construct semantic classification model training samples based on text encoding information and text information labels;

[0111] Based on the semantic classification model training samples, the pre-built semantic classification sub-model is trained to obtain a trained semantic classification sub-model.

[0112] The image semantic understanding device provided by the embodiment of the present invention can execute the image semantic understanding method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0113] Figure 5 It is a structural diagram of a computer device provided by an embodiment of the present invention. Figure 5 A block diagram of an exemplary computer device 512 suitable for implementing embodiments of the present invention is shown. Figure 5 The computer device 512 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention.

[0114] like Figure 5 As shown, computer device 512 is implemented as a general-purpose computing device. Components of computer device 512 may include, but are not limited to, one or more processors 516, a system memory 528, and a bus 518 that connects various system components (including system memory 528 and processor 516).

[0115] Bus 518 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a local bus to processor 516, or a bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0116] The computer device 512 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 512, including volatile and non-volatile media, removable and non-removable media.

[0117] System memory 528 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 530 and / or cache memory 532. Computer device 512 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage device 534 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 5 Not shown, often called a "hard drive"). Although Figure 5Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 518 via one or more data media interfaces. Memory 528 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0118] A program / utility 540 having a set (at least one) of program modules 542 may be stored, for example, in memory 528. Such program modules 542 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 542 generally implement the functions and / or methodologies of the embodiments described herein.

[0119] The computer device 512 can also communicate with one or more external devices 514 (e.g., a keyboard, pointing device, display 524, etc.), one or more devices that enable a user to interact with the computer device 512, and / or any device that enables the computer device 512 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication can occur via an input / output (I / O) interface 522. Furthermore, the computer device 512 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 520. As shown, the network adapter 520 communicates with the other modules of the computer device 512 via a bus 518. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with the computer device 512, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0120] The processor 516 executes various functional applications and data processing by running programs stored in the system memory 528, such as implementing the image semantic understanding method provided by an embodiment of the present invention, which includes:

[0121] Obtain target image information;

[0122] Input the target image information into a pre-trained semantic understanding model to obtain the semantic classification result output by the semantic understanding model, which is trained based on the associated image information and text information;

[0123] The semantic understanding result is determined based on the semantic classification result.

[0124] Of course, those skilled in the art will appreciate that the processor may also implement the technical solution of the image semantic understanding method provided by any embodiment of the present invention.

[0125] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the image semantic understanding method provided by an embodiment of the present invention is implemented. The method includes:

[0126] Obtain target image information;

[0127] Input the target image information into a pre-trained semantic understanding model to obtain the semantic classification result output by the semantic understanding model, which is trained based on the associated image information and text information;

[0128] The semantic understanding result is determined based on the semantic classification result.

[0129] Of course, the computer-readable storage medium provided by an embodiment of the present invention stores a computer program which is not limited to the above method operations, but can also perform related operations of the image semantic understanding method provided by any embodiment of the present invention.

[0130] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0131] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0132] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0133] Computer program code for carrying out the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0134] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for image semantic understanding, characterized in that: include: Obtain target image information; Inputting the target image information into a pre-trained semantic understanding model to obtain a semantic classification result output by the semantic understanding model, wherein the semantic understanding model is trained based on associated image information and text information; Determining a semantic understanding result based on the semantic classification result; The semantic understanding model includes an image encoding sub-model and a semantic classification sub-model. Inputting the target image information into the pre-trained semantic understanding model to obtain the semantic classification result output by the semantic understanding model includes: Inputting the target image information into the image coding sub-model to obtain a target image semantic vector output by the image coding sub-model; Inputting the target image semantic vector into the semantic classification sub-model to obtain a semantic classification result output by the semantic classification sub-model; The training of the semantic understanding model includes: Generating positive sample data and negative sample data based on the associated graphic and text information in the evaluation information, and generating training sample data based on the positive sample data and the negative sample data; Based on the training sample data, a pre-built semantic understanding model is trained to obtain a trained semantic understanding model; The image encoding sub-model includes a text encoder and an image encoder. The pre-built semantic understanding model is trained based on the training sample data to obtain a trained semantic understanding model, including: For the image-text information pairs in the training sample data, inputting the sample text information of the image-text information pairs into the text encoder to obtain target text encoding information output by the text encoder, and inputting the sample image information of the image-text information pairs into the image encoder to obtain target image encoding information output by the image encoder; Determining a target loss value based on the target text encoding information and the target image encoding information, and training the image encoding sub-model with the target loss value reaching a convergence condition as a goal, to obtain a trained image encoding sub-model; Training the semantic classification sub-model based on the trained image encoding sub-model to obtain a trained semantic classification sub-model; The text encoder and the image encoder are respectively connected to the semantic classification sub-model, and the semantic classification sub-model is trained based on the trained image encoding sub-model to obtain the trained semantic classification sub-model, including: For the text information in the training sample data, determining text encoding information of the text information based on the text encoder; Constructing a semantic classification model training sample based on the text encoding information and the label of the text information; Based on the semantic classification model training samples, the pre-built semantic classification sub-model is trained to obtain a trained semantic classification sub-model.

2. The method according to claim 1, characterized in that The generating of positive sample data and negative sample data based on the associated graphic and text information in the evaluation information includes: Obtaining the associated graphic and text information in the evaluation information as sample basic information; Performing data normalization processing on the sample basic information to obtain the positive sample data; The image information and text information in the evaluation information are randomly combined to obtain the negative sample data.

3. The method according to claim 1, characterized in that The image-text information pair includes at least two sample image information, and inputting the sample image information of the image-text information pair into the image encoder to obtain target image coding information output by the image encoder includes: superimposing the sample image information to obtain superimposed image information; The superimposed image information is input into the image encoder to obtain target image coding information output by the image encoder.

4. The method according to claim 1, wherein The image-text information pair includes at least two sample image information, and inputting the sample image information of the image-text information pair into the image encoder to obtain the image encoding information output by the image encoder includes: Inputting each of the sample image information into the image encoder respectively to obtain image coding information of each of the sample image information output by the image encoder; The image coding information of each of the sample image information is summed to obtain the target image coding information.

5. An image semantic understanding device, characterized in that: include: A target image acquisition module is used to acquire target image information; A model classification module is used to input the target image information into a pre-trained semantic understanding model to obtain a semantic classification result output by the semantic understanding model, wherein the semantic understanding model is trained based on associated image information and text information; A semantic understanding module, configured to determine a semantic understanding result based on the semantic classification result; The semantic understanding model includes an image encoding sub-model and a semantic classification sub-model. The model classification module is specifically used to: Input the target image information into the image coding sub-model to obtain the target image semantic vector output by the image coding sub-model; Input the target image semantic vector into the semantic classification sub-model to obtain the semantic classification result output by the semantic classification sub-model; The device includes a model training module, including: The sample construction submodule is used to generate positive sample data and negative sample data based on the image and text information associated with the evaluation information, and generate training sample data based on the positive sample data and the negative sample data; The model training submodule is used to train the pre-built semantic understanding model based on the training sample data to obtain the trained semantic understanding model; The image encoding sub-model includes a text encoder and an image encoder, and the model training sub-module includes: A coding information acquisition unit is configured to input sample text information of the image-text information pair in the training sample data into a text encoder to obtain target text coding information output by the text encoder, and input sample image information of the image-text information pair into an image encoder to obtain target image coding information output by the image encoder; a loss value calculation unit, configured to determine a target loss value based on target text encoding information and target image encoding information, and train the image encoding sub-model with the target loss value reaching a convergence condition as a goal, thereby obtaining a trained image encoding sub-model; A sub-model training unit, configured to train the semantic classification sub-model based on the trained image encoding sub-model to obtain a trained semantic classification sub-model; The text encoder and image encoder are connected to the semantic classification sub-model respectively. The sub-model training unit is specifically used to: Determining text encoding information of the text information based on a text encoder for the text information in the training sample data; Construct semantic classification model training samples based on text encoding information and text information labels; Based on the semantic classification model training samples, the pre-built semantic classification sub-model is trained to obtain a trained semantic classification sub-model.

6. A computer device, characterized in that: The device comprises: one or more processors; a storage device for storing one or more programs; When one or more programs are executed by one or more processors, the one or more processors implement the image semantic understanding method as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image semantic understanding method as described in any one of claims 1 to 4 is implemented.

Images