Image-text object processing method and device

Abstract

The invention provides an image-text object processing method and device, and relates to the technical field of artificial intelligence and the technical field of data cleaning.The image-text object processing method comprises the steps that an initial image-text object is obtained, and the initial image-text object comprises an initial image and an initial description text of the initial image; performing image analysis on the initial image from at least one dimension to obtain a confidence coefficient of the initial image, and calculating a first similarity between the initial description text and the initial image; and when the confidence is greater than a first threshold and the first similarity is greater than a second threshold, determining a target image-text object according to the initial image and the initial description text. According to the method, the target image-text object with higher quality can be obtained, so that a better text graph model can be trained.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and data cleaning technology, and in particular to a method and device for processing graphic objects. Background Art

[0002] With the progress of artificial intelligence in various fields, the combination of computer vision and natural language processing has also received widespread attention. For example, the text-based image model, which generates images based on text information, has become a research hotspot due to its advantages such as image, vividness and fluency.

[0003] When training a text-based graph model, a large number of image-text samples are required, namely images and the corresponding image description text. However, due to poor image quality or inaccurate description text, the trained text-based graph model has poor effect. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0004] In view of the problems existing in the prior art, an embodiment of the present invention provides a method and device for processing a graphic object.

[0005] The present invention provides a method for processing graphic objects, comprising:

[0006] Acquire an initial graphic object, wherein the initial graphic object includes an initial image and an initial description text of the initial image;

[0007] Performing image analysis on the initial image from at least one dimension to obtain a confidence level of the initial image, and calculating a first similarity between the initial description text and the initial image;

[0008] When the confidence is greater than a first threshold and the first similarity is greater than a second threshold, a target graphic object is determined according to the initial image and the initial description text.

[0009] According to a method for processing a graphic object provided by the present invention, the calculating of the first similarity between the initial description text and the initial image comprises:

[0010] Splitting the initial description text into at least two initial sub-texts;

[0011] determining at least one target subtext from the at least two initial subtexts;

[0012] Connecting the target sub-texts to obtain a target description text;

[0013] A first similarity between the target description text and the initial image is calculated.

[0014] According to a method for processing a graphic object provided by the present invention, the step of determining at least one target subtext from the at least two initial subtexts comprises:

[0015] Calculating a second similarity between each of the initial subtexts and the initial image;

[0016] A target similarity greater than a third threshold value among the second similarities is determined, and the initial subtext corresponding to the target similarity is determined as a target subtext.

[0017] According to a method for processing a graphic object provided by the present invention, the step of determining a target graphic object according to the initial image and the initial description text comprises:

[0018] The target description text corresponding to the initial image and the initial description text is determined as the target graphic object.

[0019] According to a method for processing graphic objects provided by the present invention, performing image analysis on the initial image from at least one dimension to obtain the confidence of the initial image includes:

[0020] Extract image features from the initial image in at least one dimension to obtain image features of the initial image in each dimension;

[0021] The confidence level of the initial image is determined according to each of the image features.

[0022] According to a method for processing graphic objects provided by the present invention, the at least one dimension includes at least one of an aesthetic dimension, a watermark dimension, and a violation identification dimension;

[0023] Accordingly, the extracting image features from the initial image in at least one dimension to obtain the image features of the initial image in each dimension includes at least one of the following:

[0024] Extracting image features from the initial image from an aesthetic dimension to obtain image features of the initial image in the aesthetic dimension;

[0025] Extracting image features of the initial image from the watermark dimension to obtain image features of the initial image at the watermark dimension;

[0026] Image features of the initial image are extracted from the violation identification dimension to obtain image features of the initial image in the violation identification dimension.

[0027] According to a method for processing graphic objects provided by the present invention, determining the confidence of the initial image according to each of the image features includes:

[0028] Performing feature averaging processing on each of the image features to obtain comprehensive image features of the initial image;

[0029] Probabilistic prediction is performed on the initial image according to the comprehensive image features to obtain the confidence of the initial image.

[0030] According to a method for processing graphic objects provided by the present invention, extracting image features from the initial image from at least one dimension to obtain image features of the initial image in each dimension includes:

[0031] For each dimension, the initial image is input into an image processing model corresponding to the dimension to extract image features, so as to obtain image features of the initial image in each dimension;

[0032] The image processing model is trained based on a first sample image carrying a dimension label, where the dimension label is a probability value of the first sample image under the dimension.

[0033] According to a method for processing a graphic object provided by the present invention, the calculating of the first similarity between the initial description text and the initial image comprises:

[0034] Encoding the initial description text and the initial image respectively to obtain a text embedding vector and an image embedding vector;

[0035] A similarity calculation is performed on the text embedding vector and the image embedding vector to obtain a first similarity between the initial description text and the initial image.

[0036] According to a method for processing graphic objects provided by the present invention, the encoding process is performed on the initial description text and the initial image to obtain a text embedding vector and an image embedding vector, including:

[0037] Inputting the initial description text into the text encoding layer of the pre-trained similarity prediction model for text encoding processing to obtain a text embedding vector, and inputting the initial image into the image encoding layer of the similarity prediction model for image encoding processing to obtain an image embedding vector;

[0038] Accordingly, the performing similarity calculation on the text embedding vector and the image embedding vector to obtain a first similarity between the initial description text and the initial image includes:

[0039] The text embedding vector and the image embedding vector are input into the similarity prediction layer of the similarity prediction model to perform similarity calculation, so as to obtain a first similarity between the initial description text and the initial image.

[0040] According to a method for processing graphic and text objects provided by the present invention, before inputting the initial description text into the text encoding layer of the similarity prediction model for text encoding processing, the method further includes:

[0041] Obtaining an untrained similarity prediction model and a sample image-text object carrying a similarity label, wherein the similarity prediction model includes a text encoding layer, an image encoding layer, and a similarity prediction layer, and the sample image-text object includes a second sample image and a sample description text of the second sample image;

[0042] Inputting the sample description text into the text encoding layer to obtain a predicted text embedding vector, and inputting the second sample image into the image encoding layer to obtain a predicted image embedding vector;

[0043] Inputting the predicted text embedding vector and the predicted image embedding vector into the similarity prediction layer to obtain predicted similarity;

[0044] Calculating a loss value according to the similarity label and the predicted similarity;

[0045] The similarity prediction model is gradient optimized according to the loss value, and the step of obtaining sample graphic objects carrying similarity labels is continued until a preset stop condition is reached to obtain the trained similarity prediction model.

[0046] The present invention also provides a graphic object processing device, comprising:

[0047] An acquisition module is configured to acquire an initial graphic object, wherein the initial graphic object includes an initial image and an initial description text;

[0048] an analysis module, configured to perform image analysis on the initial image from at least one dimension, obtain a confidence level of the initial image, and calculate a first similarity between the initial description text and the initial image;

[0049] The determination module is configured to determine the target graphic object according to the initial image and the initial description text when the confidence is greater than a first threshold and the first similarity is greater than a second threshold.

[0050] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned graphic object processing methods is implemented.

[0051] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for processing graphic objects as described above is implemented.

[0052] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned graphic object processing methods.

[0053] The method and device for processing graphic objects provided by the present invention obtain an initial graphic object, which includes an initial image and an initial description text of the initial image; perform image analysis on the initial image from at least one dimension to obtain the confidence of the initial image, and calculate the first similarity between the initial description text and the initial image; when the confidence is greater than a first threshold and the first similarity is greater than a second threshold, determine the target graphic object according to the initial image and the initial description text. The present invention performs data cleaning from two aspects, at least one dimension representing image quality and description accuracy, and adopts a voting method for cleaning, that is, cleaning based on a threshold, so that a higher quality target graphic object is cleaned to train a better Vincent graph model. Supports the training of multimodal models such as Vincent graphs. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 This is one of the flow charts of the method for processing graphic objects provided by the present invention;

[0056] Figure 2 It is a flowchart of text filtering provided by the present invention;

[0057] Figure 3 It is a schematic diagram of a flow chart of calculating confidence provided by the present invention;

[0058] Figure 4 is a schematic diagram of a flow chart of calculating a first similarity provided by the present invention;

[0059] Figure 5 It is a schematic diagram of the process of training the similarity prediction model provided by the present invention;

[0060] Figure 6 This is the second flow chart of the method for processing graphic objects provided by the present invention;

[0061] Figure 7 It is a structural schematic diagram of the graphic object processing device provided by the present invention;

[0062] Figure 8It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] In order to facilitate a clearer understanding of the embodiments of the present invention, some relevant background knowledge is first introduced as follows.

[0065] With the progress of artificial intelligence in various fields, the combination of computer vision and natural language processing has also received widespread attention. For example, the text-based image model, which generates images based on text information, has become a research hotspot due to its advantages such as image, vividness and fluency.

[0066] Training a text-based graph model requires a large number of image and text samples, i.e., images and corresponding image description texts. However, due to poor image quality or inaccurate description texts, the trained text-based graph model has poor results.

[0067] In order to solve the problem of poor effect of text-generated graph models, the existing technology cleans the text-generated graph data based on the accuracy of the description or description of the text, and uses the cleaned text-generated graph data to train the text-generated graph model. Although the effect of the text-generated graph model is improved to a certain extent, the text-generated graph model trained by the above method still cannot meet the requirements of scenes with higher image requirements.

[0068] Therefore, the present invention provides a method and device for processing graphic objects, by obtaining an initial graphic object, the initial graphic object includes an initial image and an initial description text of the initial image; performing image analysis on the initial image from at least one dimension to obtain the confidence of the initial image, and calculating the first similarity between the initial description text and the initial image; when the confidence is greater than a first threshold value, and the first similarity is greater than a second threshold value, determining a target graphic object according to the initial image and the initial description text. The present invention performs data cleaning from two aspects, at least one dimension representing image quality and description accuracy, and adopts a voting method for cleaning, that is, cleaning based on a threshold, so that a higher quality target graphic object is cleaned to train a better Vincent graph model. Supports the training of multimodal models such as Vincent graphs.

[0069] Combine the following Figure 1-Figure 7 The present invention describes a method and device for processing graphic objects.

[0070] Figure 1 This is one of the flowcharts of the method for processing graphic objects provided by the present invention, see Figure 1 As shown, it includes steps 101 to 103, wherein:

[0071] Step 101: Acquire an initial graphic object, wherein the initial graphic object includes an initial image and an initial description text of the initial image.

[0072] First of all, it should be noted that the execution subject of the present invention can be any electronic device that processes graphic objects, for example, it can be any one of a smart phone, a smart watch, a desktop computer, a laptop computer, etc.

[0073] Specifically, the graphic object is also graphic data. The initial graphic object is a graphic object that has not been processed or data cleaned. The initial image is a picture or image in the initial graphic object. The initial description text is the text data in the initial graphic object used to describe the initial image.

[0074] In actual applications, there are many methods for obtaining the initial graphic object. For example, a user uploads the initial graphic object through the upload page provided by the graphic object processing platform, and accordingly, the execution subject obtains the initial graphic object. For another example, the execution subject receives a graphic object processing instruction or an initial graphic object acquisition instruction, and accordingly, the execution subject obtains the initial graphic object from the storage area pointed to by the graphic object processing instruction or the initial graphic object acquisition instruction. The present invention is not limited to this.

[0075] It should be noted that the present invention uses an initial graphic object as an example to illustrate the graphic object processing method. The present invention can also be used to process multiple initial graphic objects or a graphic object set containing multiple initial graphic objects. For each initial graphic object, the present invention provides a graphic object processing method for processing. That is, the graphic object processing method provided by the present invention can process or clean data for a single initial graphic object, and can also process or clean data for multiple initial graphic objects.

[0076] Step 102: performing image analysis on the initial image from at least one dimension to obtain the confidence of the initial image, and calculating a first similarity between the initial description text and the initial image.

[0077] Specifically, the dimension is the angle from which the image quality is measured, such as clarity and content completeness. Image analysis refers to extracting information from the initial image or understanding the image content. Confidence refers to the image quality of the initial image obtained by integrating various dimensions, or the probability that the initial image is a high-quality image obtained by integrating various dimensions. The higher the confidence, the higher the image quality of the initial image, or the greater the probability that the initial image is a high-quality image. Similarity, that is, the degree of similarity between images and texts, characterizes the accuracy of the description text in describing or describing the image. The higher the similarity, the higher the accuracy. The first similarity is the similarity of the initial description text to the initial image.

[0078] In practical applications, for the confidence, the initial image can be analyzed from each dimension to obtain the image analysis results in each dimension, and then the confidence of the initial image can be calculated according to each image analysis result; each dimension is spliced ​​to obtain a comprehensive dimension, and then the initial image is analyzed from the comprehensive dimension to obtain the confidence of the initial image. It can also be other, and the present invention does not limit this.

[0079] For the first similarity, the initial description text and the initial image may be calculated by inputting into a pre-trained similarity prediction model to obtain the first similarity; or the initial description text and the initial image may be calculated for similarity according to a set calculation strategy to obtain the first similarity.

[0080] It should be noted that the initial image may be analyzed from at least one dimension to obtain the confidence of the initial image, and then the first similarity between the initial description text and the initial image may be calculated; the first similarity between the initial description text and the initial image may be calculated, and then the initial image may be analyzed from at least one dimension to obtain the confidence of the initial image; and the first similarity between the initial description text and the initial image may be calculated while the initial image may be analyzed from at least one dimension to obtain the confidence of the initial image. The present invention does not impose any limitation on this.

[0081] Step 103: When the confidence is greater than a first threshold and the first similarity is greater than a second threshold, determine a target graphic object according to the initial image and the initial description text.

[0082] Specifically, the threshold refers to a pre-set value, which can be set based on human experience or survey data. The first threshold, also known as the confidence threshold, is used to measure whether the image quality meets the standard. The second threshold, also known as the first similarity threshold, is used to measure whether the accuracy of the description text describing the image meets the standard. The first threshold and the second threshold can be the same or different. The target image object refers to the processed image object, which can be used as a training sample to train the text-image model.

[0083] In practical applications, if the confidence level is greater than the first threshold, it means that the image quality of the initial image meets the standard, that is, the initial image is a high-quality image; if the confidence level is less than or equal to the first threshold, it means that the image quality of the initial image does not meet the standard, that is, the initial image is a low-quality image.

[0084] If the first similarity is greater than the second threshold, it means that the initial description text describes the initial image with accuracy that meets the standard; if the first similarity is less than or equal to the second threshold, it means that the initial description text describes the initial image with accuracy that meets the standard.

[0085] When the confidence is greater than the first threshold and the first similarity is greater than the second threshold, it means that the image quality of the initial image meets the standard and the accuracy of the initial description text in describing the initial image meets the standard. Then, the target image and text object can be determined based on the initial image and the initial description text for training the text-image model.

[0086] The initial image-text object can be directly determined as the target image-text object, that is, the initial image is used as the target image, the initial description text is used as the target description text, and the target image and the target description text constitute the target image-text object. Alternatively, the initial image can be used as the target image, and the initial description text is filtered to obtain the target description text, and then the target image and the target description text constitute the target image-text object.

[0087] In addition, when the confidence is less than or equal to the first threshold, and / or the first similarity is less than or equal to the second threshold, it means that the initial image is a low-quality image (the image quality does not meet the standard) and / or the accuracy of the initial description text in describing the initial image does not meet the standard and cannot be used to train the text-image model, and the initial image-text object can be discarded.

[0088] It should be noted that, in order to reduce the amount of data processing, in the case where the confidence is obtained first and then the first similarity is calculated, the confidence is obtained first, and then the confidence is compared with the first threshold. If the confidence is greater than the first threshold, the first similarity is calculated; if the confidence is less than or equal to the first threshold, there is no need to calculate the first similarity, and the initial image and text object can be discarded. In the case where the first similarity is calculated first and then the confidence is obtained, the first similarity is calculated first, and then the first similarity is compared with the second threshold. If the first similarity is greater than the second threshold, the confidence is obtained; if the first similarity is less than or equal to the second threshold, there is no need to obtain the confidence, and the initial image and text object can be discarded.

[0089] The method for processing graphic objects provided by the present invention obtains an initial graphic object, wherein the initial graphic object includes an initial image and an initial description text of the initial image; performs image analysis on the initial image from at least one dimension to obtain the confidence of the initial image, and calculates a first similarity between the initial description text and the initial image; when the confidence is greater than a first threshold value, and the first similarity is greater than a second threshold value, determines a target graphic object according to the initial image and the initial description text. The present invention performs data cleaning from two aspects, at least one dimension representing image quality and description accuracy, and adopts a voting method for cleaning, that is, cleaning based on a threshold, so as to clean a higher quality target graphic object and train a better Vincent graph model. The training of multimodal models such as Vincent graphs is supported.

[0090] In one or more optional embodiments of the present invention, the calculating of the first similarity between the initial description text and the initial image may be specifically implemented as follows:

[0091] Splitting the initial description text into at least two initial sub-texts;

[0092] determining at least one target subtext from the at least two initial subtexts;

[0093] Connecting the target sub-texts to obtain a target description text;

[0094] A first similarity between the target description text and the initial image is calculated.

[0095] Specifically, the initial subtexts are several texts directly obtained after the initial description text is divided. The target subtexts are subtexts selected from the initial subtexts.

[0096] In practical applications, the initial description text is first divided into multiple initial sub-texts, that is, at least two initial sub-texts: the initial description text can be divided into multiple initial sub-texts with different semantics based on semantics; each sentence in the initial description text can be used as an initial sub-text; the initial description text can also be divided into multiple initial sub-texts based on punctuation marks, wherein the punctuation marks can include commas, periods, semicolons, exclamation marks, etc.

[0097] Furthermore, the target subtext is screened out from the obtained multiple initial subtexts. The initial subtexts without actual semantics in each initial subtext can be filtered to obtain the target subtext with actual semantics, such as deleting the initial subtexts that are modal particles or stop words in each initial subtext. The semantic relevance between each initial subtext and the initial description text can also be calculated, and the initial subtext with a semantic relevance greater than a relevance threshold can be used as the target subtext, or the initial subtext corresponding to the largest first N semantic relevances can be used as the target subtext, where N is a positive integer, such as arranging each initial subtext in order of semantic relevance from large to small, and using the first N initial subtexts as target subtexts respectively.

[0098] On the basis of obtaining a target subtext, the target subtext can be used as the target description text, or the punctuation marks of the target subtext can be adjusted first, and the adjusted target subtext can be used as the target description text. On the basis of obtaining multiple target subtexts, the target subtexts can be spliced, such as splicing with punctuation marks or directly splicing, to obtain the target description text. After that, the first similarity between the target description text and the initial image is calculated.

[0099] In this way, the initial description text is divided into a plurality of initial subtexts, and the target subtexts in the plurality of initial subtexts are spliced ​​to obtain the target description text, thereby filtering the initial description text. While ensuring that the text semantics remain unchanged, the data volume of the target description text can be reduced, thereby reducing the data processing volume while ensuring the accuracy of the first similarity, and improving the efficiency of determining the first similarity.

[0100] In one or more optional embodiments of the present invention, the determining of at least one target subtext from the at least two initial subtexts may be specifically implemented as follows:

[0101] Calculating a second similarity between each of the initial subtexts and the initial image;

[0102] A target similarity greater than a third threshold value among the second similarities is determined, and the initial subtext corresponding to the target similarity is determined as a target subtext.

[0103] Specifically, the second similarity refers to the similarity between the initial subtext and the initial image. The third threshold, also known as the second similarity threshold, is used to measure whether the accuracy of the subtext describing the image meets the standard.

[0104] In practical applications, after obtaining each initial subtext, the second similarity between each initial subtext and the initial image is calculated respectively. The calculation process of the second similarity may be the same as or different from the calculation process of the first similarity.

[0105] Then, each second similarity is compared with the third threshold, and the second similarity greater than the third threshold is used as the target similarity, and the initial subtext corresponding to the target similarity is determined as the target subtext. Alternatively, the second similarity corresponding to the largest first M second similarities is used as the target similarity, where M is a positive integer, and the initial subtext corresponding to the target similarity is determined as the target subtext.

[0106] In this way, by determining a target similarity with a higher similarity, the initial sub-text corresponding to the target similarity is determined as the target sub-text, and the target sub-texts are spliced ​​to obtain a target description text, thereby filtering the initial description text, reducing the data volume of the target description text, and thereby reducing the data processing volume, and improving the efficiency of determining the first similarity. At the same time, since the target similarity between the target sub-text and the initial image is relatively high, the target description text formed by splicing the target sub-texts can describe the initial image more accurately than the initial description text, that is, the first similarity is greatly improved.

[0107] It should be noted that if there is no target similarity greater than the third threshold value in each second similarity, it means that the accuracy of each initial subtext describing the image is not up to standard, that is, the target description text that can accurately describe the initial image cannot be filtered out from the initial description text. At this time, it can be assumed that the first similarity is less than the second threshold value, and the initial image and text object can be discarded. In this way, the amount of data processing can be reduced to a certain extent, and the processing efficiency of the image and text object can be improved.

[0108] In one or more optional embodiments of the present invention, the specific implementation process of determining the target graphic object according to the initial image and the initial description text may be as follows:

[0109] The target description text corresponding to the initial image and the initial description text is determined as the target graphic object.

[0110] In practical applications, since the target sub-text has a high target similarity with the initial image, the target description text formed by the target sub-text can describe the initial image more accurately than the initial description text. Therefore, the target description text and the initial image constitute the target image-text object and are used to train the text-image model. This can greatly improve the high quality of the images generated by the text-image model, that is, improve the efficiency of data cleaning.

[0111] For example, see Figure 2 , Figure 2It is a flowchart of text filtering provided by the present invention: the obtained initial description text is divided into multiple sub-initial texts according to punctuation marks (comma, period, semicolon, exclamation mark), that is, the sentence is divided by punctuation marks to obtain multiple sub-initial texts; the second similarity between each sub-initial text and the initial image is calculated, and the sub-initial text with the second similarity greater than 0.5 (third threshold) is screened out as the target sub-text, and all the target sub-texts are spliced ​​together, that is, the target sub-text with the second similarity greater than the third threshold is screened out and spliced ​​to obtain the target description text; the first similarity between the target description text and the initial image is calculated, and it is determined whether the first similarity is greater than 0.5 (second threshold). If the first similarity is greater than 0.5, the target description text and the initial image are retained for training the text-image model, otherwise the initial description text and the initial image are discarded.

[0112] In one or more optional embodiments of the present invention, the image analysis is performed on the initial image from at least one dimension to obtain the confidence of the initial image. The specific implementation process may be as follows:

[0113] Extract image features from the initial image in at least one dimension to obtain image features of the initial image in each dimension;

[0114] The confidence level of the initial image is determined according to each of the image features.

[0115] Specifically, image features refer to the characteristic information in the image that is not easily disturbed by random factors. Image feature extraction refers to the process of processing and analyzing the information in the image to extract the characteristic information.

[0116] In practical applications, image features can be extracted from the initial image in each dimension to obtain image features of the initial image in each dimension. Further, each image feature is concatenated or weighted to obtain the target image feature. Further, probability prediction is performed based on the target image feature to obtain the confidence of the initial image.

[0117] In this way, acquiring image features from multiple dimensions can improve the comprehensiveness of image analysis, and then determine the confidence level based on the image features in each dimension, greatly improving the accuracy and reliability of the confidence level.

[0118] It should be noted that there are many dimensions for evaluating image quality, such as resolution, color depth, image distortion, aesthetics, watermark, violation identification, and size. Since the present invention needs to obtain high-quality target graphic objects to train the text-based image model, and the text-based image model needs to generate images with better image content; therefore, when processing graphic objects, the present invention focuses on image quality assessment at the image content level, such as aesthetic dimensions, watermark dimensions, and violation identification dimensions. Therefore, the initial image can be analyzed from at least one of the aesthetic dimensions, watermark dimensions, and violation identification dimensions to determine the confidence of the initial image at the image content level.

[0119] In one or more optional embodiments of the present invention, the at least one dimension includes at least one of an aesthetic dimension, a watermark dimension, and a violation identification dimension;

[0120] Accordingly, the extracting image features from the initial image in at least one dimension to obtain the image features of the initial image in each dimension includes at least one of the following:

[0121] Extracting image features from the initial image from an aesthetic dimension to obtain image features of the initial image in the aesthetic dimension;

[0122] Extracting image features of the initial image from the watermark dimension to obtain image features of the initial image at the watermark dimension;

[0123] Image features of the initial image are extracted from the violation identification dimension to obtain image features of the initial image in the violation identification dimension.

[0124] Specifically, the aesthetic dimension refers to whether the image conforms to human aesthetics, which can be scored using a model. The watermark dimension refers to whether the image contains a watermark. The violation identification dimension refers to whether the image content violates the law, that is, the image safety dimension, including the pornography identification dimension and the violence identification dimension. Among them, the pornography identification dimension refers to whether the image content contains pornographic content, and the violence identification dimension refers to whether the image content contains violent content.

[0125] In practical applications, the at least one dimension may be at least one of an aesthetic dimension, a watermark dimension, and a violation identification dimension. Therefore, when extracting image features, it is necessary to extract image features from at least one of the aesthetic dimension, the watermark dimension, and the violation identification dimension.

[0126] For the aesthetic dimension, image features of the initial image can be extracted from the aesthetic dimension to obtain aesthetic image features, that is, image features of the initial image under the aesthetic dimension or image features of the initial image in the aesthetic dimension.

[0127] For the watermark dimension, image features of the initial image can be extracted from the watermark dimension to obtain watermark image features, that is, image features of the initial image under the watermark dimension or image features of the initial image in the watermark dimension.

[0128] For the violation identification dimension, image features of the initial image can be extracted from the violation identification dimension to obtain violation identification image features, that is, image features of the initial image under the violation identification dimension.

[0129] In this way, by extracting image features from at least one of the aesthetic dimension, watermark dimension and violation identification dimension, the confidence of at least one of the fused aesthetic dimension, watermark dimension and violation identification dimension can be obtained, thereby improving the reliability and comprehensiveness of the confidence.

[0130] It should be noted that the image features extracted in different dimensions are different: aesthetic image features represent the features corresponding to the ornamental value and / or beauty of the initial image; watermark image features represent whether the initial image has a watermark and / or the features corresponding to the existing watermark; violation identification image features represent whether the initial image has illegal content and / or the features corresponding to the existing illegal content.

[0131] In one or more optional embodiments of the present invention, the confidence of the initial image is determined according to each of the image features, and the specific implementation process may be as follows:

[0132] Performing feature averaging processing on each of the image features to obtain comprehensive image features of the initial image;

[0133] Probabilistic prediction is performed on the initial image according to the comprehensive image features to obtain the confidence of the initial image.

[0134] Specifically, the comprehensive image feature is the target image feature obtained by averaging the features of each image.

[0135] In practical applications, when there is only one dimension, the image features of the initial image under this dimension can be directly used as the comprehensive image features: for example, if the dimension is an aesthetic dimension, the aesthetic image features can be used as the comprehensive image features; for example, if the dimension is a watermark dimension, the watermark image features can be used as the comprehensive image features; for example, if the dimension is a violation identification dimension, the violation identification image features can be used as the comprehensive image features.

[0136] When there are at least two dimensions, the image features of the initial image in each dimension can be directly averaged: the image features are added and then divided by the number of dimensions, that is, feature average processing is performed to obtain a comprehensive image feature. For example, if at least one dimension includes an aesthetic dimension, a watermark dimension, and a violation identification dimension, the aesthetic image feature, the watermark image feature, and the violation identification image feature are averaged to obtain a comprehensive image feature.

[0137] On the basis of obtaining the comprehensive image features, further substituting the comprehensive image features into the sigmoid function, the probability of including the three information of violation identification, aesthetics, and watermark detection can be obtained, that is, the confidence of the initial image.

[0138] In this way, not only can we obtain the confidence that contains at least one dimension of information, but also improve the reliability of the confidence. When there are multiple dimensions, the confidence is obtained by feature averaging. Compared with determining the confidence based on each image feature separately, it avoids the phenomenon that the image quality of the initial image is slightly deviated in a certain dimension, and the image quality is very high in other dimensions, and the initial image and text objects will be discarded, which further ensures the efficiency of data cleaning, thereby providing more training samples for the text-based graph model. That is, if the confidence comparison conditions of the three dimensions are independent and strictly controlled at the same time, there will be fewer initial images that meet the requirements. Therefore, the three are feature averaged, so that some high-quality data can be retained. For example, an initial image is not an illegal image, it is extremely beautiful, and although there will be some watermarks, it can still be saved.

[0139] In one or more optional embodiments of the present invention, the image features of the initial image are extracted from at least one dimension to obtain the image features of the initial image in each dimension. The specific implementation process may be as follows:

[0140] For each dimension, the initial image is input into an image processing model corresponding to the dimension to extract image features, so as to obtain image features of the initial image in each dimension;

[0141] The image processing model is trained based on a first sample image carrying a dimension label, where the dimension label is a probability value of the first sample image under the dimension.

[0142] Specifically, the image processing model refers to a pre-trained model for extracting image features or evaluating images. The sample image refers to an image used to train the model. The first sample image refers to an image used to train the image processing model. The dimension label refers to the true confidence or probability value of the first sample image in the dimension.

[0143] In practical applications, image processing models can be used to extract image features, and each dimension has a dedicated image processing model. For example, the image processing model corresponding to the aesthetic dimension is the aesthetic model, the image processing model corresponding to the watermark dimension is the watermark detection model, and the image processing model corresponding to the violation identification dimension is the violation identification model.

[0144] Exemplarily, the initial image is input into the aesthetic model to extract image features, and the aesthetic image features of the initial image are obtained; the initial image is input into the watermark detection model to extract image features, and the watermark image features of the initial image are obtained; the initial image is input into the violation identification model to extract image features, and the violation identification image features of the initial image are obtained.

[0145] It should be noted that the training process of the image processing models corresponding to each dimension may be the same or different, and the image processing models corresponding to each dimension may be obtained by training the same model or different models.

[0146] Preferably, the image processing model corresponding to each dimension is the same model, which is trained using sample images with different labels. That is, before inputting the initial image into the image processing model corresponding to the dimension for image feature extraction, it also includes:

[0147] Obtain an untrained image processing model, and obtain a first sample image corresponding to the dimension, where the first sample image carries a dimension label under the dimension;

[0148] Inputting the first sample image into the feature extraction layer of the image processing model to obtain predicted image features;

[0149] Inputting the predicted image features into the probability prediction layer of the image processing model to obtain the predicted probability;

[0150] Calculate the loss value according to the predicted probability and the dimension label;

[0151] The image processing model is gradient optimized according to the loss value, and the step of obtaining the first sample image corresponding to the dimension is continued until a preset stopping condition is reached, thereby obtaining the trained image processing model corresponding to the dimension.

[0152] Specifically, an untrained image processing model refers to an image processing model that has not been guided and trained for a certain dimension. The image processing model can be a visual transformer, wherein the visual transformer is a model that introduces the Transformer into the image field. The feature extraction layer refers to a processing unit in the image processing model used to extract image features. The probability prediction layer refers to a function in the image processing model used to determine the probability of an image in this dimension, which can be a sigmoid function. The predicted probability is the probability of the image predicted by the image processing model in this dimension. The preset stopping condition can be that the loss value is less than the loss threshold, the number of iterations reaches the number threshold, the rate of change of the loss value is lower than the rate of change threshold, or a combination of the above.

[0153] In practical applications, the violation identification model, aesthetic model and watermark detection model have the same structure, all of which are trained based on visual transformers, and the dimension (also called dimension) of the image features output by the feature output layer is Nx1, where N is the number of data samples captured in one training (Batch Size).

[0154] For the aesthetic dimension: obtain a visual transformer and obtain a first sample image carrying an aesthetic dimension label; input the first sample image into the feature extraction layer of the visual transformer to obtain predicted aesthetic image features; input the predicted aesthetic image features into the probability prediction layer of the visual transformer to obtain the aesthetic prediction probability that the first sample image has aesthetics; calculate the loss value based on the aesthetic prediction probability and the aesthetic dimension label; perform gradient optimization on the visual transformer based on the loss value, continue to obtain the first sample image carrying the aesthetic dimension label, and so on, train and optimize the visual transformer until the preset stop condition is reached to obtain the aesthetic model.

[0155] For the watermark dimension: obtain a visual transformer and obtain a first sample image carrying a watermark dimension label; input the first sample image into the feature extraction layer of the visual transformer to obtain predicted watermark image features; input the predicted watermark image features into the probability prediction layer of the visual transformer to obtain the watermark prediction probability that there is no watermark in the first sample image; calculate the loss value based on the watermark prediction probability and the watermark dimension label; perform gradient optimization on the visual transformer based on the loss value, continue to obtain the first sample image carrying the watermark dimension label, and so on, train and optimize the visual transformer until the preset stop condition is reached to obtain a watermark detection model.

[0156] For the violation identification dimension: obtain a visual transformer and obtain a first sample image carrying a violation identification dimension label; input the first sample image into the feature extraction layer of the visual transformer to obtain predicted violation identification image features; input the predicted violation identification image features into the probability prediction layer of the visual transformer to obtain a violation identification prediction probability that the first sample image is not a violation image; calculate a loss value based on the violation identification prediction probability and the violation identification dimension label; perform gradient optimization on the visual transformer based on the loss value, continue to obtain the first sample image carrying the violation identification dimension label, and so on, train and optimize the visual transformer until a preset stop condition is reached to obtain a violation identification model.

[0157] In this way, the training efficiency and robustness of each image processing model can be improved.

[0158] It should be noted that the use process of the violation identification model, aesthetic model and watermark detection model is as follows: a single image is extracted through the feature extraction layers of the violation identification model, aesthetic model and watermark detection model respectively. The dimension of the image features output by each feature extraction layer is 1x1. After the image features pass through the probability prediction layer (sigmoid function), they are a number between 0 and 1, i.e., the probability value. Among them, the violation identification model represents the probability that the image is a violation image (1 is a normal image, 0 is a violation image), the aesthetic model represents the probability that the image is beautiful (having aesthetics) (1 is a high-quality beautiful image, 0 is a low-quality image), and the watermark detection model represents the probability that the image does not contain a watermark (1 is no watermark, 0 is a watermark).

[0159] If the violation identification model, aesthetic model and watermark detection model are used, the initial image is analyzed from the violation identification dimension, aesthetic dimension and watermark dimension to obtain the confidence of the initial image, see Figure 3 The process is as follows: the initial image is input into the feature extraction layers of the violation identification model, aesthetic model and watermark detection model respectively to obtain the violation identification image features, aesthetic image features and watermark image features; the violation identification image features, aesthetic image features and watermark image features are averaged, that is, the features output by the feature extraction layers of the three models are averaged to obtain the comprehensive image features; the comprehensive image features are input into the sigmoid function to obtain the probability (confidence) containing the information of violation identification, aesthetics and watermark detection, that is, the confidence is output through sigmoid. This confidence will be considered more comprehensively and used together with the subsequent similarity as a condition for judging whether the initial image is retained.

[0160] In one or more optional embodiments of the present specification, the calculating of the first similarity between the initial description text and the initial image may be specifically implemented as follows:

[0161] Encoding the initial description text and the initial image respectively to obtain a text embedding vector and an image embedding vector;

[0162] A similarity calculation is performed on the text embedding vector and the image embedding vector to obtain a first similarity between the initial description text and the initial image.

[0163] Specifically, the text embedding vector refers to the embedding vector corresponding to the initial description text. The image embedding vector refers to the embedding vector corresponding to the initial image.

[0164] In practical applications, the initial description text and the initial image can be encoded respectively to obtain a text embedding vector and an image embedding vector. Further, the text embedding vector and the image embedding vector are similarly calculated to obtain a first similarity. In this way, the initial description text and the initial image are mapped from different data types to the same data type and then similarity is calculated, which can improve the accuracy, speed and reliability of the similarity.

[0165] In one or more optional embodiments of the present specification, the encoding process is performed on the initial description text and the initial image to obtain a text embedding vector and an image embedding vector, and the specific implementation process may be as follows:

[0166] Inputting the initial description text into the text encoding layer of the pre-trained similarity prediction model for text encoding processing to obtain a text embedding vector, and inputting the initial image into the image encoding layer of the similarity prediction model for image encoding processing to obtain an image embedding vector;

[0167] Accordingly, the performing similarity calculation on the text embedding vector and the image embedding vector to obtain a first similarity between the initial description text and the initial image includes:

[0168] The text embedding vector and the image embedding vector are input into the similarity prediction layer of the similarity prediction model to perform similarity calculation, so as to obtain a first similarity between the initial description text and the initial image.

[0169] Specifically, the similarity prediction model refers to a model used to calculate the similarity between images and texts, such as a comparative language image pre-training model. The text encoding layer, i.e., the text encoder, refers to the unit that encodes text in the similarity prediction model, and can be a Transformer-based Bert model; the image encoding layer, i.e., the image encoder, refers to the unit that encodes images in the similarity prediction model, and can be a visual transformer. The similarity prediction layer refers to the unit that calculates the similarity of images in the similarity prediction model.

[0170] In practical applications, the text encoding layer of the similarity prediction model can be used to perform text encoding on the initial description text to obtain a text embedding vector, and the image encoding layer of the similarity prediction model can be used to perform image encoding on the initial image to obtain an image embedding vector. Then, the similarity prediction layer of the similarity prediction model performs similarity calculation on the text embedding vector and the image embedding vector to obtain a first similarity. In addition, there are multiple algorithms for calculating similarity. The similarity prediction layer can perform dot product calculation on the text embedding vector and the image embedding vector to obtain a first similarity, or perform cosine distance calculation on the text embedding vector and the image embedding vector to obtain a first similarity, or other algorithms, which are not limited in the present invention.

[0171] For example, see Figure 4 , Figure 4 It is a flow chart of calculating the first similarity provided by the present invention: the similarity prediction model includes a text encoder (text encoding layer), an image encoder (text encoding layer) and a similarity prediction layer, wherein the text encoder is based on the Bert model of Transformer, and the image encoder is a visual transformer. After a large amount of data training by contrast learning, the text encoder and the image encoder can encode the initial image, and the image encoder encodes the initial description text, and maps the text embedding (text embedding vector) and the image embedding (image embedding vector) to the same space, and then the cosine distance between the text embedding and the image embedding of the similarity prediction layer, that is, the cosine distance is calculated to obtain the first similarity between the initial image and the initial description text, which is used to determine whether the initial image and the initial description text match.

[0172] In one or more optional embodiments of the present specification, before using the similarity prediction model, it is necessary to train the similarity prediction model. That is, before inputting the initial description text into the text encoding layer of the similarity prediction model for text encoding processing, it also includes:

[0173] Obtaining an untrained similarity prediction model and a sample image-text object carrying a similarity label, wherein the similarity prediction model includes a text encoding layer, an image encoding layer, and a similarity prediction layer, and the sample image-text object includes a second sample image and a sample description text of the second sample image;

[0174] Inputting the sample description text into the text encoding layer to obtain a predicted text embedding vector, and inputting the second sample image into the image encoding layer to obtain a predicted image embedding vector;

[0175] Inputting the predicted text embedding vector and the predicted image embedding vector into the similarity prediction layer to obtain predicted similarity;

[0176] Calculating a loss value according to the similarity label and the predicted similarity;

[0177] The similarity prediction model is gradient optimized according to the loss value, and the step of obtaining sample graphic objects carrying similarity labels is continued until a preset stop condition is reached to obtain the trained similarity prediction model.

[0178] Specifically, the second sample image refers to a sample image used to train a similarity prediction model. The sample description text refers to the text describing the second sample image. The predicted similarity is the similarity between the second sample image and the sample description text predicted by the similarity prediction model. The preset stop condition can be that the loss value is less than the loss threshold, the number of iterations reaches the number threshold, the rate of change of the loss value is lower than the rate of change threshold, or a combination of the foregoing. The similarity label refers to the true similarity between the second sample image and the sample description text.

[0179] In practical applications, first obtain an untrained similarity prediction model and obtain a sample image object with a similarity label. Then the text encoding layer encodes the sample description text to obtain a predicted text embedding vector, and the image encoding layer encodes the second sample image to obtain a predicted image embedding vector. Furthermore, the similarity prediction layer performs similarity calculations on the predicted text embedding vector and the predicted image embedding vector, such as dot product calculations or cosine distance calculations, to obtain predicted similarity. Afterwards, the similarity label and the predicted similarity are subjected to loss calculations, such as binary cross entropy loss calculations, mean square error loss calculations, etc., to obtain loss values. The similarity prediction model is gradient optimized according to the loss value, and sample image objects are continuously obtained. By analogy, the similarity prediction model is trained and optimized until the preset stop condition is reached to obtain a trained similarity prediction model.

[0180] For example, see Figure 5 , Figure 5It is a flow chart of the training similarity prediction model provided by the present invention: the batch size of the untrained similarity prediction model is 1, the sample description text is encoded into a text embedding (predicted text embedding vector) through the Bert model (text encoder), and the dimension is 1x256; the second sample image is encoded into an image embedding (predicted image embedding vector) through a visual transformer (image encoding layer), and the dimension is 1x256; the text embedding and the image embedding are dot-producted to obtain a 1x1 Logit, i.e., the predicted similarity, and the Logit and the similarity label are subjected to binary cross entropy loss calculation to obtain Loss (loss value), and the similarity prediction model is gradient optimized. Two methods are used during training. When the image and text match, the Label (similarity label) is 1, and when the image and text do not match, the label is 0. In this way, after a large amount of training, the similarity prediction model can calculate the first similarity between the initial image and the description text.

[0181] Combine the following Figure 6 The method for processing graphic objects provided by the present invention is further described. Figure 6 This is the second flow chart of the method for processing graphic objects provided by the present invention.

[0182] 1) Set up three-dimensional image processing models: violation identification model, aesthetic model and watermark detection model. These three image processing models have the same structure and are all trained based on visual transformers. The dimension (also called dimension) of the image features output by the feature output layer is Nx1, where N is the number of data samples captured in one training (BatchSize). A single image is extracted by the feature extraction layers of the violation identification model, aesthetic model and watermark detection model respectively. The dimension of the image features output by each feature extraction layer is 1x1. After the image features pass through the probability prediction layer (sigmoid function), they are a number between 0 and 1, i.e., the probability value. Among them, the violation identification model represents the probability that the image is a violation image (1 is a normal image, 0 is a violation image), the aesthetic model represents the probability that the image is beautiful (having aesthetics) (1 is a high-quality beautiful image, 0 is a low-quality image), and the watermark detection model represents the probability that the image does not contain a watermark (1 is no watermark, 0 is a watermark).

[0183] The initial image is input into the feature extraction layers of the violation identification model, aesthetic model and watermark detection model respectively to obtain the violation identification image features, aesthetic image features and watermark image features; the violation identification image features, aesthetic image features and watermark image features are processed by feature averaging, that is, the features output by the feature extraction layers of the three models are averaged to obtain the comprehensive image features; the comprehensive image features are input into the sigmoid function to obtain the probability of including the information of violation identification, aesthetics and watermark detection, that is, the confidence. This confidence will be considered more comprehensively and used together with the subsequent similarity as a condition for judging whether the initial image is retained. If the confidence comparison conditions of the three dimensions are independent and strictly controlled at the same time, there will be fewer initial images that meet the requirements. Therefore, the three are processed by feature averaging, so that some high-quality data can be retained. For example, an initial image is not a violation image and is extremely beautiful. Although there will be some watermarks, it can still be saved.

[0184] 2) The similarity prediction model includes a text encoder (text encoding layer), an image encoder (image encoding layer) and a similarity prediction layer, wherein the text encoder is based on the Bert model of Transformer, and the image encoder is a visual transformer. After the text encoder and the image encoder are trained on a large amount of data by contrastive learning, the image encoder can encode the initial image, and the image encoder encodes the initial description text, mapping the text embedding and the image embedding to the same space, and then the similarity prediction layer calculates the cosine distance between the text embedding and the image embedding to obtain the first similarity between the initial image and the target description text, which is used to determine whether the initial image and the target description text match. If the first similarity is greater than the second threshold (such as 0.5), it is determined that the initial image and the target description text match, otherwise they do not match.

[0185] The similarity prediction training process is as follows: the batch size of the untrained similarity prediction model is 1, the sample description text is encoded into a text embedding through the Bert model, and the dimension is 1x256; the second sample image is encoded into an image embedding through the visual transformer, and the dimension is 1x256; the text embedding and the image embedding are dot-producted to obtain a 1x1 Logit, i.e., the predicted similarity, and the Logit and the similarity label are subjected to binary cross entropy loss calculation to obtain the Loss (loss value), and the similarity prediction model is gradient optimized. Two methods are used during training. When the image and text match, the Label (similarity label) is 1, and when the image and text do not match, the label is 0. In this way, after a lot of training, the similarity prediction model can calculate the first similarity between the initial image and the description text.

[0186] Perform text filtering: divide the obtained initial description text into multiple sub-initial texts according to punctuation marks (comma, period, semicolon, exclamation mark); calculate the second similarity between each sub-initial text and the initial image through the similarity prediction model, screen out the sub-initial text with a second similarity greater than 0.5 (third threshold) as the target sub-text, and splice all the target sub-texts together to obtain the target description text; the similarity prediction model calculates the first similarity between the target description text and the initial image. If the first similarity is greater than 0.5 (second threshold), the target description text and the initial image are retained for training the text-image model, otherwise the initial description text and the initial image are discarded.

[0187] 3) Combining the results of 1) and 2), determine whether the confidence is greater than the first threshold and whether the first similarity is greater than the second threshold. If both of them are greater than the corresponding thresholds, that is, the confidence is greater than the first threshold and the first similarity is greater than the second threshold, retain the target description text and the initial image for training the text-image model. Otherwise, filter out the initial description text, the target description text and the initial image.

[0188] In this way, data cleaning is performed from four dimensions: aesthetics, watermark, violation identification, and description accuracy. Voting is used for cleaning, that is, cleaning is performed based on thresholds, so that higher-quality target graphics and text objects can be cleaned to train a better text-based graph model. Multimodal model training such as text-based graphs is supported.

[0189] The graphic object processing device provided by the present invention is described below. The graphic object processing device described below and the graphic object processing method described above can be referred to each other.

[0190] Figure 7 is a schematic diagram of the structure of the graphic object processing device provided by the present invention, such as Figure 7 As shown, the graphic object processing device 700 includes: an acquisition module 701, an analysis module 702 and a determination module 703, wherein:

[0191] The acquisition module 701 is configured to acquire an initial graphic object, wherein the initial graphic object includes an initial image and an initial description text;

[0192] The analysis module 702 is configured to perform image analysis on the initial image from at least one dimension to obtain the confidence of the initial image and calculate a first similarity between the initial description text and the initial image;

[0193] The determination module 703 is configured to determine a target graphic object according to the initial image and the initial description text when the confidence is greater than a first threshold and the first similarity is greater than a second threshold.

[0194] The graphic object processing device provided by the present invention obtains an initial graphic object, wherein the initial graphic object includes an initial image and an initial description text of the initial image; performs image analysis on the initial image from at least one dimension to obtain the confidence of the initial image, and calculates a first similarity between the initial description text and the initial image; when the confidence is greater than a first threshold value, and the first similarity is greater than a second threshold value, determines a target graphic object according to the initial image and the initial description text. The present invention performs data cleaning from two aspects, at least one dimension representing image quality and description accuracy, and adopts a voting method for cleaning, that is, cleaning based on a threshold, so that a higher quality target graphic object is cleaned to train a better Vincent graph model. The training of multimodal models such as Vincent graphs is supported.

[0195] In one or more optional embodiments of the present invention, the analysis module 702 is further configured to:

[0196] Splitting the initial description text into at least two initial sub-texts;

[0197] determining at least one target subtext from the at least two initial subtexts;

[0198] Connecting the target sub-texts to obtain a target description text;

[0199] A first similarity between the target description text and the initial image is calculated.

[0200] In one or more optional embodiments of the present invention, the analysis module 702 is further configured to:

[0201] Calculating a second similarity between each of the initial subtexts and the initial image;

[0202] A target similarity greater than a third threshold value among the second similarities is determined, and the initial subtext corresponding to the target similarity is determined as a target subtext.

[0203] In one or more optional embodiments of the present invention, the determining module 703 is further configured to:

[0204] The target description text corresponding to the initial image and the initial description text is determined as the target graphic object.

[0205] In one or more optional embodiments of the present invention, the analysis module 702 is further configured to:

[0206] Extract image features from the initial image in at least one dimension to obtain image features of the initial image in each dimension;

[0207] The confidence level of the initial image is determined according to each of the image features.

[0208] In one or more optional embodiments of the present invention, the at least one dimension includes at least one of an aesthetic dimension, a watermark dimension, and a violation identification dimension;

[0209] Accordingly, the analysis module 702 is further configured to perform at least one of the following:

[0210] Extracting image features from the initial image from an aesthetic dimension to obtain image features of the initial image in the aesthetic dimension;

[0211] Extracting image features of the initial image from the watermark dimension to obtain image features of the initial image at the watermark dimension;

[0212] Image features of the initial image are extracted from the violation identification dimension to obtain image features of the initial image in the violation identification dimension.

[0213] In one or more optional embodiments of the present invention, the analysis module 702 is further configured to:

[0214] Performing feature averaging processing on each of the image features to obtain comprehensive image features of the initial image;

[0215] Probabilistic prediction is performed on the initial image according to the comprehensive image features to obtain the confidence of the initial image.

[0216] In one or more optional embodiments of the present invention, the analysis module 702 is further configured to:

[0217] For each dimension, the initial image is input into an image processing model corresponding to the dimension to extract image features, so as to obtain image features of the initial image in each dimension;

[0218] The image processing model is trained based on a first sample image carrying a dimension label, where the dimension label is a probability value of the first sample image under the dimension.

[0219] In one or more optional embodiments of the present invention, the analysis module 702 is further configured to:

[0220] Encoding the initial description text and the initial image respectively to obtain a text embedding vector and an image embedding vector;

[0221] A similarity calculation is performed on the text embedding vector and the image embedding vector to obtain a first similarity between the initial description text and the initial image.

[0222] In one or more optional embodiments of the present invention, the analysis module 702 is further configured to:

[0223] Inputting the initial description text into the text encoding layer of the pre-trained similarity prediction model for text encoding processing to obtain a text embedding vector, and inputting the initial image into the image encoding layer of the similarity prediction model for image encoding processing to obtain an image embedding vector;

[0224] The text embedding vector and the image embedding vector are input into the similarity prediction layer of the similarity prediction model to perform similarity calculation, so as to obtain a first similarity between the initial description text and the initial image.

[0225] In one or more optional embodiments of the present invention, the graphic object processing device 700 further includes a training module configured to:

[0226] Obtaining an untrained similarity prediction model and a sample image-text object carrying a similarity label, wherein the similarity prediction model includes a text encoding layer, an image encoding layer, and a similarity prediction layer, and the sample image-text object includes a second sample image and a sample description text of the second sample image;

[0227] Inputting the sample description text into the text encoding layer to obtain a predicted text embedding vector, and inputting the second sample image into the image encoding layer to obtain a predicted image embedding vector;

[0228] Inputting the predicted text embedding vector and the predicted image embedding vector into the similarity prediction layer to obtain predicted similarity;

[0229] Calculating a loss value according to the similarity label and the predicted similarity;

[0230] The similarity prediction model is gradient optimized according to the loss value, and the step of obtaining sample graphic objects carrying similarity labels is continued until a preset stop condition is reached to obtain the trained similarity prediction model.

[0231] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the image and text object processing method, which includes: obtaining an initial image and text object, the initial image and text object including an initial image and an initial description text of the initial image; performing image analysis on the initial image from at least one dimension to obtain the confidence of the initial image, and calculating the first similarity between the initial description text and the initial image; when the confidence is greater than a first threshold value and the first similarity is greater than a second threshold value, determining the target image and text object according to the initial image and the initial description text.

[0232] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0233] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the graphic object processing method provided by the above-mentioned methods, and the method includes: obtaining an initial graphic object, and the initial graphic object includes an initial image and an initial description text of the initial image; performing image analysis on the initial image from at least one dimension to obtain the confidence of the initial image, and calculating a first similarity between the initial description text and the initial image; when the confidence is greater than a first threshold and the first similarity is greater than a second threshold, determining a target graphic object based on the initial image and the initial description text.

[0234] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the graphic object processing method provided by the above-mentioned methods, the method comprising: obtaining an initial graphic object, the initial graphic object comprising an initial image and an initial description text of the initial image; performing image analysis on the initial image from at least one dimension to obtain a confidence level of the initial image, and calculating a first similarity between the initial description text and the initial image; when the confidence level is greater than a first threshold and the first similarity is greater than a second threshold, determining a target graphic object based on the initial image and the initial description text.

[0235] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0236] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0237] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing graphic objects, characterized in that: include: Acquire an initial graphic object, wherein the initial graphic object includes an initial image and an initial description text of the initial image; Performing image analysis on the initial image from at least one dimension to obtain a confidence level of the initial image, and calculating a first similarity between the initial description text and the initial image; When the confidence is greater than a first threshold and the first similarity is greater than a second threshold, a target graphic object is determined according to the initial image and the initial description text.

2. The method for processing graphic objects according to claim 1, characterized in that: The calculating a first similarity between the initial description text and the initial image includes: Splitting the initial description text into at least two initial sub-texts; determining at least one target subtext from the at least two initial subtexts; Connecting the target sub-texts to obtain a target description text; A first similarity between the target description text and the initial image is calculated.

3. The method for processing graphic objects according to claim 2, characterized in that: The step of determining at least one target subtext from the at least two initial subtexts comprises: Calculating a second similarity between each of the initial subtexts and the initial image; A target similarity greater than a third threshold value among the second similarities is determined, and the initial subtext corresponding to the target similarity is determined as a target subtext.

4. The method for processing graphic objects according to claim 2 or 3, characterized in that: The step of determining the target graphic object according to the initial image and the initial description text comprises: The target description text corresponding to the initial image and the initial description text is determined as the target graphic object.

5. The method for processing graphic objects according to claim 1, characterized in that: The performing image analysis on the initial image from at least one dimension to obtain the confidence of the initial image includes: Extract image features from the initial image in at least one dimension to obtain image features of the initial image in each dimension; The confidence level of the initial image is determined according to each of the image features.

6. The method for processing graphic objects according to claim 5, characterized in that: The at least one dimension includes at least one of an aesthetic dimension, a watermark dimension, and a violation identification dimension; Accordingly, the extracting image features from the initial image in at least one dimension to obtain the image features of the initial image in each dimension includes at least one of the following: Extracting image features from the initial image from an aesthetic dimension to obtain image features of the initial image in the aesthetic dimension; Extracting image features of the initial image from the watermark dimension to obtain image features of the initial image at the watermark dimension; Image features of the initial image are extracted from the violation identification dimension to obtain image features of the initial image in the violation identification dimension.

7. The method for processing graphic objects according to claim 5 or 6, characterized in that: Determining the confidence of the initial image according to each of the image features includes: Performing feature averaging processing on each of the image features to obtain comprehensive image features of the initial image; Probabilistic prediction is performed on the initial image according to the comprehensive image features to obtain the confidence of the initial image.

8. The method for processing graphic objects according to claim 5 or 6, characterized in that: The extracting image features of the initial image from at least one dimension to obtain the image features of the initial image in each dimension includes: For each dimension, the initial image is input into an image processing model corresponding to the dimension to extract image features, so as to obtain image features of the initial image in each dimension; The image processing model is trained based on a first sample image carrying a dimension label, where the dimension label is a probability value of the first sample image under the dimension.

9. The method for processing graphic objects according to any one of claims 1 to 3, characterized in that: The calculating a first similarity between the initial description text and the initial image includes: Encoding the initial description text and the initial image respectively to obtain a text embedding vector and an image embedding vector; A similarity calculation is performed on the text embedding vector and the image embedding vector to obtain a first similarity between the initial description text and the initial image.

10. The method for processing graphic objects according to claim 9, characterized in that: The encoding process is performed on the initial description text and the initial image to obtain a text embedding vector and an image embedding vector, respectively, including: Inputting the initial description text into the text encoding layer of the pre-trained similarity prediction model for text encoding processing to obtain a text embedding vector, and inputting the initial image into the image encoding layer of the similarity prediction model for image encoding processing to obtain an image embedding vector; Accordingly, the performing similarity calculation on the text embedding vector and the image embedding vector to obtain a first similarity between the initial description text and the initial image includes: The text embedding vector and the image embedding vector are input into the similarity prediction layer of the similarity prediction model to perform similarity calculation, so as to obtain a first similarity between the initial description text and the initial image.

11. A graphic object processing device, characterized in that: include: An acquisition module is configured to acquire an initial graphic object, wherein the initial graphic object includes an initial image and an initial description text; an analysis module, configured to perform image analysis on the initial image from at least one dimension, obtain a confidence level of the initial image, and calculate a first similarity between the initial description text and the initial image; The determination module is configured to determine the target graphic object according to the initial image and the initial description text when the confidence is greater than a first threshold and the first similarity is greater than a second threshold.

Images