A semantic integration analysis method and system for deep image and text analysis

By performing semantic segmentation and multi-dimensional semantic correction of comics, combined with situation and action analysis, the problem of lack of semantic correlation in traditional graphic and text analysis is solved, and more accurate graphic and text information understanding and deep meaning mining are achieved.

CN118799877BActive Publication Date: 2024-11-29CHENGDU YUNLAN TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411268225.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-11-29
Estimated Expiration
2044-09-11

AI Technical Summary

Technical Problem

Traditional graphic analysis methods lack the integration of semantic relationships between images and text, resulting in inaccurate analysis results, making it difficult to capture the rich emotional expressions, character interactions and complex plot clues in comics.

Method used

By semantic segmentation of the target graphics and text, the subject image, background image and text image are extracted, and situation type analysis and action prediction are combined, multi-dimensional semantic correction and fusion are performed, and the text semantics with the greatest probability of fusion semantics are selected as the semantic integration result.

Benefits of technology

It improves the accuracy of graphic and text analysis, fully explores the deep meaning and value of graphic and text information, and solves the problems of one-sidedness and inaccuracy of analysis results in traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118799877B_ABST
    Figure CN118799877B_ABST
Patent Text Reader

Abstract

The present invention discloses a semantic integration analysis method and system for deep image and text analysis, and relates to the field of data processing technology. The method comprises: performing semantic segmentation processing on the target image and text, segmenting and extracting to obtain the main image, background image and text image; recognizing the text image to obtain multiple text semantic probabilities; performing situation type analysis to obtain multiple background semantic probabilities; obtaining the main situation type; performing action prediction on the main image to obtain a main action image sequence, and performing main action situation analysis to obtain multiple main semantic probabilities; fusing processing to obtain multiple fused semantic probabilities, screening the text semantics with the largest fused semantic probability, and displaying them as semantic integration results. The method solves the technical problem that the traditional image and text analysis lacks the integration of the semantic association between images and texts, resulting in inaccurate analysis results. By performing multi-dimensional correction and fusion of text semantics, the technical effect of improving the accuracy of image and text analysis is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a semantic integration analysis method and system for deep image and text analysis. Background Art

[0002] In the digital age, comics, as a unique form of visual narrative, are not only loved by readers around the world, but also become an important carrier of cultural communication and creative expression. However, as comic works become increasingly rich and complex, how to efficiently identify and understand the semantics of images and texts in comics has become a key technical challenge. Traditional image and text recognition methods often focus on a single aspect of image recognition or text analysis, ignoring the inherent connection and interaction between images and texts, which makes it difficult to capture the rich emotional expressions, character interactions, and complex plot clues in comics, resulting in one-sided and inaccurate understanding of image and text content, making it difficult to fully explore the deep meaning and value of image and text information. Summary of the invention

[0003] The present application provides a semantic integration analysis method and system for deep image and text analysis, which solves the technical problem that traditional image and text analysis lacks integration of semantic associations between images and texts, resulting in inaccurate analysis results.

[0004] In view of the above problems, the present application provides a semantic integration analysis method and system for deep image and text analysis.

[0005] In a first aspect of the present application, a semantic integration analysis method for deep image and text analysis is provided, the method comprising:

[0006] The target image to be analyzed is subjected to semantic segmentation processing, and the segmentation and extraction are performed to obtain the main image, background image and text image; the text image is recognized to obtain multiple text semantics, and multiple text semantic probabilities of the multiple text semantics are obtained; a context type analysis is performed based on the background image to obtain the background context type, and the correlation coefficient with the multiple text semantics is analyzed, and the multiple text semantic probabilities are subjected to background correction to obtain multiple background semantic probabilities; a context type analysis is performed on the main image to obtain the main context type; an action prediction is performed on the main image to obtain a main action image sequence, and a main action context analysis is performed to obtain the main action context type, and the multiple text semantic probabilities are subjected to main correction in combination with the main context type to obtain multiple main semantic probabilities; a fusion processing is performed based on the multiple background semantic probabilities and the multiple main semantic probabilities to obtain multiple fused semantic probabilities, and the text semantic with the largest fused semantic probability is selected and displayed as the semantic integration result.

[0007] The second aspect of the present application provides a semantic integration analysis system for deep image and text analysis, the system comprising:

[0008] a segmentation module, the segmentation module is used to perform semantic segmentation processing on the target image to be analyzed, and the segmentation and extraction obtain the main image, background image and text image; a recognition module, the recognition module is used to recognize the text image, obtain multiple text semantics, and obtain multiple text semantic probabilities of the multiple text semantics; a first analysis module, the first analysis module is used to perform context type analysis based on the background image, obtain the background context type, analyze the correlation coefficient with the multiple text semantics, perform background correction on the multiple text semantic probabilities, and obtain multiple background semantic probabilities; a second analysis module, the second analysis module is used to perform context type analysis on the main image, and obtain the main context type; a third analysis module, the third analysis module is used to perform action prediction on the main image, obtain a main action image sequence, and perform main action context analysis to obtain the main action context type, and perform main correction on the multiple text semantic probabilities in combination with the main context type to obtain multiple main semantic probabilities; a fusion processing module, the fusion processing module is used to obtain multiple fusion semantic probabilities based on the multiple background semantic probabilities and the multiple main semantic probabilities, and select the text semantic with the largest fusion semantic probability to display as a semantic integration result.

[0009] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0010] By performing semantic segmentation on the target image to be analyzed, the subject image, background image and text image are obtained by segmentation and extraction. Then, the text image is recognized to obtain multiple text semantics, and multiple text semantic probabilities of the multiple text semantics are obtained. Next, the context type analysis is performed according to the background image to obtain the background context type, and the correlation coefficient with the multiple text semantics is analyzed. The multiple text semantic probabilities are corrected for the background to obtain multiple background semantic probabilities. Further, the subject image is analyzed for the context type to obtain the subject context type; at the same time, the subject image is predicted for action to obtain the subject action image sequence, and the subject action context analysis is performed to obtain the subject action context type. Combined with the subject context type, the multiple text semantic probabilities are corrected for the subject to obtain multiple subject semantic probabilities. Finally, according to the multiple background semantic probabilities and the multiple subject semantic probabilities, the fusion processing is performed to obtain multiple fusion semantic probabilities, and the text semantics with the largest fusion semantic probability is selected and displayed as the semantic integration result. The technical problem that the traditional image and text analysis lacks the integration of the semantic association between images and texts, resulting in inaccurate analysis results is solved. By multi-dimensional correction and fusion of text semantics, the technical effect of improving the accuracy of image and text analysis is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A flowchart of a semantic integration analysis method for deep image and text analysis provided in an embodiment of the present application;

[0013] Figure 2 A schematic diagram of the structure of a semantic integration analysis system for deep image and text analysis provided in an embodiment of the present application.

[0014] Explanation of the reference numerals: segmentation module 11 , recognition module 12 , first analysis module 13 , second analysis module 14 , third analysis module 15 , fusion processing module 16 . DETAILED DESCRIPTION

[0015] This application solves the technical problem that traditional image and text analysis lacks integration of semantic associations between images and texts, resulting in inaccurate analysis results, by providing a semantic integration analysis method and system for deep image and text analysis.

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0017] It should be noted that the terms "including" and "having" are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules that are not explicitly listed or are inherent to these processes, methods, products or devices.

[0018] Embodiment 1, as Figure 1 As shown, the present application provides a semantic integration analysis method for deep image and text analysis, wherein the method includes:

[0019] The target image and text to be analyzed are subjected to semantic segmentation, and the main image, background image and text image are obtained by segmentation and extraction.

[0020] The target image refers to a cartoon to be semantically analyzed. The target image is segmented and extracted to obtain the main image, background image and text image. That is, the image content in the cartoon is subdivided into different semantic units, including the main image (such as characters, key items, etc.), the background image (such as scenes, environments, etc.), and the text image (such as text and labels in dialog boxes, etc.).

[0021] Furthermore, the target image to be analyzed is subjected to semantic segmentation, and the subject image, background image and text image are obtained by segmentation and extraction, including:

[0022] Based on the target image and text, a collection of similar images and texts is obtained; multiple images and texts in the same family in the collection of images and texts are divided and marked into main images, background images and text images to obtain a sample division result set; based on the semantic segmentation network, an encoder and a decoder are constructed to form an image and text segmentation path; the image and text segmentation path is supervised and trained using the collection of similar images and texts and the sample division result set until convergence requirements are met; the target image and text is input into the image and text segmentation path for segmentation, and the main image, background image and text image are obtained by division and extraction.

[0023] The same family of images and texts refer to images and texts that are similar or related in content, style, theme or role. According to the target image and text, the same family of images and texts can be obtained from big data to form a set of same family of images and texts; each image and text in the same family of image and text set is divided, and the main image, background image and text image are distinguished by annotation, and the divided images and texts and their corresponding labels (main body, background, text) are saved as a sample segmentation result set for subsequent training and verification. Based on the semantic segmentation network (such as U-Net, DeepLab, etc.), an encoder-decoder architecture suitable for image and text segmentation is designed. The encoder is used to extract image features, and the decoder is used to convert feature maps back to pixel-level segmentation maps; the path for image and text segmentation is clearly divided in the network, including the input layer, encoder layer, decoder layer and output layer; the images and texts in the same family of image and text set are used as training data and matched with the corresponding sample segmentation result set, and the training data is input into the network, the loss is calculated through forward propagation, and then the network weights are updated through back propagation. This process is repeated until the convergence requirements are met (such as the loss value no longer decreases significantly, or the preset training rounds are reached). The target image to be segmented is input into the trained semantic segmentation network, and the segmentation map of the target image is output through the forward propagation of the network. According to the segmentation map, the target image is divided into main image, background image and text image.

[0024] The text image is recognized to obtain a plurality of text semantics, and a plurality of text semantic probabilities of the plurality of text semantics are obtained.

[0025] By recognizing text images, multiple text semantics and their corresponding text semantic probabilities can be obtained. In other words, by recognizing text images, the text type (such as English, Japanese, etc.) can be obtained, and after translation and recognition, multiple text semantics can be obtained, that is, one foreign language may correspond to multiple semantics.

[0026] Furthermore, recognizing the text image to obtain a plurality of text semantics, and obtaining a plurality of text semantic probabilities of the plurality of text semantics, includes:

[0027] According to the text semantic translation record of the image and text, a set of sample text images and a set of sample text recognition results are collected; the sample text image set and the sample text recognition result set are used to train a basic text image recognition path based on a convolutional neural network; the text image is input into the basic text image recognition path to obtain a text recognition result through recognition; the text recognition result is input into a text semantic library, multiple text semantics of the text recognition result are obtained through indexing, and multiple text semantic probabilities are obtained according to the appearance ratios of the multiple text semantics in the text semantic library, wherein the text semantic library is constructed based on multiple sample text recognition results and multiple sample text semantic sets of the same family of images and texts.

[0028] Preferably, according to the text semantic translation record of the graphic (comic), sample text images containing relevant text are collected from big data to form a sample text image set, and the text content of each sample text image is annotated to form a sample text recognition result set; a convolutional neural network architecture suitable for text image recognition is selected, such as VGG, ResNet, etc., and the sample text image set and the corresponding sample text recognition result set are used as training data to train the convolutional neural network, and after the training is completed, a basic text image recognition path is obtained; by inputting the text image into the basic text image recognition path, the basic text image recognition path will recognize the text image and output the text recognition result; a text semantic library is constructed based on multiple sample text recognition results of the same family of graphics and multiple sample text semantic sets, and the text semantic library can contain text semantics, corresponding context information, occurrence frequency, etc.; the text recognition result is input into the text semantic library as a query, and the text semantics matching or similar to the query result is found through the index mechanism. For each matched text semantic, its occurrence ratio in the text semantic library is counted, and the occurrence ratio is output as the probability of the text semantic, and multiple text semantic probabilities are obtained, wherein the higher the probability value, the more relevant the text semantics is to the query result.

[0029] A context type analysis is performed based on the background image to obtain a background context type, a correlation coefficient with the multiple text semantics is analyzed, and background correction is performed on the multiple text semantic probabilities to obtain multiple background semantic probabilities.

[0030] The background image is analyzed for situation type to identify the situation of the cartoon (e.g., the ellipsis in the background can reflect an embarrassing situation); based on the situation type, the correlation between the situation and multiple text semantics is analyzed to obtain the correlation coefficient. For example, when the situation is embarrassing, the correlation of semantics related to embarrassment is relatively large, that is, the correlation coefficient is relatively large; based on the correlation coefficient, the text semantic probability is corrected for the background. The purpose of the correction is to adjust the probability of the text semantics to make it more consistent with the context of the current background situation. Specifically, for each text semantic, its original probability (text semantic probability) is multiplied by the corresponding correlation coefficient to obtain the corrected probability (background semantic probability).

[0031] Further, performing a context type analysis according to the background image to obtain a background context type, analyzing the correlation coefficient with the multiple text semantics, performing background correction on the multiple text semantic probabilities to obtain multiple background semantic probabilities, including:

[0032] A set of sample background images and a set of sample background context types are collected, and a background context identifier is trained based on a convolutional neural network; the background context identifier is used to perform context type analysis on the background images to obtain background context types; a set of sample text semantics is collected, and combined with the set of sample background context types, identifications are analyzed to obtain a set of sample association coefficients, and a background context semantic association analyzer is trained to perform background context semantic association analysis on the background context types and multiple text semantics to obtain multiple background semantic association coefficients; the multiple background semantic association coefficients are used to perform correction calculations on the multiple text semantic probabilities to obtain multiple background semantic probabilities.

[0033] Background images of different situation types are collected from the database to form a sample background image set, and each sample background image is labeled with a situation type to form a sample background situation type set; a convolutional neural network suitable for image classification is selected, such as VGG, ResNet, etc., and the sample background image set and the corresponding sample background situation type set are used as training data to train the convolutional neural network, and a background situation recognizer is obtained after the training is completed; the background image is input into the background situation recognizer, and the background situation recognizer will perform a situation type analysis on the background image and output the background situation type; text semantics related to the background image are collected, such as text descriptions, titles, labels, etc. of the image-text pairing to form a sample text semantic set; the sample background situation type set and the sample text semantic set are combined, The occurrence frequency or relevance of each text semantics under different background context types is analyzed, and a set of sample relevance coefficients is calculated; the sample relevance coefficient set, the sample text semantic set and the sample background context type set are used as training data to train a background context semantic association analyzer; the background context type and multiple text semantics are input into the background context semantic association analyzer to perform association analysis and obtain multiple background semantic association coefficients; based on the obtained background semantic association coefficients, the original multiple text semantic probabilities are corrected and calculated, and the probability of each text semantic can be multiplied by the corresponding association coefficient, and normalized to ensure that the sum of the probabilities is 1; after the correction calculation, each text semantic obtains a more reasonable probability in the current background context, namely, the background semantic probability.

[0034] Performing situation type analysis on the subject image to obtain a subject situation type.

[0035] A similar method to the situation type analysis of the background image is used to conduct situation type analysis on the subject image to identify the status (good or bad) and emotions (happy, sad, embarrassed, etc.) of the characters in the comics and obtain the subject situation type.

[0036] Perform action prediction on the subject image to obtain a subject action image sequence, perform subject action context analysis to obtain a subject action context type, and perform subject correction on the multiple text semantic probabilities in combination with the subject context type to obtain multiple subject semantic probabilities.

[0037] The action of the subject image is predicted to obtain a subject action image sequence. Further, the subject action context analysis is performed on the subject action image sequence to obtain the subject action context type. Based on the subject context type and the subject action context type, multiple text semantic probabilities are subject-corrected to obtain multiple subject semantic probabilities.

[0038] Furthermore, performing action prediction on the subject image to obtain a subject action image sequence, performing subject action context analysis to obtain a subject action context type, and performing subject correction on the plurality of text semantic probabilities in combination with the subject context type, including:

[0039] Perform action prediction on the subject image to generate a subject action image sequence; collect a set of sample subject action image sequences, and process the identifiers to obtain a set of sample subject action context types; use the set of sample subject action image sequences and the set of sample subject action context types to train an action context recognition path based on a spatiotemporal action recognition network; input the subject action image sequence into the action context recognition path to identify and obtain the subject action context type; analyze and obtain multiple subject semantic association coefficients based on the subject context type and the multiple text semantics, wherein a set of sample subject context types, a set of sample text semantics, and a set of sample subject semantic association coefficients are collected, and a subject context semantic association analyzer is trained for analysis; analyze and obtain multiple action semantic association coefficients based on the subject action context type and the multiple text semantics, and obtain multiple semantic association coefficients by weighted calculation in combination with the multiple subject semantic association coefficients, wherein a set of sample subject action context types, a set of sample text semantics, and a set of sample action semantic association coefficients are collected, and an action context semantic association analyzer is trained for analysis; use the multiple semantic association coefficients to perform correction calculation on the text semantic probability to obtain multiple subject semantic probabilities.

[0040] Preferably, action prediction is performed on the subject image to generate a subject action image sequence; image sequences containing different subject actions and situations are collected to form a sample subject action image sequence set, and action situation annotation is performed on each sample action image sequence to obtain a sample subject action situation type set; the sample subject action image sequence set and the sample subject action situation type set are used as training data to train a spatiotemporal action recognition network, which can receive the action image sequence as input and output the corresponding action situation type, and an action situation recognition path is obtained after the training is completed; the subject action image sequence is input into the trained action situation recognition path to obtain a recognition result of the subject action situation type; a plurality of subject images annotated with situation types are collected to form a sample subject situation type set, and text semantics related to the subject image, such as text descriptions, titles, labels, etc. paired with images and texts, are collected to form a sample text semantic set, and the subject situation type set and the sample text semantic set are combined to analyze the occurrence frequency or relevance of each text semantic under different subject situation types, and calculate the sample subject semantic correlation coefficient; the sample subject situation type set is used to identify the subject action situation type. The type set, the sample text semantic set and the sample subject semantic association coefficient set are used as training data to train the subject context semantic association analyzer, which is used to analyze the association between the subject context type and the text semantics to obtain the subject semantic association coefficient; similarly, the sample subject action context type set, the sample text semantic set and the sample action semantic association coefficient set are used as training data to train the action context semantic association analyzer, which is used to analyze the association between the action context type and the text semantics to obtain the action semantic association coefficient; the subject semantic association coefficient and the action semantic association coefficient are combined to obtain the comprehensive association coefficient (i.e., multiple semantic association coefficients) of each text semantic through weighted calculation, and the multiple semantic association coefficients obtained by calculation are used to correct the original text semantic probability, multiply the probability of each text semantic by the corresponding comprehensive association coefficient, and normalize it to ensure that the sum of the probabilities is 1; after the correction calculation, each text semantic obtains a more reasonable probability under the subject action and context context, i.e., the subject semantic probability.

[0041] Furthermore, performing action prediction on the subject image to generate a subject action image sequence includes:

[0042] A set of sample subject images is obtained, and action images are supplemented to obtain a set of sample subject action image sequences; based on a generative adversarial network, a generator and an adversary in an action image prediction path are constructed; the action image prediction path is trained using the sample subject image set and the sample subject action image sequence set until convergence requirements are met; the subject image is input into the action image prediction path for action prediction to generate a subject action image sequence.

[0043] Preferably, multiple images containing different subjects are collected from big data to form a sample subject image set, and each subject image is supplemented with an action image. Through animation simulation, an action image sequence corresponding to each subject image is obtained to form a sample subject action image sequence set; based on a generative adversarial network, a generator and an adversary in the action image prediction path are constructed, the goal of the generator is to receive a subject image as input and generate a corresponding action image sequence, and the task of the adversary is to distinguish whether the input action image sequence is from a real data set or generated by a generator; using the sample subject image set and the corresponding sample subject action image sequence set as training data, in each In the iteration, a subject image is randomly selected as the input of the generator, and the generator generates an action image sequence. The adversary receives the generated action image sequence and the real action image sequence as input and tries to distinguish them. According to the output of the adversary, the loss function (such as cross entropy loss) is calculated, and the parameters of the generator and the adversary are updated through the back propagation algorithm. The above process is repeated until the convergence requirements are met (such as the loss function value is stable, the generated image quality meets the predetermined standard, etc.), and the action image prediction path is obtained; the subject image is input into the trained action image prediction path, and the generator in the action image prediction path will generate a corresponding action image sequence based on the input subject image.

[0044] According to the multiple background semantic probabilities and the multiple subject semantic probabilities, a fusion process is performed to obtain multiple fusion semantic probabilities, and the text semantics with the largest fusion semantic probability is screened and displayed as a semantic integration result.

[0045] After obtaining multiple background semantic probabilities and multiple subject semantic probabilities, multiple fused semantic probabilities are obtained by fusion processing, that is, by assigning a weight to each background semantic probability and subject semantic probability, and then weighted averaging them to obtain multiple fused semantic probabilities, wherein the weight assignment can be determined according to the specific situation, for example, it can be adjusted according to the importance of background information and subject information in a specific scene. After obtaining multiple fused semantic probabilities, the text semantics with the largest fused semantic probability is selected and displayed as a semantic integration result, such as by displaying it on the screen or by voice to the user, thereby improving the effect, accuracy, and efficiency of reading foreign language comics.

[0046] Furthermore, according to the multiple background semantic probabilities and the multiple subject semantic probabilities, a fusion process is performed to obtain multiple fusion semantic probabilities, and the text semantics with the largest fusion semantic probability is screened, including:

[0047] Obtaining image area ratios of the subject image and the background image, configuring a subject weight and a background weight according to the image area ratios; using the subject weight and the background weight, performing weighted calculation on the multiple subject semantic probabilities and the multiple background semantic probabilities to obtain multiple fused semantic probabilities;

[0048] The text semantics with the highest probability of fusion semantics are selected and displayed as the semantic integration result of the target image and text.

[0049] Preferably, the area ratios of the main image and the background image in the overall image are calculated, and corresponding main weights and background weights are configured according to the image area ratios of the main image and the background image. The area ratios can be used as the main weights and the background weights; based on the main weights and the background weights, multiple main semantic probabilities and multiple background semantic probabilities are weightedly calculated to obtain multiple fused semantic probabilities; the text semantics with the largest fused semantic probability is selected from all fused semantic probabilities as the semantic integration result of the target image and text, and the semantic integration result is displayed to the user through the screen or voice.

[0050] In summary, the embodiments of the present application have at least the following technical effects:

[0051] By performing semantic segmentation on the target image to be analyzed, the subject image, background image and text image are obtained by segmentation and extraction. Then, the text image is recognized to obtain multiple text semantics, and multiple text semantic probabilities of the multiple text semantics are obtained. Next, the context type analysis is performed according to the background image to obtain the background context type, and the correlation coefficient with the multiple text semantics is analyzed. The multiple text semantic probabilities are corrected for the background to obtain multiple background semantic probabilities. Further, the subject image is analyzed for the context type to obtain the subject context type; at the same time, the subject image is predicted for action to obtain the subject action image sequence, and the subject action context analysis is performed to obtain the subject action context type. Combined with the subject context type, the multiple text semantic probabilities are corrected for the subject to obtain multiple subject semantic probabilities. Finally, according to the multiple background semantic probabilities and the multiple subject semantic probabilities, the fusion processing is performed to obtain multiple fusion semantic probabilities, and the text semantics with the largest fusion semantic probability is selected and displayed as the semantic integration result. The technical problem that the traditional image and text analysis lacks the integration of the semantic association between images and texts, resulting in inaccurate analysis results is solved. By multi-dimensional correction and fusion of text semantics, the technical effect of improving the accuracy of image and text analysis is achieved.

[0052] Embodiment 2, based on the same inventive concept as the semantic integration analysis method of deep image and text analysis in the above embodiment, Figure 2 As shown, the present application provides a semantic integration analysis system for deep image and text analysis, wherein the system includes:

[0053] A segmentation module 11 is used to perform semantic segmentation on the target image to be analyzed, and to extract the subject image, background image and text image by segmentation; a recognition module 12 is used to recognize the text image, obtain multiple text semantics, and obtain multiple text semantic probabilities of the multiple text semantics; a first analysis module 13 is used to perform context type analysis based on the background image, obtain background context type, analyze the correlation coefficient with the multiple text semantics, perform background correction on the multiple text semantic probabilities, and obtain multiple background semantic probabilities; a second analysis module 14 ...5 is used to perform context type analysis based on the background image, obtain background context type, analyze the correlation coefficient with the multiple text semantics, perform background correction on the multiple text semantic probabilities, and obtain multiple background semantic probabilities; a second analysis module 16 is used to perform context type analysis based on the background image, obtain background context type, analyze the correlation coefficient with the multiple text semantics, perform background correction on the multiple text semantic probabilities, and obtain multiple background semantic probabilities; a second analysis module 17 is used to perform context type analysis based on the background image, obtain background context type, and obtain multiple background semantic probabilities; a second analysis module 18 is used to perform context type analysis based on the background image, obtain background context type, analyze the correlation coefficient with the multiple text semantics, perform background correction on the multiple text semantic probabilities, and obtain multiple background semantic probabilities; a second analysis module 19 is used to perform context type analysis based on the background image, obtain background context type, and obtain background context type; a second analysis module 11 is used to perform context type analysis based on the background image, obtain background context type, and obtain background context type; Block 14 is used to perform situation type analysis on the subject image to obtain the subject situation type; a third analysis module 15 is used to perform action prediction on the subject image to obtain a subject action image sequence, and perform subject action situation analysis to obtain the subject action situation type, and perform subject correction on the multiple text semantic probabilities in combination with the subject situation type to obtain multiple subject semantic probabilities; a fusion processing module 16 is used to obtain multiple fused semantic probabilities based on the multiple background semantic probabilities and the multiple subject semantic probabilities through fusion processing, and screen the text semantics with the largest fused semantic probability to display as the semantic integration result.

[0054] Furthermore, the segmentation module 11 is used to perform the following method:

[0055] Based on the target image and text, a collection of similar images and texts is obtained; multiple images and texts in the same family in the collection of images and texts are divided and marked into main images, background images and text images to obtain a sample division result set; based on the semantic segmentation network, an encoder and a decoder are constructed to form an image and text segmentation path; the image and text segmentation path is supervised and trained using the collection of similar images and texts and the sample division result set until convergence requirements are met; the target image and text is input into the image and text segmentation path for segmentation, and the main image, background image and text image are obtained by division and extraction.

[0056] Furthermore, the identification module 12 is used to execute the following method:

[0057] According to the text semantic translation record of the image and text, a set of sample text images and a set of sample text recognition results are collected; the sample text image set and the sample text recognition result set are used to train a basic text image recognition path based on a convolutional neural network; the text image is input into the basic text image recognition path to obtain a text recognition result through recognition; the text recognition result is input into a text semantic library, multiple text semantics of the text recognition result are obtained through indexing, and multiple text semantic probabilities are obtained according to the appearance ratios of the multiple text semantics in the text semantic library, wherein the text semantic library is constructed based on multiple sample text recognition results and multiple sample text semantic sets of the same family of images and texts.

[0058] Furthermore, the first analysis module 13 is used to perform the following method:

[0059] A set of sample background images and a set of sample background context types are collected, and a background context identifier is trained based on a convolutional neural network; the background context identifier is used to perform context type analysis on the background images to obtain background context types; a set of sample text semantics is collected, and combined with the set of sample background context types, identifications are analyzed to obtain a set of sample association coefficients, and a background context semantic association analyzer is trained to perform background context semantic association analysis on the background context types and multiple text semantics to obtain multiple background semantic association coefficients; the multiple background semantic association coefficients are used to perform correction calculations on the multiple text semantic probabilities to obtain multiple background semantic probabilities.

[0060] Furthermore, the third analysis module 15 is used to perform the following method:

[0061] Perform action prediction on the subject image to generate a subject action image sequence; collect a set of sample subject action image sequences, and process the identifiers to obtain a set of sample subject action context types; use the set of sample subject action image sequences and the set of sample subject action context types to train an action context recognition path based on a spatiotemporal action recognition network; input the subject action image sequence into the action context recognition path to identify and obtain the subject action context type; analyze and obtain multiple subject semantic association coefficients based on the subject context type and the multiple text semantics, wherein a set of sample subject context types, a set of sample text semantics, and a set of sample subject semantic association coefficients are collected, and a subject context semantic association analyzer is trained for analysis; analyze and obtain multiple action semantic association coefficients based on the subject action context type and the multiple text semantics, and obtain multiple semantic association coefficients by weighted calculation in combination with the multiple subject semantic association coefficients, wherein a set of sample subject action context types, a set of sample text semantics, and a set of sample action semantic association coefficients are collected, and an action context semantic association analyzer is trained for analysis; use the multiple semantic association coefficients to perform correction calculation on the text semantic probability to obtain multiple subject semantic probabilities.

[0062] Furthermore, the third analysis module 15 is used to perform the following method:

[0063] A set of sample subject images is obtained, and action images are supplemented to obtain a set of sample subject action image sequences; based on a generative adversarial network, a generator and an adversary in an action image prediction path are constructed; the action image prediction path is trained using the sample subject image set and the sample subject action image sequence set until convergence requirements are met; the subject image is input into the action image prediction path for action prediction to generate a subject action image sequence.

[0064] Furthermore, the fusion processing module 16 is used to perform the following method:

[0065] The image area ratios of the subject image and the background image are obtained, and the subject weight and the background weight are configured according to the image area ratios; the subject weight and the background weight are used to perform weighted calculation on the multiple subject semantic probabilities and the multiple background semantic probabilities to obtain multiple fused semantic probabilities; the text semantics with the largest fused semantic probability are screened and displayed as the semantic integration result of the target image and text.

[0066] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0067] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0068] This specification and drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A semantic integration analysis method for deep image and text analysis, characterized in that: The method comprises: Perform semantic segmentation on the target image and text to be analyzed, and extract the subject image, background image and text image; Recognize the text image to obtain multiple text semantics, and obtain multiple text semantic probabilities of the multiple text semantics; Performing a context type analysis according to the background image to obtain a background context type, analyzing the correlation coefficients with the multiple text semantics, and performing background correction on the multiple text semantic probabilities to obtain multiple background semantic probabilities, wherein the background correction includes multiplying the multiple text semantic probabilities with the corresponding correlation coefficients of the multiple text semantics, and performing normalization processing to ensure that the sum of the probabilities is 1, thereby obtaining multiple background semantic probabilities; Performing situation type analysis on the subject image to obtain a subject situation type; Performing action prediction on the subject image to obtain a subject action image sequence, and performing subject action context analysis to obtain a subject action context type, and performing subject correction on the multiple text semantic probabilities in combination with the subject context type to obtain multiple subject semantic probabilities, wherein the subject correction includes performing weighted calculation on the subject semantic association coefficient and the action semantic association coefficient to obtain multiple semantic association coefficients, multiplying the multiple text semantic probabilities with the corresponding multiple semantic association coefficients, and performing normalization processing to ensure that the sum of the probabilities is 1, thereby obtaining multiple subject semantic probabilities; According to the multiple background semantic probabilities and the multiple subject semantic probabilities, a fusion process is performed to obtain multiple fused semantic probabilities, and the text semantics with the largest fused semantic probability are screened and displayed as the semantic integration result, wherein the fusion process includes obtaining the image area ratios of the subject image and the background image, configuring the subject weight and the background weight according to the image area ratios, and using the subject weight and the background weight to perform weighted calculation on the multiple subject semantic probabilities and the multiple background semantic probabilities to obtain multiple fused semantic probabilities.

2. The semantic integration analysis method of deep image and text analysis according to claim 1 is characterized in that: The target image to be analyzed is subjected to semantic segmentation, and the subject image, background image and text image are obtained by segmentation and extraction, including: Based on the target image and text, request a collection of images and texts of the same family; Dividing and marking the main image, background image and text image of the plurality of images and texts of the same family in the image and text set of the same family to obtain a sample division result set; Based on the semantic segmentation network, an encoder and decoder are constructed to form a graph-text segmentation path; Using the same family of image and text sets and the sample segmentation result set, supervised training is performed on the image and text segmentation path until convergence requirements are met; The target image and text are input into the image and text segmentation path for segmentation, and the main image, background image and text image are obtained by separation and extraction.

3. The semantic integration analysis method of deep image and text analysis according to claim 1 is characterized in that: Recognizing the text image to obtain multiple text semantics, and obtaining multiple text semantic probabilities of the multiple text semantics, including: According to the text semantic translation record of the image and text, a sample text image set is collected, and a sample text recognition result set is collected; Using the sample text image set and the sample text recognition result set, based on a convolutional neural network, a basic text image recognition path is trained; Inputting the text image into the basic text image recognition path to obtain a text recognition result; The text recognition result is input into a text semantic library, multiple text semantics of the text recognition result are obtained by indexing, and multiple text semantic probabilities are obtained according to the appearance ratios of the multiple text semantics in the text semantic library, wherein the text semantic library is constructed based on multiple sample text recognition results of the same family of images and texts and multiple sample text semantic sets.

4. The semantic integration analysis method of deep image and text analysis according to claim 1 is characterized in that: Performing a context type analysis according to the background image to obtain a background context type, analyzing the correlation coefficient with the multiple text semantics, performing background correction on the multiple text semantic probabilities to obtain multiple background semantic probabilities, including: Collect a set of sample background images and a set of sample background situation types, and train a background situation identifier based on a convolutional neural network; Using the background context identifier, performing context type analysis on the background image to obtain the background context type; Collecting a sample text semantic set, combining the sample background context type set, analyzing the identification to obtain a sample association coefficient set, training a background context semantic association analyzer, and performing background context semantic association analysis on the background context type and multiple text semantics to obtain multiple background semantic association coefficients; The multiple background semantic association coefficients are used to modify and calculate the multiple text semantic probabilities to obtain multiple background semantic probabilities.

5. The semantic integration analysis method of deep image and text analysis according to claim 1 is characterized in that: Predicting the subject image to obtain a subject action image sequence, performing subject action context analysis to obtain a subject action context type, and subject correcting the plurality of text semantic probabilities in combination with the subject context type, including: Performing action prediction on the subject image to generate a subject action image sequence; Collecting a set of sample subject action image sequences, and processing the identification to obtain a set of sample subject action situation types; Using the sample subject action image sequence set and the sample subject action situation type set, based on the spatiotemporal action recognition network, training an action situation recognition path; Inputting the subject action image sequence into the action context recognition path to identify and obtain the subject action context type; According to the subject context type and the multiple text semantics, multiple subject semantic association coefficients are obtained through analysis, wherein a sample subject context type set, a sample text semantic set and a sample subject semantic association coefficient set are collected, and a subject context semantic association analyzer is trained to perform analysis; According to the subject action context type and the multiple text semantics, multiple action semantic association coefficients are obtained by analysis, and multiple semantic association coefficients are obtained by weighted calculation in combination with the multiple subject semantic association coefficients, wherein a set of sample subject action context types, a set of sample text semantics, and a set of sample action semantic association coefficients are collected, and an action context semantic association analyzer is trained for analysis; The multiple semantic association coefficients are used to modify and calculate the text semantic probabilities to obtain multiple subject semantic probabilities.

6. The semantic integration analysis method of deep image and text analysis according to claim 5 is characterized in that: Performing action prediction on the subject image to generate a subject action image sequence includes: Acquire a sample subject image set, and supplement it with action images to obtain a sample subject action image sequence set; Based on the generative adversarial network, the generator and adversary in the action image prediction path are constructed; Using the sample subject image set and the sample subject action image sequence set, the action image prediction path is trained until convergence requirements are met; The subject image is input into the action image prediction path to perform action prediction, thereby generating a subject action image sequence.

7. The semantic integration analysis method of deep image and text analysis according to claim 1 is characterized in that: According to the multiple background semantic probabilities and the multiple subject semantic probabilities, a fusion process is performed to obtain multiple fusion semantic probabilities, and the text semantics with the largest fusion semantic probability is screened, including: Obtaining image area ratios of the subject image and the background image, and configuring a subject weight and a background weight according to the image area ratios; Using the subject weight and the background weight, weighted calculation is performed on the multiple subject semantic probabilities and the multiple background semantic probabilities to obtain multiple fused semantic probabilities; The text semantics with the highest probability of fusion semantics are selected and displayed as the semantic integration result of the target image and text.

8. A semantic integration analysis system for deep image and text analysis, characterized in that: A semantic integration analysis method for implementing the deep image-text analysis according to any one of claims 1 to 7, the system comprising: A segmentation module, which is used to perform semantic segmentation processing on the target image and text to be analyzed, and to extract the subject image, background image and text image; A recognition module, the recognition module is used to recognize the text image, obtain multiple text semantics, and obtain multiple text semantic probabilities of the multiple text semantics; A first analysis module, the first analysis module is used to perform a context type analysis according to the background image to obtain a background context type, analyze the correlation coefficient with the multiple text semantics, perform background correction on the multiple text semantic probabilities to obtain multiple background semantic probabilities, wherein the background correction includes multiplying the multiple text semantic probabilities with the corresponding correlation coefficients of the multiple text semantics, and performing normalization processing to ensure that the sum of the probabilities is 1, thereby obtaining multiple background semantic probabilities; A second analysis module, the second analysis module is used to perform situation type analysis on the subject image to obtain a subject situation type; A third analysis module, the third analysis module is used to perform action prediction on the subject image to obtain a subject action image sequence, and perform subject action context analysis to obtain a subject action context type, and perform subject correction on the multiple text semantic probabilities in combination with the subject context type to obtain multiple subject semantic probabilities, wherein the subject correction includes performing weighted calculation on the subject semantic association coefficient and the action semantic association coefficient to obtain multiple semantic association coefficients, multiplying the multiple text semantic probabilities with the corresponding multiple semantic association coefficients, and performing normalization processing to ensure that the sum of the probabilities is 1, thereby obtaining multiple subject semantic probabilities; A fusion processing module, the fusion processing module is used to obtain multiple fused semantic probabilities based on the multiple background semantic probabilities and the multiple subject semantic probabilities through fusion processing, screen the text semantics with the largest fused semantic probability, and display them as semantic integration results, wherein the fusion processing includes obtaining the image area ratios of the subject image and the background image, configuring the subject weight and the background weight according to the image area ratios, using the subject weight and the background weight to perform weighted calculation on the multiple subject semantic probabilities and the multiple background semantic probabilities to obtain multiple fused semantic probabilities.

Citation Information

Patent Citations

  • Image character semantic segmentation method based on multi-neural network model fusion processing

    CN110390251A

  • Image-text sentiment analysis method based on multi-modal fusion

    CN113627550A