Graphic content review method based on multimodal large model

Abstract

The present invention discloses a method and system for reviewing graphic and text content based on a multimodal large model. Through the collaborative work of various modules, deep semantic understanding and cross-modal integration of unstructured data are achieved; semantic association graphs are generated by the multimodal large model, which improves the system's accuracy in identifying the context and information association of graphics, audio, and video, and solves the problem of inaccurate semantic understanding in existing systems; the multimodal data integration module realizes unified alignment of cross-modal data and hierarchical mapping of information density, context relevance, and semantic consistency, constructs a graphic and text association vector space, and thus eliminates data fragmentation in the review process; the content anomaly quantification module is combined with the intelligent control module to make multi-level review decisions by quantitatively calculating and comparing the review weight value Hqwz and the content anomaly index Cxzs, automatically triggering manual review and directly publishing automatically, effectively improving review efficiency and reducing the burden of manual review.

Description

Technical Field

[0001] The present invention relates to the field of content audit technology, and in particular to a graphic and text content audit method based on a multimodal large model. Background Art

[0002] Content review is a crucial step in the information dissemination process of news media. Standardizing the dissemination of text, images, audio, and video content can prevent public misinformation, enhance the guiding power and credibility of news media, and promote social stability. Existing manual content review methods offer high accuracy but low efficiency. Supervised deep learning models, which are specialized for specific scenarios, place a heavy burden on human reviewers during the review and reasoning process. Existing intelligent content review methods, based on specific datasets, lack active learning capabilities, making it difficult to acquire features beyond prior knowledge and unable to understand multimodal content. To address these challenges, we first designed a multimodal, large-scale content review method for specific content review scenarios. By leveraging the large model's ability to process unstructured data, such as text and images, and its contextual understanding capabilities, we enhanced its deep understanding of text and images. Secondly, we constructed four news content review methods for text and images in different scenarios and conducted comparative experiments. The experimental results show that using multimodal datasets for large-scale model training and testing improves the accuracy of intelligent content review and reduces the burden on content reviewers.

[0003] Based on the limitations of existing news content review systems and combined with the advantages of multimodal large models, the following technical shortcomings still exist in practical applications:

[0004] 1. Difficulty understanding unstructured data: Traditional news content review systems lack deep semantic understanding when processing unstructured data such as images, text, and audio. They find it difficult to accurately identify complex contexts and information relevance in images and text, resulting in limited review accuracy.

[0005] 2. Lack of multimodal data integration: Existing systems mostly process different types of data independently and lack cross-modal integration capabilities. For example, the relationship between images and text is difficult to understand in the same context, resulting in information fragmentation in the review of combined image and text content.

[0006] 3. Reliance on manual review, resulting in a heavy review burden: Although traditional systems have certain intelligent review capabilities, they still rely on manual review for multiple rounds of screening and judgment, which increases the workload of reviewers and makes it difficult to effectively improve review efficiency. Summary of the Invention

[0007] The purpose of the present invention is to provide a method for reviewing graphic and text content based on a multimodal large model to solve the above problems.

[0008] The present invention is achieved through the following technical solutions:

[0009] A method for reviewing text and image content based on a multimodal large model is applied to a text and image content review system based on a multimodal large model. The system includes an unstructured data processing module, a multimodal data integration module, a content analysis module, a content anomaly quantification module, and an intelligent control module.

[0010] The unstructured data processing module is used to perform semantic analysis on unstructured data including news graphics, audio, and video using a multimodal large model, generate a semantic association graph, extract the core context and information relevance in text and images through in-depth analysis, and construct a preliminary review data set;

[0011] The multimodal data integration module is used to integrate the graphic data, audio data and video data in a unified context, convert them into a graphic-text association vector space, and transmit them to the content analysis module for in-depth analysis of the content context and emotion;

[0012] The content analysis module is used to preliminarily determine the degree of match between text and image content based on the semantic association graph generated by the unstructured data processing module, generate a contextual factor Qycs, and analyze the potential sensitive content in each modal data to generate a sensitivity factor Gmyz; then, based on the contextual factor Qycs and the sensitivity factor Gmyz, perform a hierarchical analysis of the potential emotional factors, socially sensitive information, and potential misleading factors in the image and text content to generate a multimodal association factor Mmgyz;

[0013] The content anomaly quantification module calculates the audit weight value Hqwz based on the context factor Qycs, the sensitivity factor Gmyz, and the multimodal correlation factor Mmgyz; then obtains the system audit risk factor Rxz by calculation and associates it with the audit weight value Hqwz, thereby generating the final content anomaly index Cxzs;

[0014] The calculation formula for the audit weight value Hqwz is:

[0015] Hqwz=b1×Qycs+b2×Gmyz+b3×Mmgyz+B;

[0016] Among them, b1, b2 and b3 represent the preset weight coefficients of the situational factor Qycs, the sensitive factor Gmyz and the multimodal correlation factor Mmgyz respectively, and B represents the audit correction coefficient;

[0017] The intelligent control module is used to set the compliance threshold T and the audit risk threshold R, and compare the audit weight value Hqwz with the compliance threshold T to determine whether the current content meets the audit standards, and compare the audit risk threshold R with the content anomaly index Cxzs to obtain the final audit result, and selectively trigger manual review or automatic release control plans based on the audit results.

[0018] The method comprises the following steps:

[0019] Step 1: Use a multimodal large model to perform semantic analysis on unstructured data, including news graphics, audio, and video, to generate a semantic association graph. Through in-depth analysis, extract the core context and information relevance in the text and images, and construct a preliminary review dataset.

[0020] Step 2: After integrating the text, audio, and video data in a unified context, the data is converted into a text-image correlation vector space and transmitted to the content analysis module for in-depth analysis of the content context and emotion.

[0021] Step 3: Based on the semantic association graph generated by the unstructured data processing module, the degree of match between the text and image content is preliminarily determined to generate a contextual factor Qycs. The potential sensitive content in each modal data is analyzed to generate a sensitivity factor Gmyz. Then, based on the contextual factor Qycs and the sensitivity factor Gmyz, a hierarchical analysis is performed on the potential emotional factors, socially sensitive information, and potential misleading factors in the image and text content to generate a multimodal association factor Mmgyz.

[0022] Step 4: Calculate the audit weight value Hqwz based on the contextual factor Qycs, the sensitivity factor Gmyz, and the multimodal correlation factor Mmgyz. Then, obtain the system audit risk factor Rxz through calculation and associate it with the audit weight value Hqwz to generate the final content anomaly index Cxzs.

[0023] Step 5. Set the compliance threshold T and the audit risk threshold R, and compare the audit weight value Hqwz with the compliance threshold T to determine whether the current content meets the audit standards. Compare the audit risk threshold R with the content anomaly index Cxzs to obtain the final audit result, and selectively trigger manual review or automatic release control plans based on the audit results.

[0024] Preferably, the unstructured data processing module is first used to receive unstructured data of news images, audio and video to be reviewed, and perform format standardization processing on the unstructured data, including image resolution adjustment, text language normalization processing and audio and video transcoding;

[0025] The pre-trained network of the multimodal large model is then used to extract multi-level features from news images, audio, and video, including visual features, semantic features, and emotional features. These features are then normalized to generate a unimodal feature matrix. Based on the unimodal feature matrix, the multi-level features of news images, audio, and video are mapped in a multimodal association vector space, and a cross-modal semantic association graph is generated using the semantic embedding mechanism of the multimodal large model.

[0026] Secondly, we use cross-modal semantic association graphs to conduct a detailed analysis of the deep connections between text and images, extracting the core contextual elements in the text and images. We also simultaneously analyze the emotional states and expression trends in audio and video, identifying potential sensitive information, misleading information, and social influencing factors in news content in real time, and forming multimodal semantic association indicators.

[0027] Finally, a preliminary review dataset is generated.

[0028] Preferably, the multimodal data integration module is used to receive the preliminary review data set generated by the unstructured data processing module, and perform unified alignment processing of the graphic data, audio data and video data therein in terms of time sequence and content through a frame synchronization algorithm;

[0029] Then, a multimodal feature mapping algorithm is applied to fuse the contextual, emotional, and semantic features of images, text, audio, and video into a multimodal feature vector. A correlation matrix is generated based on the correlation between the modalities, and a hierarchical mapping is performed on information density, contextual relevance, and semantic consistency.

[0030] Subsequently, a graphic-text association vector space is constructed based on the association matrix, and the multimodal data is quantified in a unified semantic space to generate an integrated semantic association representation vector. At the same time, a context adjustment algorithm is used to correct the semantic deviation caused by modality differences.

[0031] Finally, the integrated image-text association vector space is transferred to the content analysis module.

[0032] Preferably, the content analysis module includes a preliminary discrimination unit, a multi-layer sensitivity analysis unit and a correlation factor generation unit;

[0033] The preliminary discrimination unit identifies semantic deviation-related data and content matching-related data by analyzing the semantic similarity and context consistency between the image and text; extracts the semantic similarity Ycy, context consistency Sjy, context-related deviation Qjl, and image-text emotion consistency Twq from the semantic deviation-related data and content matching-related data, performs dimensionless processing, and calculates the context factor Qycs using the following formula:

[0034]

[0035] The preset context threshold Q is compared with the context factor Qycs to evaluate the contextual relevance of text and images in the review scenario. The specific comparative evaluation content is as follows:

[0036] If the context factor Qycs ≥ the context threshold Q, it indicates that the contexts of the text and image match, and their semantics, emotions, and context are consistent. The review is passed and marked as "normal context association";

[0037] If the contextual factor Qycs is less than the contextual threshold Q, it indicates that the contexts of the text and image do not match, and their semantics, emotions, and context are inconsistent. The review will fail and further analysis will be conducted on the specific causes of the deviation, including potential sensitive content.

[0038] Preferably, the multi-layer sensitivity analysis unit is used to further perform in-depth analysis on the potential sensitive content of the graphic data, including analyzing the emotion-related data, social sensitivity-related data, and potential misleading-related data in the text and image, aggregating and dimensionlessly processing the emotion-related data, social sensitivity-related data, and potential misleading-related data in the analyzed text and image, and constructing a sensitive content data set; extracting the sensitive content data set and generating a sensitivity factor Gmyz through the following formula:

[0039]

[0040] Where Qmq represents the sentiment intensity coefficient in the sensitive content data set, Mgc represents the sensitive word index in the sensitive content data set, and Xwg represents the possibility of information misleading in the sensitive content data set.

[0041] Preferably, the correlation factor generating unit is used to quantify the correlation degree between emotion, context and sensitive content based on the context factor Qycs and the sensitivity factor Gmyz by fusing multi-level information in the image-text correlation vector space, and generate a multimodal correlation factor Mmgyz by calculating using the following formula;

[0042]

[0043] Preferably, the content anomaly quantification module includes a content evaluation unit and an anomaly index acquisition unit.

[0044] The content evaluation unit is used to construct a system audit risk factor Rxz. The semantic deviation degree Ycp and content consistency index Nry are obtained by extracting semantic deviation related data and content matching related data. The high sensitivity trigger rate Gmg is obtained by extracting sensitive content data sets. The system audit risk factor Rxz is calculated by combining the following formula:

[0045]

[0046] Preferably, the abnormality index obtaining unit is used to calculate and obtain the final content abnormality index Cxzs, and its specific calculation formula is as follows:

[0047]

[0048] Preferably, the intelligent control module compares and evaluates the audit weight value Hqwz with the compliance threshold T, and compares and evaluates the audit risk threshold R with the content anomaly index Cxzs, and specifically generates the following evaluation content:

[0049] Compliance standards comparison:

[0050] If the audit weight value Hqwz ≥ the compliance threshold T, it means that the content meets the audit standards and the system marks it as "compliant"

[0051] If the audit weight value Hqwz is less than the compliance threshold T, it indicates that the content does not meet the audit standards and is insufficiently compliant, requiring further review or adjustment.

[0052] Abnormal risk assessment:

[0053] If the content anomaly index Cxzs ≥ the audit risk threshold R, it indicates that the content has an abnormal risk, and the system generates an "abnormal content" mark, triggering the manual review process;

[0054] If the content anomaly index Cxzs is less than the audit risk threshold R, it indicates that the content does not have abnormal risk, and the system generates a "normal content" mark, and then enters the automatic publishing process;

[0055] When the evaluation results of the review weight value Hqwz and the content anomaly index Cxzs are marked as "compliant" and "normal content" at the same time, the content will be automatically marked as "passed" and published directly;

[0056] When the evaluation result of the review weight value Hqwz and the content anomaly index Cxzs is that the content does not meet the review standards or has an abnormal risk, the content will be automatically marked as "pending review" and the selective review process will be triggered.

[0057] It is used to set the compliance threshold T and the audit risk threshold R, and to determine whether the current content meets the audit standards, obtain the final audit results, and selectively trigger manual review or automatic release control plans based on the audit results.

[0058] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0059] 1. Through the collaborative work of the unstructured data processing module, the multimodal data integration module, the content analysis module, the content anomaly quantification module, and the intelligent control module, this invention provides a more comprehensive and in-depth semantic understanding capability for unstructured data. This solves the problem of inaccurate semantic understanding in existing systems when processing unstructured data, including images, text, audio, and video. It uses a large multimodal model to generate a semantic association graph and extracts the core context and information relevance in text and images through in-depth semantic analysis, thereby improving the system's recognition accuracy for complex contexts and information relevance, significantly enhancing audit accuracy.

[0060] 2. The present invention achieves unified integration of cross-modal data through a multimodal data integration module, uniformly aligns the timing and content of graphic, text, audio, and video data, generates multimodal feature vectors using a multimodal feature mapping algorithm, and hierarchically maps information density, contextual relevance, and semantic consistency in a unified semantic space through an association matrix to construct a graphic-text association vector space. This ensures contextual consistency and contextual relevance for different types of data, equips the system with information integration capabilities in cross-modal content review, and resolves the fragmentation problem that exists in traditional systems when reviewing combined graphic and text content.

[0061] 3. The present invention realizes an intelligent audit process through the quantitative calculation and comparison of the audit weight value Hqwz, the system audit risk factor Rxz and the content anomaly index Cxzs through the content anomaly quantification module and the intelligent control module, sets the compliance threshold T and the audit risk threshold R to make multi-level audit decisions. When the content anomaly index Cxzs is higher than the audit risk threshold R, it is marked as "abnormal content" to trigger the manual review process; when the audit weight value Hqwz is higher than the compliance threshold T, it is marked as "compliant" and enters the automatic release process. This mechanism avoids tedious manual review, reduces the burden on auditors, and significantly improves the audit efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0063] Figure 1 This is a schematic diagram of the framework structure of a graphic content review system based on a multimodal large model of the present invention;

[0064] Figure 2 This is a schematic diagram of the steps of a method for reviewing graphic content based on a multimodal large model according to the present invention; DETAILED DESCRIPTION

[0065] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the examples and accompanying drawings. The exemplary embodiments of the present invention and their descriptions are only used to explain the present invention and are not intended to limit the present invention. It should be noted that the present invention is already in the actual development and use stage.

[0066] Example 1

[0067] like Figure 1 As shown, this embodiment includes a graphic content review system based on a multimodal large model, including an unstructured data processing module, a multimodal data integration module, a content analysis module, a content anomaly quantification module, and an intelligent control module;

[0068] The unstructured data processing module is used to perform semantic analysis on unstructured data including news graphics, audio, and video using a multimodal large model, generate a semantic association graph, extract the core context and information relevance in text and images through in-depth analysis, and construct a preliminary review data set;

[0069] The multimodal data integration module is used to integrate the graphic data, audio data and video data in a unified context, convert them into a graphic-text association vector space, and transmit them to the content analysis module for in-depth analysis of the content context and emotion;

[0070] The content analysis module is used to preliminarily determine the degree of match between text and image content based on the semantic association graph generated by the unstructured data processing module, generate a contextual factor Qycs, and analyze the potential sensitive content in each modal data to generate a sensitivity factor Gmyz; then, based on the contextual factor Qycs and the sensitivity factor Gmyz, perform a hierarchical analysis of the potential emotional factors, socially sensitive information, and potential misleading factors in the image and text content to generate a multimodal association factor Mmgyz;

[0071] The content anomaly quantification module calculates the audit weight value Hqwz based on the context factor Qycs, the sensitivity factor Gmyz, and the multimodal correlation factor Mmgyz; then obtains the system audit risk factor Rxz by calculation and associates it with the audit weight value Hqwz, thereby generating the final content anomaly index Cxzs;

[0072] The calculation formula for the audit weight value Hqwz is:

[0073] Hqwz=b1×Qycs+b2×Gmyz+b3×Mmgyz+B;

[0074] Among them, b1, b2 and b3 represent the preset weight coefficients of the situational factor Qycs, the sensitive factor Gmyz and the multimodal correlation factor Mmgyz respectively, and B represents the audit correction coefficient;

[0075] The intelligent control module is used to set the compliance threshold T and the audit risk threshold R, and compare the audit weight value Hqwz with the compliance threshold T to determine whether the current content meets the audit standards, and compare the audit risk threshold R with the content anomaly index Cxzs to obtain the final audit result, and selectively trigger manual review or automatic release control plans based on the audit results.

[0076] In this embodiment, the unstructured data processing module uses a multimodal large model to perform semantic analysis on unstructured data including news graphics, audio and video, and generates a semantic association graph, which can deeply extract the core context and information correlation in text and images, thereby improving the accuracy of understanding complex contexts and helping to build a more complete preliminary review data set; the multimodal data integration module integrates graphic data, audio data and video data in a unified context, converts it into a graphic-text association vector space and transmits it to the content analysis module, so that the system has context consistency and information integration capabilities across modal data, and solves the problem of information fragmentation in the existing review system in the combination of graphics and text; the content analysis module can preliminarily determine the degree of matching between text and image content based on the semantic association graph generated by the unstructured data processing module, and generate context factor Qycs, sensitivity factor Gmyz and multimodal relationship. The linkage factor Mmgyz realizes the hierarchical analysis of potential emotional factors, social sensitive information and potential misleading factors, thereby improving the system's ability to conduct detailed review of contexts and sensitive content; the content anomaly quantification module calculates the review weight value Hqwz based on the context factor Qycs, the sensitive factor Gmyz and the multimodal correlation factor Mmgyz, and generates the final content anomaly index Cxzs by calculating the system review risk factor Rxz and correlating it with the review weight value Hqwz, which can effectively quantify the degree of content anomaly and provide a quantitative reference basis for the review; the intelligent control module sets the compliance threshold T and the review risk threshold R, and compares and evaluates the review weight value Hqwz with the compliance threshold T and the content anomaly index Cxzs with the review risk threshold R, selectively triggering manual review or automatic release plan, thereby realizing automation and intelligent control of the review process, effectively reducing the burden of manual review and improving review efficiency.

[0077] Example 2

[0078] The unstructured data processing module is first used to receive unstructured data of news images, audio and video to be reviewed, and perform format standardization processing on the unstructured data, including image resolution adjustment, text language normalization processing and audio and video transcoding;

[0079] The pre-trained network of the multimodal large model is then used to extract multi-level features from news images, audio, and video, including visual features, semantic features, and emotional features. These features are then normalized to generate a unimodal feature matrix. Based on the unimodal feature matrix, the multi-level features of news images, audio, and video are mapped in a multimodal association vector space, and a cross-modal semantic association graph is generated using the semantic embedding mechanism of the multimodal large model.

[0080] Secondly, we use cross-modal semantic association graphs to conduct a detailed analysis of the deep connections between text and images, extracting the core contextual elements in the text and images. We also simultaneously analyze the emotional states and expression trends in audio and video, identifying potential sensitive information, misleading information, and social influencing factors in news content in real time, and forming multimodal semantic association indicators.

[0081] Finally, a preliminary review dataset is generated.

[0082] The multimodal data integration module is used to receive the preliminary review data set generated by the unstructured data processing module, and perform unified alignment processing of the graphic data, audio data and video data therein in terms of time sequence and content through a frame synchronization algorithm;

[0083] Then, a multimodal feature mapping algorithm is applied to fuse the contextual, emotional, and semantic features of images, text, audio, and video into a multimodal feature vector. A correlation matrix is generated based on the correlation between the modalities, and a hierarchical mapping is performed on information density, contextual relevance, and semantic consistency.

[0084] Subsequently, a graphic-text association vector space is constructed based on the association matrix, and the multimodal data is quantified in a unified semantic space to generate an integrated semantic association representation vector. At the same time, a context adjustment algorithm is used to correct the semantic deviation caused by modality differences.

[0085] Finally, the integrated image-text association vector space is transferred to the content analysis module.

[0086] In this embodiment, the multimodal large-model graphic content review system realizes the standardization and feature extraction of unstructured data such as news graphics, audio and video through the hierarchical processing of the unstructured data processing module and the multimodal data integration module, so as to be able to deeply analyze the visual features, semantic features and emotional features in the multimodal data, and ensure that the review content has semantic and emotional integrity; wherein, the unstructured data processing module extracts the multi-level features in the news graphics, audio and video through the pre-trained network, and generates a single-modal feature matrix, and then generates a cross-modal semantic association graph, realizing the accurate analysis of the deep correlation between text, images, audio and video, extracting the core contextual elements and emotional states in the content, and effectively It can effectively identify potential sensitive information, misleading information and social influencing factors, thereby constructing a complete preliminary review data set; the multimodal data integration module aligns the graphic data, audio data and video data in terms of time sequence and content through the frame synchronization algorithm, and applies the multimodal feature mapping algorithm to generate multimodal feature vectors, so that the system has the contextual consistency and correlation matrix of cross-modal information, quantifies and hierarchically maps the information density, contextual relevance and semantic consistency in a unified semantic space, and eliminates the semantic deviation between modalities through the context adjustment algorithm, generating an integrated semantic association representation vector, thereby providing a complete and quantitative data foundation for the content analysis module, and improving the accuracy and comprehensiveness of the system's review in complex situations.

[0087] Example 3

[0088] The content analysis module includes a preliminary discrimination unit, a multi-layer sensitivity analysis unit and a correlation factor generation unit;

[0089] The preliminary discrimination unit identifies semantic deviation-related data and content matching-related data by analyzing the semantic similarity and context consistency between the image and text; extracts the semantic similarity Ycy, context consistency Sjy, context-related deviation Qjl, and image-text emotion consistency Twq from the semantic deviation-related data and content matching-related data, performs dimensionless processing, and calculates the context factor Qycs using the following formula:

[0090]

[0091] The preset context threshold Q is compared with the context factor Qycs to evaluate the contextual relevance of text and images in the review scenario. The specific comparative evaluation content is as follows:

[0092] If the context factor Qycs ≥ the context threshold Q, it indicates that the contexts of the text and image match, and their semantics, emotions, and context are consistent. The review is passed and marked as "normal context association";

[0093] If the contextual factor Qycs is less than the contextual threshold Q, it indicates that the contexts of the text and image do not match, and their semantics, emotions, and context are inconsistent. The review will fail and further analysis will be conducted on the specific causes of the deviation, including potential sensitive content.

[0094] The multi-layer sensitivity analysis unit is used to further perform in-depth analysis of the potential sensitive content of the graphic data, including analyzing the emotion-related data, social sensitivity-related data, and potentially misleading-related data in the text and image, aggregating and dimensionlessly processing the emotion-related data, social sensitivity-related data, and potentially misleading-related data in the analyzed text and image, and constructing a sensitive content data set; extracting the sensitive content data set and generating a sensitivity factor Gmyz using the following formula:

[0095]

[0096] Where Qmq represents the sentiment intensity coefficient in the sensitive content data set, Mgc represents the sensitive word index in the sensitive content data set, and Xwg represents the possibility of information misleading in the sensitive content data set.

[0097] The correlation factor generation unit is used to quantify the correlation degree between emotion, context and sensitive content based on the context factor Qycs and the sensitivity factor Gmyz by fusing multi-level information in the image-text correlation vector space, and calculate and generate the multimodal correlation factor Mmgyz by the following formula;

[0098]

[0099] In this embodiment, the content analysis module enables the system to comprehensively evaluate the contextual relevance, potential sensitivity and multimodal relevance of the graphic content through multi-level analysis of the preliminary judgment unit, the multi-layer sensitivity analysis unit and the correlation factor generation unit, thereby greatly improving the accuracy and comprehensiveness of the review; wherein, the preliminary judgment unit extracts semantic deviation-related data and content matching-related data such as semantic similarity Ycy, context consistency Sjy, contextual relevance deviation Qjl and graphic-text emotional consistency Twq by analyzing the semantic similarity and contextual consistency of the graphic and text, and performs dimensionless processing, and generates the contextual factor Qycs by calculation, effectively quantifying the contextual consistency of the graphic content, and determines whether the context matches by comparing with the preset context threshold Q, thereby realizing basic context evaluation; wherein, the semantic similarity Ycy is used to measure the text content and the image represented by the image. The similarity of the content at the semantic level is expressed by inputting the text into a pre-trained multimodal model to obtain a text vector representation; context consistency Sjy mainly measures the consistency of multi-dimensional information such as time, place, people or events between the text context and the image context, and is obtained by comparing the context feature set obtained from the text with the context feature set identified in the image or video; context association deviation Qjl is used to quantify the potential context mismatch between text and image or video. After aligning the text and image, the inconsistent parts of the "core scene or main object" are identified, and the inconsistent categories or numbers are counted to obtain the result; image-text emotion consistency Twq measures the degree of consistency between the emotion conveyed by the text and the emotion conveyed by the image, and is obtained by calculating the similarity between the text emotion distribution and the image emotion distribution;

[0100] The multi-layer sensitivity analysis unit deeply analyzes the emotions, social sensitivity and potential misleading data in texts and images, extracts parameters such as the emotion intensity coefficient Qmq, the sensitive word index Mgc and the possibility of information misleading Xwg to construct a sensitive content data set, and further calculates and generates the sensitivity factor Gmyz, quantifying the sensitivity of the graphic content and ensuring the audit's ability to identify potential risks. Among them, the emotion intensity coefficient Qmq is used to reflect the emotion or emotional intensity in the news content and is obtained through text emotion intensity analysis; the sensitive word index Mgc is mainly used to measure the frequency and intensity of sensitive information, sensitive terms, banned words or sensitive symbols in the text, and is obtained through sensitive dictionary and rule matching; the possibility of information misleading Xwg is used to assess whether there is a potential tendency in the news content to "confuse the audience", "falsify information" or "mislead the public", and is obtained through cross-modal comparison;

[0101] The correlation factor generation unit is based on the contextual factor Qycs and the sensitive factor Gmyz, and integrates the multi-level information in the image-text correlation vector space to generate a multimodal correlation factor Mmgyz. Through the correlation evaluation of emotions, contexts and sensitive content, it realizes the comprehensive review of image and text content, enabling the system to have the ability of cross-modal context association and risk identification, effectively improving the accuracy and depth of content review.

[0102] Example 4

[0103] The content anomaly quantification module includes a content evaluation unit and an anomaly index acquisition unit.

[0104] The content evaluation unit is used to construct a system audit risk factor Rxz. The semantic deviation degree Ycp and content consistency index Nry are obtained by extracting semantic deviation related data and content matching related data. The high sensitivity trigger rate Gmg is obtained by extracting sensitive content data sets. The system audit risk factor Rxz is calculated by combining the following formula:

[0105]

[0106] The anomaly index acquisition unit is used to calculate the final content anomaly index Cxzs. The specific calculation formula is as follows:

[0107]

[0108] The intelligent control module compares and evaluates the audit weight value Hqwz with the compliance threshold T, and compares and evaluates the audit risk threshold R with the content anomaly index Cxzs, and specifically generates the following evaluation content:

[0109] Compliance standards comparison:

[0110] If the audit weight value Hqwz ≥ the compliance threshold T, it means that the content meets the audit standards and the system marks it as "compliant"

[0111] If the audit weight value Hqwz is less than the compliance threshold T, it indicates that the content does not meet the audit standards and is insufficiently compliant, requiring further review or adjustment.

[0112] Abnormal risk assessment:

[0113] If the content anomaly index Cxzs ≥ the audit risk threshold R, it indicates that the content has an abnormal risk, and the system generates an "abnormal content" mark, triggering the manual review process;

[0114] If the content anomaly index Cxzs is less than the audit risk threshold R, it indicates that the content does not have abnormal risk, and the system generates a "normal content" mark, and then enters the automatic publishing process;

[0115] When the evaluation results of the review weight value Hqwz and the content anomaly index Cxzs are marked as "compliant" and "normal content" at the same time, the content will be automatically marked as "passed" and published directly;

[0116] When the evaluation result of the review weight value Hqwz and the content anomaly index Cxzs is that the content does not meet the review standards or has an abnormal risk, the content will be automatically marked as "pending review" and the selective review process will be triggered.

[0117] It is used to set the compliance threshold T and the audit risk threshold R, and to determine whether the current content meets the audit standards, obtain the final audit results, and selectively trigger manual review or automatic release control plans based on the audit results.

[0118] In this embodiment, the content anomaly quantification module provides the system with comprehensive content anomaly quantitative analysis capabilities through step-by-step calculations by the content evaluation unit and the anomaly index acquisition unit, thereby improving the accuracy and automation level of the audit; the content evaluation unit quantifies the degree of deviation in semantics and consistency of the graphic content by extracting the semantic deviation degree Ycp from the semantic deviation related data and the content consistency index Nry from the content matching related data, and calculates and generates the system audit risk factor Rxz in combination with the high sensitivity trigger rate Gmg in the sensitive content data set, thereby comprehensively quantifying the potential anomaly risk of the content and providing accurate risk data support for the subsequent audit process; the anomaly index acquisition unit calculates the content anomaly index Cxzs based on the system audit risk factor Rxz and the audit weight value Hqwz. It effectively quantifies the comprehensive performance of content in terms of compliance and risk, enabling the system to accurately evaluate the suitability of content release; the intelligent control module sets the compliance threshold T and the audit risk threshold R, and compares and evaluates the audit weight value Hqwz with the compliance threshold T, and the content anomaly index Cxzs with the audit risk threshold R, respectively, to achieve multi-level automated review and judgment of content. When the evaluation results of the audit weight value Hqwz and the content anomaly index Cxzs are both "compliant" and "normal content", the content is directly passed and automatically released; when the evaluation results show that the content does not meet the audit standards or there is an abnormal risk, the content is marked as "pending review" and the manual review process is triggered, achieving efficient control and intelligent decision-making of content release review, and effectively improving the accuracy and efficiency of system review.

[0119] Example 5

[0120] like Figure 2 As shown, a method for reviewing text content based on a multimodal large model, according to a system for reviewing text content based on a multimodal large model, includes the following steps:

[0121] Step 1: Use a multimodal large model to perform semantic analysis on unstructured data, including news graphics, audio, and video, to generate a semantic association graph. Through in-depth analysis, extract the core context and information relevance in the text and images, and construct a preliminary review dataset.

[0122] Step 2: After integrating the text, audio, and video data in a unified context, the data is converted into a text-image correlation vector space and transmitted to the content analysis module for in-depth analysis of the content context and emotion.

[0123] Step 3: Based on the semantic association graph generated by the unstructured data processing module, the degree of match between the text and image content is preliminarily determined to generate a contextual factor Qycs. The potential sensitive content in each modal data is analyzed to generate a sensitivity factor Gmyz. Then, based on the contextual factor Qycs and the sensitivity factor Gmyz, a hierarchical analysis is performed on the potential emotional factors, socially sensitive information, and potential misleading factors in the image and text content to generate a multimodal association factor Mmgyz.

[0124] Step 4: Calculate the audit weight value Hqwz based on the contextual factor Qycs, the sensitivity factor Gmyz, and the multimodal correlation factor Mmgyz. Then, obtain the system audit risk factor Rxz through calculation and associate it with the audit weight value Hqwz to generate the final content anomaly index Cxzs.

[0125] Step 5. Set the compliance threshold T and the audit risk threshold R, and compare the audit weight value Hqwz with the compliance threshold T to determine whether the current content meets the audit standards. Compare the audit risk threshold R with the content anomaly index Cxzs to obtain the final audit result, and selectively trigger manual review or automatic release control plans based on the audit results.

[0126] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for reviewing graphic content based on a multimodal large model, characterized by: Applied to a graphic content review system based on a multimodal large model, the system includes an unstructured data processing module, a multimodal data integration module, a content analysis module, a content anomaly quantification module, and an intelligent control module; The unstructured data processing module is used to perform semantic analysis on unstructured data including news graphics, audio, and video using a multimodal large model, generate a semantic association graph, extract the core context and information relevance in text and images through in-depth analysis, and construct a preliminary review data set; The multimodal data integration module is used to integrate the graphic data, audio data and video data in a unified context, convert them into a graphic-text association vector space, and transmit them to the content analysis module for in-depth analysis of the content context and emotion; The content analysis module is used to preliminarily determine the degree of match between text and image content based on the semantic association graph generated by the unstructured data processing module, generate a contextual factor Qycs, and analyze the potential sensitive content in each modal data to generate a sensitivity factor Gmyz; then, based on the contextual factor Qycs and the sensitivity factor Gmyz, perform a hierarchical analysis of the potential emotional factors, socially sensitive information, and potential misleading factors in the image and text content to generate a multimodal association factor Mmgyz; The content anomaly quantification module includes a content evaluation unit and an anomaly index acquisition unit; The content evaluation unit is used to construct a system audit risk factor Rxz. The semantic deviation degree Ycp and content consistency index Nry are obtained by extracting semantic deviation related data and content matching related data. The high sensitivity trigger rate Gmg is obtained by extracting sensitive content data sets. The system audit risk factor Rxz is calculated by combining the following formula: The content anomaly quantification module calculates the audit weight value Hqwz based on the context factor Qycs, the sensitivity factor Gmyz, and the multimodal correlation factor Mmgyz; then obtains the system audit risk factor Rxz by calculation and associates it with the audit weight value Hqwz, thereby generating the final content anomaly index Cxzs; The calculation formula for the audit weight value Hqwz is: Hqwz=b1×Qycs+b2×Gmyz+b3×Mmgyz+B; Among them, b1, b2 and b3 represent the preset weight coefficients of the situational factor Qycs, the sensitive factor Gmyz and the multimodal correlation factor Mmgyz respectively, and B represents the audit correction coefficient; The intelligent control module is used to set the compliance threshold T and the audit risk threshold R, and compare the audit weight value Hqwz with the compliance threshold T to determine whether the current content meets the audit standards, and compare the audit risk threshold R with the content anomaly index Cxzs to obtain the final audit result, and selectively trigger manual review or automatic release control plans based on the audit results. The method comprises the following steps: Step 1: Use a multimodal large model to perform semantic analysis on unstructured data, including news graphics, audio, and video, to generate a semantic association graph. Through in-depth analysis, extract the core context and information relevance in the text and images, and construct a preliminary review dataset. Step 2: After integrating the text, audio, and video data in a unified context, the data is converted into a text-image correlation vector space and transmitted to the content analysis module for in-depth analysis of the content context and emotion. Step 3: Based on the semantic association graph generated by the unstructured data processing module, the degree of match between the text and image content is preliminarily determined to generate a contextual factor Qycs. The potential sensitive content in each modal data is analyzed to generate a sensitivity factor Gmyz. Then, based on the contextual factor Qycs and the sensitivity factor Gmyz, a hierarchical analysis is performed on the potential emotional factors, socially sensitive information, and potential misleading factors in the image and text content to generate a multimodal association factor Mmgyz. Step 4: Calculate the audit weight value Hqwz based on the contextual factor Qycs, the sensitivity factor Gmyz, and the multimodal correlation factor Mmgyz. Then, obtain the system audit risk factor Rxz through calculation and associate it with the audit weight value Hqwz to generate the final content anomaly index Cxzs. Step 5. Set the compliance threshold T and the audit risk threshold R, and compare the audit weight value Hqwz with the compliance threshold T to determine whether the current content meets the audit standards. Compare the audit risk threshold R with the content anomaly index Cxzs to obtain the final audit result, and selectively trigger manual review or automatic release control plans based on the audit results.

2. The method for reviewing graphic content based on a multimodal large model according to claim 1, characterized in that: The unstructured data processing module is first used to receive unstructured data of news images, audio and video to be reviewed, and perform format standardization processing on the unstructured data, including image resolution adjustment, text language normalization processing and audio and video transcoding; The pre-trained network of the multimodal large model is then used to extract multi-level features from news images, audio, and video, including visual features, semantic features, and emotional features. These features are then normalized to generate a unimodal feature matrix. Based on the unimodal feature matrix, the multi-level features of news images, audio, and video are mapped in a multimodal association vector space, and a cross-modal semantic association graph is generated using the semantic embedding mechanism of the multimodal large model. Secondly, we use cross-modal semantic association graphs to conduct a detailed analysis of the deep connections between text and images, extracting the core contextual elements in the text and images. We also simultaneously analyze the emotional states and expression trends in audio and video, identifying potential sensitive information, misleading information, and social influencing factors in news content in real time, and forming multimodal semantic association indicators. Finally, a preliminary review dataset is generated.

3. The method for reviewing graphic content based on a multimodal large model according to claim 2, characterized in that: The multimodal data integration module is used to receive the preliminary review data set generated by the unstructured data processing module, and perform unified alignment processing of the graphic data, audio data and video data therein in terms of time sequence and content through a frame synchronization algorithm; Then, a multimodal feature mapping algorithm is applied to fuse the contextual, emotional, and semantic features of images, text, audio, and video into a multimodal feature vector. A correlation matrix is generated based on the correlation between the modalities, and a hierarchical mapping is performed on information density, contextual relevance, and semantic consistency. Subsequently, a graphic-text association vector space is constructed based on the association matrix, and the multimodal data is quantified in a unified semantic space to generate an integrated semantic association representation vector. At the same time, a context adjustment algorithm is used to correct the semantic deviation caused by modality differences. Finally, the integrated image-text association vector space is transferred to the content analysis module.

4. The method for reviewing graphic content based on a multimodal large model according to claim 3 is characterized by: The content analysis module includes a preliminary discrimination unit, a multi-layer sensitivity analysis unit and a correlation factor generation unit; The preliminary discrimination unit identifies semantic deviation-related data and content matching-related data by analyzing the semantic similarity and context consistency between the image and text; extracts the semantic similarity Ycy, context consistency Sjy, context-related deviation Qjl, and image-text emotion consistency Twq from the semantic deviation-related data and content matching-related data, performs dimensionless processing, and calculates the context factor Qycs using the following formula: The preset context threshold Q is compared with the context factor Qycs to evaluate the contextual relevance of text and images in the review scenario. The specific comparative evaluation content is as follows: If the context factor Qycs ≥ the context threshold Q, it indicates that the contexts of the text and image match, and their semantics, emotions, and context are consistent. The review is passed and marked as "contextual correlation normal"; If the contextual factor Qycs is less than the contextual threshold Q, it indicates that the contexts of the text and image do not match, and their semantics, emotions, and context are inconsistent. The review will fail and further analysis will be conducted on the specific causes of the deviation, including potential sensitive content.

5. The method for reviewing graphic content based on a multimodal large model according to claim 4 is characterized by: The multi-layer sensitivity analysis unit is used to further perform in-depth analysis of the potential sensitive content of the graphic data, including analyzing the emotion-related data, social sensitivity-related data, and potentially misleading-related data in the text and image, aggregating and dimensionlessly processing the emotion-related data, social sensitivity-related data, and potentially misleading-related data in the analyzed text and image, and constructing a sensitive content data set; extracting the sensitive content data set and generating a sensitivity factor Gmyz using the following formula: Where Qmq represents the sentiment intensity coefficient in the sensitive content data set, Mgc represents the sensitive word index in the sensitive content data set, and Xwg represents the possibility of information misleading in the sensitive content data set.

6. The method for reviewing graphic content based on a multimodal large model according to claim 5, characterized in that: The correlation factor generation unit is used to quantify the correlation degree between emotion, context and sensitive content based on the context factor Qycs and the sensitivity factor Gmyz by fusing multi-level information in the image-text correlation vector space, and calculate and generate the multimodal correlation factor Mmgyz by the following formula; 7. The method for reviewing graphic content based on a multimodal large model according to claim 6, characterized in that: The anomaly index acquisition unit is used to calculate the final content anomaly index Cxzs. The specific calculation formula is as follows:

8. The method for reviewing graphic content based on a multimodal large model according to claim 7, characterized in that: The intelligent control module compares and evaluates the audit weight value Hqwz with the compliance threshold T, and compares and evaluates the audit risk threshold R with the content anomaly index Cxzs, and specifically generates the following evaluation content: Compliance standards comparison: If the review weight value Hqwz ≥ the compliance threshold T, it means that the content meets the review standards and the system marks it as "compliant"; If the audit weight value Hqwz < the compliance threshold T, it indicates that the content does not meet the audit standards and is insufficiently compliant, requiring further review or adjustment. Abnormal risk assessment: If the content anomaly index Cxzs ≥ the audit risk threshold R, it indicates that the content has an abnormal risk. The system generates an "abnormal content" mark and triggers the manual review process. If the content anomaly index Cxzs is less than the audit risk threshold R, it indicates that the content does not have abnormal risk, and the system generates a "normal content" mark, and then enters the automatic publishing process; When the evaluation results of the review weight value Hqwz and the content anomaly index Cxzs are both marked as "compliant" and "normal content", the content will be automatically marked as "passed" and published directly; When the evaluation results of the review weight value Hqwz and the content anomaly index Cxzs indicate that the content does not meet the review standards or has an abnormality risk, the content will be automatically marked as "pending review" and the selective review process will be triggered; It is used to set the compliance threshold T and the audit risk threshold R, and to determine whether the current content meets the audit standards, obtain the final audit results, and selectively trigger manual review or automatic release control plans based on the audit results.

Images