Multi-modal content auditing method and system and computer storage medium

Abstract

The invention discloses a multi-modal content auditing method and system and a computer storage medium, and relates to the technical field of content auditing, and the method comprises the steps: collecting texts, pictures, audio and video resources on the Internet in real time; pre-training texts, pictures, audio and video resources through self-supervised learning to generate a multi-modal large model base; the multi-modal large model base adapts to the auditing task through fine tuning training, and a multi-modal model is obtained; utilizing a prompt project to guide the multi-modal model to generate an understanding question and answer data set as a fine tuning data set; based on the fine tuning data set, constructing a multi-modal auditing model through iterative fine tuning process training; according to the method, the auditing rule and the situation adaptability scoring function are dynamically generated, context perception analysis is carried out on the real-time auditing request, a more accurate auditing strategy is generated, the method has remarkable technical advantages in the field of multi-modal content auditing, and powerful support is provided for achieving an efficient and safe network environment.

Description

Technical Field

[0001] The present invention relates to the technical field of content auditing, and in particular to a multimodal content auditing method, system and computer storage medium. Background Art

[0002] With the rapid development of information technology, content review technology has become an important means to ensure the safety and health of the network environment. Especially on social media and video sharing platforms, with the rapid growth of user-generated content, traditional review methods face huge challenges. Early reviews mainly relied on manual review and rule-based automation systems. These systems can provide certain efficiency when processing single-modal data, but when faced with multi-modal data (such as text, pictures, audio and video), the efficiency and accuracy are insufficient. With the development of deep learning technology, deep learning-based review models have gradually emerged. However, existing models are mostly focused on a single modality and lack the ability to comprehensively understand and process multi-modal data.

[0003] The shortcomings of existing technologies are mainly reflected in two aspects: first, the existing single-modal audit system cannot fully explore the correlation between the modes when processing complex multimodal content, resulting in inaccurate audit results; second, the existing models often lack real-time performance and cannot adapt to the rapidly changing network environment, resulting in low audit efficiency. These shortcomings make it difficult for content audits to meet industry standards in terms of efficiency and accuracy. Therefore, in response to the real-time audit needs of multimodal data, a new audit method is urgently needed to improve the intelligence and automation level of audits. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a multimodal content review method to solve the problems of low efficiency and insufficient accuracy of multimodal content review.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides a multimodal content review method, which includes real-time collection of text, picture, audio and video resources on the Internet; Through self-supervised learning, pre-train text, image, audio and video resources to generate a multi-modal large model base; The multimodal large model base is fine-tuned to adapt to the audit task to obtain a multimodal model; Use prompt engineering to guide multimodal models to generate understanding question answering datasets as fine-tuning datasets; Based on the fine-tuning dataset, a multimodal audit model is constructed through iterative fine-tuning training. Receive audit requests through the multimodal audit model, activate the context-aware intelligent engine and dynamically generate audit rules based on the audit tasks to make audit decisions; Real-time collection of text, pictures, audio and video resources on the Internet. The specific steps are as follows: Select social media platforms and news websites as data sources; Based on the data source, Kafka is used to coordinate the distributed crawler nodes of the Scrapy framework to perform text, image, audio and video resource crawling; Preprocess the captured resources through OpenCV and FFmpeg; The pre-processed text, image, audio and video resources are encrypted and stored in Ceph, forming a data stream containing text, image, audio and video.

[0007] As a preferred solution of the multimodal content review method of the present invention, wherein: through self-supervised learning, pre-training of text, picture, audio and video resources is performed to generate a multimodal large model base, the specific steps are as follows: Integrate text, image, audio and video data streams, and annotate them to generate multimodal datasets with annotated information; Based on multimodal datasets, the CLIP model is used as the basis, self-supervised learning strategy is used, and contrastive learning method is used for pre-training; After pre-training, adjust the hyperparameters to optimize performance, and evaluate the generalization ability of the CLIP model through the validation set; Save the evaluated and optimized CLIP model as a multimodal large model base.

[0008] As a preferred solution of the multimodal content audit method described in the present invention, the multimodal large model base is fine-tuned and trained to adapt to the audit task to obtain a multimodal model. The specific steps are as follows: Initialize the multimodal large model base, using the CLIP model as a starting point; Based on the CLIP model and multimodal datasets, an adaptive parameterized audit function is created by combining exponential decay integral normalization and complex summation function, which is defined as: ; in, is the feature vector of the content to be reviewed, is the corresponding expected output, is the sample size, For the The learning rate parameter of the sample, is the sample index variable, is the attenuation factor To measure the The feature vector of the content to be reviewed of samples with expected output The distance metric function of the difference between Indicates time Integrate infinitesimal increments of Using the learning rate parameter and attenuation factor , calculate the adaptive parameterized audit function score for each sample in the multimodal dataset; Based on the adaptive parameterized audit function scores, the CLIP model is fine-tuned using multimodal datasets; In each fine-tuning, the adaptive parameterized audit function score is recalculated to evaluate the adaptability of the CLIP model to the audit task and to dynamically adjust the learning rate. and attenuation factor parameter; The CLIP model after multiple fine-tuning training and evaluation is used as a multimodal model.

[0009] As a preferred solution of the multimodal content review method of the present invention, wherein: using prompt engineering to guide the multimodal model to generate an understanding question and answer dataset as a fine-tuning dataset, the specific steps are as follows: Create prompt templates in the form of task instructions, contextual information, and expected responses through enhancement engineering; Apply the prompt template to the multimodal model to generate question-answer pairs; Apply graph neural network (GNN) to capture the relationship between different modal data in multimodal datasets and construct graph structures; Optimize question-answer pairs through graph structure and generate question-answer pair dataset; An evaluation mechanism is introduced to evaluate the question-answer pair dataset using a validation set. Adjust the evaluated question-answer pair dataset into a high-quality question-answer pair dataset through feedback; Integrate a high-quality question-answer pair dataset as a fine-tuning dataset.

[0010] As a preferred solution of the multimodal content audit method described in the present invention, wherein: based on the fine-tuning data set, a multimodal audit model is constructed through iterative fine-tuning process training, and the specific steps are as follows: Based on the multimodal large model base, select the basic model and fine-tuning dataset for the current audit task; Load the fine-tuning dataset into the training environment; Iteratively fine-tune the multimodal audit model using the loaded fine-tuning dataset; The generalization ability of the fine-tuned multimodal audit model is evaluated using an independent validation set.

[0011] As a preferred solution of the multimodal content review method described in the present invention, wherein: receiving the review request through the multimodal review model, activating the context-aware intelligent engine and dynamically generating the review rules in combination with the review task, and performing the review judgment, the specific steps are as follows: The multimodal review model receives the review request and parses the text, image, audio, and video metadata of the review request; Based on the parsed audit request metadata, activate the context-aware intelligent engine to analyze the contextual characteristics of the audit request; Introducing contextual adaptability scoring function , guiding the generation and adjustment of audit rules, the expression is: ; in, is the context adaptability scoring function, is the time range, is the time decay coefficient, is the information filtering function, is the reference time point; According to the results of the situational adaptability scoring function and the audit task objectives, the audit rules are dynamically generated using the adaptive parameterized audit function; Apply audit rules to make audit decisions on audit requests, and refine the application of audit rules based on context information provided by the situational adaptability scoring function.

[0012] In a second aspect, the present invention provides a multimodal content review system, including a data collection module, a model training module, a task fine-tuning module, a prompt generation module, an iterative optimization module and an intelligent review module; the data collection module is used to collect text, pictures, audio and video resources on the Internet in real time; the model training module is used to pre-train text, pictures, audio and video resources through self-supervised learning to generate a multimodal large model base; the task fine-tuning module is used to fine-tune the multimodal large model base to adapt to the review task through fine-tuning training to obtain a multimodal model; the prompt generation module is used to use prompt engineering to guide the multimodal model to generate an understanding question and answer dataset as a fine-tuning dataset; The iterative optimization module is used to build a multimodal audit model based on the fine-tuning data set through an iterative fine-tuning process training; the intelligent audit module is used to receive audit requests through the multimodal audit model, activate the context-aware intelligent engine and dynamically generate audit rules in combination with the audit tasks to make audit decisions.

[0013] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the multimodal content review method as described in the first aspect of the present invention.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the multimodal content review method as described in the first aspect of the present invention.

[0015] The beneficial effects of the present invention are as follows: the present invention collects various types of data (text, pictures, audio and video) in real time, generates a multimodal large model base using self-supervised learning technology, and integrates the pre-training process of multimodal data, so that the multimodal model can fully understand the correlation between different modalities and apply it in audit tasks. Secondly, by dynamically generating audit rules and situational adaptability scoring functions, the multimodal audit model can perform context-aware analysis on real-time audit requests and generate more accurate audit strategies. In addition, the introduction of iterative fine-tuning and feedback mechanisms enables the multimodal audit model to be continuously optimized in practical applications, further improving its performance. This series of innovative steps gives the present invention significant technical advantages in the field of multimodal content auditing, and helps to achieve an efficient and secure network environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0017] Figure 1 This is a flow chart of the multimodal content review method in Example 1; Figure 2 This is a module diagram of the multimodal content review system in Example 1. DETAILED DESCRIPTION

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0019] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0020] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0021] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a multimodal content review method, comprising the following steps: S1. Real-time collection of text, pictures, audio and video resources on the Internet.

[0022] Furthermore, social media platforms and news websites were selected as data sources; It should be noted that these platforms are characterized by fast information updates, wide coverage, and high user participation, and therefore can provide a large amount of diverse raw data. This selection ensures the freshness and relevance of the data, while also reflecting changes in social hot spots and public opinion. It achieves the effect of obtaining high-quality, timely data resources, providing a basis for subsequent data analysis and application.

[0023] Based on the data source, Kafka is used to coordinate the distributed crawler nodes of the Scrapy framework to perform text, image, audio and video resource crawling; Specifically, by introducing Kafka to coordinate the distributed crawler nodes under the Scrapy framework, an efficient and scalable multi-type resource crawling mechanism is implemented. As a message middleware, Kafka receives crawler tasks from upstream and reasonably distributes these tasks to various Scrapy nodes. Each task includes the target URL and the text, picture, audio or video to be crawled. The reliability and sequence of message transmission are guaranteed, so that crawler tasks can reasonably distribute the load among multiple nodes, and maintain stability and efficiency even when facing massive data.

[0024] Preprocess the captured resources through OpenCV and FFmpeg; Preprocessing includes converting raw materials into standardized formats. OpenCV is used for image recognition and processing, such as removing noise and adjusting resolution. FFmpeg is used for audio and video encoding and decoding and format conversion. This step helps eliminate redundant information in the data, improves subsequent storage and retrieval efficiency, and provides a unified standard data format for data analysis.

[0025] The pre-processed text, image, audio and video resources are encrypted and stored in Ceph, forming a data stream containing text, image, audio and video.

[0026] Ceph is known for its high scalability, reliability and flexibility, and is particularly suitable for large-scale data storage needs. Data encryption ensures the privacy and integrity of information and prevents unauthorized access or leakage. In addition, building a data stream that integrates multiple media types such as text, images, audio and video facilitates later data integration and analysis and supports cross-media content association research.

[0027] S2. Pre-train text, image, audio, and video resources through self-supervised learning to generate a multimodal large model base.

[0028] Furthermore, text, image, audio and video data streams are integrated and annotated to generate multimodal datasets with annotated information; Based on multimodal datasets, the CLIP model is used as the basis, self-supervised learning strategy is used, and contrastive learning method is used for pre-training; Specifically, the CLIP model is used and self-supervised learning strategies are introduced, especially contrastive learning methods, for pre-training, so that features can be effectively extracted even without a large amount of labeled data. The CLIP model can process text and images at the same time, self-supervised learning reduces the dependence on expensive labeled data, and contrastive learning methods help learn to distinguish between similar and different samples.

[0029] After pre-training, adjust the hyperparameters to optimize performance, and evaluate the generalization ability of the CLIP model through the validation set; Among them, hyperparameter adjustment is the key process to find the optimal configuration, and the validation set evaluation provides important feedback on the actual performance. This makes the CLIP model perform well on the training data and has excellent predictive ability on new data. It improves the practical application value and enables the CLIP model to better serve the problem solving in the real world.

[0030] Save the evaluated and optimized CLIP model as a multimodal large model base.

[0031] The optimal multimodal large model base can be used for transfer learning in the future, quickly adapting to new application scenarios or specific field problems, greatly shortening the R&D cycle. In addition, it can also serve as a research platform to promote further exploration and development in academia and industry. Ultimately, it lays a solid foundation for subsequent application innovation and technological progress, and promotes the development of multimodal artificial intelligence technology.

[0032] S3. The multimodal large model base is adapted to the audit task through fine-tuning training to obtain a multimodal model.

[0033] Furthermore, the multimodal large model base is initialized, using the CLIP model as a starting point; It should be noted that the pre-trained CLIP model was selected as the initial base for the multimodal large model, which enabled a quick start and the use of a CLIP model that has been trained on a large number of text-image pairs. The CLIP model can be used as an ideal starting point for text-image association tasks due to its powerful cross-modal representation capabilities.

[0034] Based on the CLIP model and multimodal dataset, an adaptive parameterized audit function is created by combining exponential decay integral normalization and complex summation function. The expression is: ; in, is the feature vector of the content to be reviewed, is the corresponding expected output, is the sample size, For the The learning rate parameter of the sample, is the sample index variable, is the attenuation factor, To measure the The feature vector of the content to be reviewed of samples with expected output The distance metric function of the difference between Indicates time Integrate infinitesimal increments of in, is the feature vector of the content to be reviewed. The content to be reviewed refers to various types of media files or data that need to be evaluated and checked during the review. These contents may include but are not limited to text, images, audio, and video. By introducing a custom adaptive parameterized review function , which enables a refined evaluation of the CLIP model output. This function takes into account the learning rate parameters of all samples , attenuation factor And the feature vector of the content to be reviewed with expected output The distance measure between .

[0035] Preferably, this design allows the CLIP model to dynamically adjust the review criteria according to the specific task requirements, thereby more accurately reflecting the importance of different samples. This step provides a flexible and precise mechanism to measure actual performance and provides a quantitative basis for subsequent optimization.

[0036] Using the learning rate parameter and attenuation factor , calculate the adaptive parameterized audit function score for each sample in the multimodal dataset; Based on the adaptive parameterized audit function scores, the CLIP model is fine-tuned using multimodal datasets; Among them, fine-tuning training uses a deep learning framework to build a training environment, ensures support for distributed training and GPU acceleration, and sets key parameters such as the initial learning rate, batch size, optimizer type, and loss function. The loss value is calculated based on the score of the adaptive parameterized audit function. Samples with higher scores in the adaptive parameterized audit function will be given greater weights, thereby affecting the loss function.

[0037] In each fine-tuning, the adaptive parameterized audit function score is recalculated to evaluate the adaptability of the CLIP model to the audit task and to dynamically adjust the learning rate. and attenuation factor parameter; Specifically, the adaptability of the CLIP model is evaluated based on the adaptive parameterized audit function score, and the learning rate is dynamically adjusted accordingly. and attenuation factor , realizing intelligent management of the training process. In this step, the CLIP model can automatically adjust key parameters according to actual performance to find the best configuration. This not only improves training efficiency, but also enhances the pertinence and accuracy of specific audit tasks. The dynamic adjustment strategy helps to overcome the problem of overfitting or underfitting caused by fixed parameters, ensuring that the CLIP model can maximize the fit of the target task while maintaining good generalization ability.

[0038] The CLIP model after multiple fine-tuning training and evaluation is used as a multimodal model.

[0039] It should be noted that the CLIP model gradually improves its prediction ability and decision-making logic by repeatedly learning feedback from the real world. Each fine-tuning is based on the results of the previous one to ensure continuous progress towards the optimal solution. After multiple fine-tuning training and evaluation, the multimodal model will have excellent audit capabilities and is suitable for various complex multimodal scenarios.

[0040] S4. Use prompt engineering to guide the multimodal model to generate a comprehension question answering dataset as a fine-tuning dataset.

[0041] Furthermore, by improving engineering, prompt templates are created in the form of task instructions, contextual information, and expected responses; It should be noted that by improving the engineering to create task instructions and context information, we ensure that the generated question-answer pairs have clear goals and structures. This process makes the multimodal model more accurate in understanding user intent and reduces ambiguity, thereby improving the quality and relevance of the generated question-answer pairs in subsequent steps.

[0042] Apply the prompt template to the multimodal model to generate question-answer pairs; Apply graph neural network GNN to capture the relationship between multimodal data and build graph structure; Among them, the application of graph neural network GNN can efficiently capture the complex relationship between different modal data and build a comprehensive graph structure. The graph structure not only improves the connectivity of multimodal data, but also provides a basis for subsequent optimization, which helps to improve the understanding and reasoning ability of multimodal models.

[0043] Optimize question-answer pairs through graph structure and generate question-answer pair dataset; An evaluation mechanism is introduced to evaluate the question-answer pair dataset using a validation set. It should be noted that BlenderBot is used as an evaluation tool combined with automatic evaluation indicators, and the output of BlenderBot is combined with the automatic evaluation indicator ROUGE to form a hybrid evaluation mechanism, which improves the quality control level of the question-answering dataset and provides reliable feedback information for training. By combining the automatic evaluation indicators with the evaluation of the dialogue model, the efficiency of the automatic evaluation is retained, the flexibility and deep understanding ability of the dialogue model are introduced, and the objectivity and subjectivity of the evaluation are balanced. This method makes full use of the advantages of existing technologies, reflects the application of innovative thinking, and has high practical value.

[0044] Preferably, an evaluation mechanism is introduced and a validation set is used to evaluate the question-answer pair dataset to ensure that the generated dataset meets high quality standards. This enables the development team to identify and correct potential problems in a timely manner, improve the effectiveness and usability of the question-answer pair, and thus provide users with better services.

[0045] Adjust the evaluated question-answer pair dataset into a high-quality question-answer pair dataset through feedback; Integrate a high-quality question-answer pair dataset as a fine-tuning dataset.

[0046] Specifically, by integrating high-quality question-answer pair datasets that have been strictly screened and optimized as the basis for fine-tuning the multimodal model, the performance of the multimodal model has been fine-tuned. During the fine-tuning process, the multimodal model can learn the subtle differences and deep features contained in these high-quality data, thereby significantly improving its understanding and generation capabilities. Ultimately, the effect of optimizing performance is achieved, making the multimodal model have higher accuracy and adaptability in specific tasks, and fully prepared for practical applications.

[0047] S5. Based on the fine-tuning dataset, a multimodal audit model is constructed through an iterative fine-tuning process.

[0048] Furthermore, based on the multimodal large model base, the base model and fine-tuning dataset for the current audit task are selected; Load the fine-tuning dataset into the training environment; It should be noted that the selected fine-tuning dataset is loaded into the training environment to achieve seamless integration of data and multimodal models. This process not only simplifies the transition from data preparation to training, but also ensures the consistency and integrity of data transmission. The configuration of the training environment allows flexible adjustment of parameter settings to adapt to datasets of different sizes, thereby improving training efficiency, reducing time waste and technical difficulties caused by improper data processing, and laying a solid foundation for subsequent fine-tuning training.

[0049] Iteratively fine-tune the multimodal audit model using the loaded fine-tuning dataset; Specifically, the loaded fine-tuning dataset is used to iteratively fine-tune the multimodal audit model to achieve a gradual improvement in performance. During each iteration, the multimodal audit model can learn more specific details about the audit task and continuously optimize its prediction ability. Iterative fine-tuning helps the multimodal audit model overcome initial deviations and gradually approach the optimal solution. Ultimately, it achieves the beneficial effect of enhancing accuracy and robustness, enabling it to perform well on specific audit tasks and more accurately respond to complex and changing practical application scenarios.

[0050] The generalization ability of the fine-tuned multimodal audit model is evaluated using an independent validation set.

[0051] The role of the independent validation set is to provide a set of unseen data samples to test whether the multimodal audit model has truly mastered the core features of the audit task, rather than just memorizing the training data. This step ensures that the multimodal audit model can not only perform well on the training data, but also maintain stable and reliable performance on new and unknown data.

[0052] S6. Receive audit requests through the multimodal audit model, activate the context-aware intelligent engine, and dynamically generate audit rules based on the audit tasks to make audit decisions.

[0053] Furthermore, the multimodal review model receives the review request and parses the text, image, audio, and video metadata of the review request; The multimodal audit model receives and parses the text, image, audio and video metadata of the audit request. After receiving the audit request, the multimodal audit model uses natural language processing (NLP), computer vision (CV) and audio processing technology to parse the text, image, audio and video metadata, extract key information such as text content, image objects, speech-to-text and video behavior, and generate structured audit request metadata, which provides comprehensive and detailed data support for subsequent situational analysis and audit decisions, thereby enhancing the accuracy and reliability of the audit process. A comprehensive understanding of complex audit requests is achieved. This step is not limited to processing a single type of data, but covers multiple sources of information such as text, images, audio and video, ensuring that the audit process can fully consider all relevant information. Ultimately, the comprehensiveness of the audit decision is enhanced, providing a solid foundation for more accurate situational analysis in the future.

[0054] Based on the parsed audit request metadata, activate the context-aware intelligent engine to analyze the contextual characteristics of the audit request; Specifically, we analyze the contextual features of review requests and use multimodal review models to compare historical data or known patterns, identify the similarities and differences between the current review request and previous cases, evaluate the risk level of the review request based on the parsed information, and try to infer the motivation or purpose behind the user submitting the review request, which helps to more accurately judge the true intention of the request.

[0055] Introducing contextual adaptability scoring function , guiding the generation and adjustment of audit rules, the expression is: ; in, is the context adaptability scoring function, is the time range, is the time decay coefficient, Is an information filtering function used to measure the moment Lower situation characteristics and review requests The matching degree between them can be realized by selecting an appropriate algorithm according to the actual situation. is the reference time point; It should be noted that the context adaptability scoring function is introduced to quantitatively evaluate the degree of consistency between the review request and the current context characteristics. The time decay coefficient in the formula and information filtering functions Working together, the scoring can dynamically reflect the changes in matching degree in different time periods. This mechanism helps to maintain sensitivity and responsiveness in changing situations. It optimizes the generation and adjustment of audit rules, ensures that the rules always meet the latest situational needs, and enhances flexibility and adaptability.

[0056] According to the results of the situational adaptability scoring function and the audit task objectives, the audit rules are dynamically generated using the adaptive parameterized audit function; The better method is to combine the results of the situational adaptability scoring function with the audit task objectives, use the adaptive parameterized audit function to dynamically generate audit rules, and realize the personalized customization of audit rules. This method flexibly adjusts the audit standards according to the specific situation characteristics, ensures the consistency of the rules, and improves the pertinence. It improves the efficiency and accuracy of the audit, so that the audit process can respond quickly and be executed accurately, meeting the diverse audit needs.

[0057] Apply audit rules to make audit decisions on audit requests, and refine the application of audit rules based on context information provided by the situational adaptability scoring function.

[0058] Among them, the context information provided by the situational adaptability scoring function refines the application of rules and realizes the refined management of audit decisions. It not only relies on preset rules, but also makes full use of real-time situation analysis results to ensure that each audit rule can be appropriately applied to specific situations. It improves the effect of audit quality, ensures that the audit results are fair and reasonable, and at the same time improves user satisfaction and credibility.

[0059] Specifically, based on the adaptive parameterized audit function, the audit rules are dynamically generated. Combined with the context information provided by the situational adaptability scoring function, the audit request is matched and screened with the audit rules. Preliminary screening is performed based on the text content and video tags to identify keywords that may involve sensitive topics. Through the screened audit requests, the audit rules are refined and adjusted based on the context information provided by the situational adaptability scoring function. For routine situations, the audit conclusion is automatically generated based on the refined audit rules. For complex situations, these audit requests are marked and manually intervened for review, and the audit judgment is completed efficiently and accurately. The real-time situation analysis results are fully utilized to ensure that each audit rule can be appropriately applied to specific situations, improving the accuracy of the audit.

[0060] The present embodiment also provides a multimodal content review system, including: a data collection module, a model training module, a task fine-tuning module, a prompt generation module, an iterative optimization module and an intelligent review module; the data collection module is used to collect text, pictures, audio and video resources on the Internet in real time; the model training module is used to pre-train text, pictures, audio and video resources through self-supervised learning to generate a multimodal large model base; the task fine-tuning module is used to adapt the multimodal large model base to the review task through fine-tuning training to obtain a multimodal model; the prompt generation module is used to use the prompt engineering to guide the multimodal model to generate an understanding question and answer data set as a fine-tuning data set; the iterative optimization module is used to construct a multimodal review model based on the fine-tuning data set through an iterative fine-tuning process training; the intelligent review module is used to receive review requests through the multimodal review model, activate the context-aware intelligent engine and dynamically generate review rules in combination with the review task to make a review decision.

[0061] This embodiment also provides a computer device suitable for a multimodal content review method, comprising: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the multimodal content review method proposed in the above embodiment.

[0062] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0063] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the multimodal content review method proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, disk or optical disk.

[0064] In summary, the present invention collects various types of data (text, pictures, audio and video) in real time, uses self-supervised learning technology to generate a multimodal large model base, and integrates the pre-training process of multimodal data, so that the multimodal model can fully understand the correlation between different modalities and apply it in audit tasks. Secondly, by dynamically generating audit rules and situational adaptability scoring functions, the multimodal audit model can perform context-aware analysis on real-time audit requests and generate more accurate audit strategies. In addition, the introduction of iterative fine-tuning and feedback mechanisms enables the multimodal audit model to be continuously optimized in practical applications and further improve its performance. This series of innovative steps gives the present invention significant technical advantages in the field of multimodal content auditing and helps to achieve an efficient and secure network environment.

[0065] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multimodal content review method, characterized by: include, Collect text, pictures, audio and video resources on the Internet in real time; Through self-supervised learning, pre-train text, image, audio and video resources to generate a multi-modal large model base; The multimodal large model base is fine-tuned to adapt to the audit task to obtain a multimodal model; Use prompt engineering to guide multimodal models to generate understanding question answering datasets as fine-tuning datasets; Based on the fine-tuning dataset, a multimodal audit model is constructed through iterative fine-tuning training. Receive audit requests through the multimodal audit model, activate the context-aware intelligent engine and dynamically generate audit rules based on the audit tasks to make audit decisions; Select social media platforms and news websites as data sources; Based on the data source, Kafka is used to coordinate the distributed crawler nodes of the Scrapy framework to perform text, image, audio and video resource crawling; Preprocess the captured text, images, audio and video resources through OpenCV and FFmpeg; The pre-processed text, image, audio and video resources are encrypted and stored in Ceph, forming a data stream containing text, image, audio and video.

2. The multimodal content review method according to claim 1, characterized in that: The self-supervised learning is used to pre-train text, pictures, audio and video resources to generate a multi-modal large model base. The specific steps are as follows: Integrate text, image, audio and video data streams, and annotate them to generate multimodal datasets with annotated information; Based on multimodal datasets, the CLIP model is used as the basis, self-supervised learning strategy is used, and contrastive learning method is used for pre-training; After pre-training, adjust the hyperparameters to optimize performance, and evaluate the generalization ability of the CLIP model through the validation set; Save the evaluated and optimized CLIP model as a multimodal large model base.

3. The multimodal content review method according to claim 2, characterized in that: The multimodal large model base is fine-tuned to adapt to the audit task to obtain a multimodal model. The specific steps are as follows: Initialize the multimodal large model base, using the CLIP model as a starting point; Based on the CLIP model and multimodal dataset, an adaptive parameterized audit function is created by combining exponential decay integral normalization and complex summation function. The expression is: ； in, is the feature vector of the content to be reviewed, is the corresponding expected output, is the total number of samples, For the The learning rate parameter of the sample, is the sample index variable, is the attenuation factor, To measure the The feature vector of the content to be reviewed of samples With expected output The distance metric function of the difference between Indicates time Integrate infinitesimal increments of Using the learning rate parameter and attenuation factor , calculate the adaptive parameterized audit function score for each sample in the multimodal dataset; Based on the adaptive parameterized audit function scores, the CLIP model is fine-tuned using multimodal datasets; In each fine-tuning, the adaptive parameterized audit function score is recalculated to evaluate the adaptability of the CLIP model to the audit task and to dynamically adjust the learning rate. and attenuation factor parameter; The CLIP model after multiple fine-tuning training and evaluation is used as a multimodal model.

4. The multimodal content review method according to claim 3 is characterized in that: The method of using prompt engineering to guide the multimodal model to generate an understanding question-answering dataset as a fine-tuning dataset has the following specific steps: Create prompt templates in the form of task instructions, contextual information, and expected responses through enhancement engineering; Apply the prompt template to the multimodal model to generate question-answer pairs; Apply graph neural network (GNN) to capture the relationship between different modal data in multimodal datasets and construct graph structures; Optimize question-answer pairs through graph structure and generate question-answer pair dataset; An evaluation mechanism is introduced to evaluate the question-answer pair dataset using a validation set. Adjust the evaluated question-answer pair dataset into a high-quality question-answer pair dataset through feedback; Integrate a high-quality question-answer pair dataset as a fine-tuning dataset.

5. The multimodal content review method according to claim 4, characterized in that: Based on the fine-tuning dataset, the multimodal audit model is constructed through iterative fine-tuning training. The specific steps are as follows: Based on the multimodal large model base, select the basic model and fine-tuning dataset for the current audit task; Load the fine-tuning dataset into the training environment; Iteratively fine-tune the multimodal audit model using the loaded fine-tuning dataset; The generalization ability of the fine-tuned multimodal audit model is evaluated using an independent validation set.

6. The multimodal content review method according to claim 5, characterized in that: The multimodal audit model receives the audit request, activates the context-aware intelligent engine, and dynamically generates the audit rules in combination with the audit task to make the audit decision. The specific steps are as follows: The multimodal review model receives the review request and parses the text, image, audio, and video metadata of the review request; Based on the parsed audit request metadata, activate the context-aware intelligent engine to analyze the contextual characteristics of the audit request; Introducing contextual adaptability scoring function , guiding the generation and adjustment of audit rules, the expression is: ； in, is the context adaptability scoring function, is the time range, is the time decay coefficient, is the information filtering function, is the reference time point; According to the results of the situational adaptability scoring function and the audit task objectives, the audit rules are dynamically generated using the adaptive parameterized audit function; Apply audit rules to make audit decisions on audit requests, and refine the application of audit rules based on context information provided by the situational adaptability scoring function.

7. A multimodal content review system, based on the multimodal content review method according to any one of claims 1 to 6, characterized in that: Including data collection module, model training module, task fine-tuning module, prompt generation module, iterative optimization module and intelligent review module; The data collection module is used to collect text, pictures, audio and video resources on the Internet in real time; The model training module is used to pre-train text, image, audio and video resources through self-supervised learning to generate a multi-modal large model base; The task fine-tuning module is used to fine-tune the multimodal large model base to adapt to the audit task and obtain a multimodal model; The prompt generation module is used to use prompt engineering to guide the multimodal model to generate an understanding question-answering dataset as a fine-tuning dataset; The iterative optimization module is used to construct a multimodal audit model through iterative fine-tuning process training based on the fine-tuning dataset; The intelligent audit module is used to receive audit requests through a multimodal audit model, activate the context-aware intelligent engine, dynamically generate audit rules in combination with the audit tasks, and make audit decisions.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multimodal content review method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal content review method described in any one of claims 1 to 6 are implemented.

Images