Multimodal media content analysis closed-loop self-learning system and method based on contradictory sample pool

Abstract

The invention provides a contradictory sample pool-based multi-modal media content analysis closed-loop self-learning system and method, and the system comprises a multi-modal analysis module which is used for obtaining media content and extracting multi-modal features of media content segments through a multi-modal analysis model; the cross-modal consistency judgment module is used for performing inter-modal consistency judgment on the multi-modal features of the fragments, storing inconsistent fragments into a contradictory sample pool, and outputting labels of consistent fragments; the manual correction module is used for receiving correction labels of the contradiction samples; the model local updating module is used for locally updating the multi-modal analysis model according to the correction result; and the label output module is used for summarizing the labels of the fragments or correcting the labels and generating unified label output of the media content. According to the method, contradictions in multi-modal content analysis can be automatically found and corrected, and the understanding precision of the model for complex media content and the consistency of label output are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal data processing technology, and in particular to a closed-loop self-learning system and method for multimodal media content analysis based on a contradictory sample pool. Background Technology

[0002] With the development of media content analysis technology, multimodal fusion analysis of video, audio, and text has received increasing attention. Traditional media content analysis systems often separate processing modules such as image recognition, speech recognition, and text recognition, allowing each to work independently without cross-modal collaboration and verification. For example, in existing technologies, video analysis typically includes object detection and image classification of the scene, speech-to-speech (ASR) of the audio to obtain text, further analysis of the text, object recognition (OCR) of text in the scene, and face detection and recognition. However, these modular results are often output in their own forms, lacking a unified semantic alignment process, which may lead to inconsistencies and biases in the description of the same media content.

[0003] Inconsistencies in multimodal content are very common in reality. For example, discrepancies between video subtitles and on-screen content, or conflicts between on-screen and voice-over descriptions, can cause difficulties for content moderation or retrieval. Research shows that contradictory multimodal inputs often exist in the real world, but many existing datasets and models assume input consistency and lack evaluation for cross-modal inconsistency detection. Current mainstream multimodal models have significant limitations in identifying such cross-modal inconsistencies. This means that when facing complex multimodal scenarios, relying solely on offline-trained static models is insufficient for timely and efficient error detection and correction.

[0004] Active learning techniques are widely used to improve model performance with limited annotation costs. Typical strategies include selecting samples based on uncertainty or inconsistencies among model committees. For example, traditional methods involve having multiple models vote on unlabeled samples, selecting the sample with the greatest prediction discrepancy for manual annotation. However, in multimodal analysis scenarios, there is currently no specific method to utilize semantic contradictions between different modal analysis results to select difficult examples for annotation learning. Therefore, how to use cross-modal inconsistencies as an information source to automatically discover model blind spots and introduce human feedback remains an unresolved challenge in current technologies.

[0005] In recent years, some cross-modal pre-trained models (such as OpenAI's CLIP) have been able to map images and text to a unified embedding space, making cross-modal similarity calculation and retrieval possible. Some researchers have attempted to use these models to cluster large numbers of images and generate descriptive labels for each cluster using large models (such as GPT-4). This method achieves unsupervised clustering optimization and label refinement to some extent. However, this is an offline, static data analysis process, lacking a real-time model update mechanism. In other words, it does not involve a closed-loop process of immediate model optimization through manual correction when inference errors occur. Furthermore, traditional methods focus more on improving clustering or retrieval performance, relying on retraining or offline fine-tuning to improve the accuracy of the classification model itself, failing to achieve continuous online self-improvement.

[0006] In summary, existing technologies for multimodal media content analysis have the following shortcomings: First, they lack a cross-modal consistency checking mechanism, leading to potential inconsistencies in descriptions from different modalities that go undetected. Second, they lack effective means to utilize these contradictory samples, failing to transform them into beneficial training increments for the model. Third, the tag outputs lack standardized integration, often presented in different formats, reducing the efficiency of downstream applications in utilizing the analysis results. Therefore, a new technical solution is urgently needed that can introduce a closed-loop self-learning mechanism into the multimodal analysis system, continuously improving the accuracy and consistency of multimodal content understanding through contradiction detection, human feedback, and model updates. Summary of the Invention

[0007] This invention provides a closed-loop self-learning system and method for multimodal media content analysis based on a contradictory sample pool, in order to solve the problems in the background art.

[0008] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0009] According to a first aspect of the present invention, a closed-loop self-learning system for multimodal media content analysis based on a contradictory sample pool is provided, comprising: The multimodal analysis module is used to acquire media content, segment the media content, and extract the multimodal features of each segment using a multimodal analysis model. The cross-modal consistency discrimination module is used to judge the intermodal consistency of the multimodal features of each segment. If any feature of any modality is inconsistent with other features, the segment is marked as a contradictory sample and stored in the contradictory sample pool; otherwise, the label of the segment is determined based on the multimodal features of the segment and output to the label output module. The contradictory sample pool is used to store contradictory samples and their identification information, multimodal features, and preliminary labels for automatic identification. The manual correction module provides an interface for manual review and annotation, and receives correction labels for contradictory samples; The model local update module is used to locally update the multimodal analysis model based on contradictory samples and correction labels; The tag output module is used to summarize the tags output by the cross-modal consistency discrimination module or the correction tags by the manual correction module for each segment, and generate a unified tag output for the media content.

[0010] In one embodiment of the present invention, the multimodal analysis model includes: The video frame image embedding extraction unit is used to extract the video frame image embedding vector of a segment. The speech-to-text embedding extraction unit is used to extract the ASR audio-to-text embedding vector of a segment; An optical character recognition text embedding extraction unit is used to extract the OCR recognition text embedding vector of a segment; The face feature embedding extraction unit is used to extract the face recognition embedding vector of the segment.

[0011] In one embodiment of the present invention, the intermodal consistency judgment process in the cross-modal consistency discrimination module includes: Based on the extracted multimodal features, intermodal relationship modeling and semantic reasoning are performed in a unified embedding representation space, and intermodal differences are inferred; the semantic reasoning includes similarity matching, semantic reference recognition, and context-assisted judgment. If the result of semantic reasoning between any two modalities indicates that there are differences between the two modalities, it is considered that the segment has features of any modality that are inconsistent with other features.

[0012] In one embodiment of the present invention, the specific working process of the model local update module includes: Based on the manually corrected contradictory samples, the multimodal analysis model is updated locally or the embedded representation space is directly optimized and adjusted.

[0013] In one embodiment of the present invention, the local parameter update includes: performing several iterations of updating the multimodal analysis model using only contradictory samples and correction labels.

[0014] In one embodiment of the present invention, the tag output module includes: The tag aggregation unit is used to aggregate the tags output by the cross-modal consistency discrimination module or the correction tags of the manual correction module for each segment; The large language model standardization unit is used to perform semantic normalization and format standardization on the tags summarized by the tag aggregation unit, and output standardized and unified tag results.

[0015] According to a second aspect of the present invention, a closed-loop self-learning method for multimodal media content analysis based on a contradictory sample pool is provided, comprising: S1. Obtain the media content to be analyzed and segment the media content. S2. For any given segment, use a multimodal analysis model to extract the multimodal features of the segment; S3. Perform intermodal consistency judgment on the multimodal features of the segment. If any feature of any modality is inconsistent with other features, the segment is marked as a contradictory sample, and the contradictory sample, its identification information, multimodal features and preliminary label are stored in the contradictory sample pool; otherwise, generate a label for the segment based on the multimodal features of the segment. S4. Manually correct the contradictory samples in the contradictory sample pool and generate correction labels; S5. The multimodal analysis model is locally updated using the correction labels; S6. Repeat S2~S5 until all segments are labeled by the cross-modal consistency discrimination module or by the correction label of the manual correction module, and generate a unified label output for the media content.

[0016] In one embodiment of the present invention, the extracted multimodal features in S2 include video frame image embedding vectors, audio ASR transcription text embedding vectors, image OCR text embedding vectors, and human face embedding vectors.

[0017] In one embodiment of the present invention, step S3, the intermodal consistency determination specifically includes: Based on the extracted multimodal features, intermodal relationship modeling and semantic reasoning are performed in a unified embedding representation space, and intermodal differences are inferred; the semantic reasoning includes similarity matching, semantic reference recognition, and context-assisted judgment. If the result of semantic reasoning between any two modalities indicates that there are differences between the two modalities, it is considered that the segment has features of any modality that are inconsistent with other features.

[0018] In one embodiment of the present invention, step S5, which involves locally updating the multimodal analysis model using the correction label, specifically includes: Based on the manually corrected contradictory samples and correction labels, the multimodal analysis model is updated locally, or the embedded representation space is directly optimized and adjusted.

[0019] Compared with existing technologies, the beneficial effects of adopting the above technical solution are as follows: 1. Automatic Detection of Cross-Modal Inconsistencies: This invention can automatically identify and mark inconsistencies between images and text / audio descriptions in video content. Compared to traditional methods relying on manual inspection, the system can quickly locate blind spots in model cognition, improving the efficiency and accuracy of problem detection.

[0020] 2. Introduction of Closed-Loop Self-Learning: Through a contradictory sample pool and a manual feedback mechanism, this invention transforms difficult samples into incremental training data for the model, enabling continuous model improvement. This closed-loop self-learning approach is similar to "inconsistent sampling" in active learning but is innovatively applied to multimodal scenarios. The model is continuously fine-tuned and optimized during actual operation, thus gradually adapting to new content distributions and maintaining improved analytical performance.

[0021] 3. Unified Tag Output Through Multimodal Fusion: The tag output module of this invention ensures the semantic and format consistency of the final output tag information. By standardizing tags through LLM, information from different modalities is fused, preventing them from contradicting each other and facilitating downstream retrieval and content understanding. Compared to the disorganized information output from multiple channels in traditional media asset systems, this invention significantly improves the usability and professional quality of the output results.

[0022] 5. Secure and efficient private cloud deployment: This invention can be deployed in a private cloud environment, ensuring secure control over sensitive media data while utilizing cloud computing resources for real-time analysis of large-scale media content. Compared to purely local deployment, the private cloud solution provides near-online processing capabilities without data leaving the intranet, meeting the dual requirements of performance and security for industrial applications.

[0023] In summary, this invention achieves a significant improvement in multimodal media content analysis technology through the ingenious design of a contradictory sample pool and a closed-loop learning mechanism. It can significantly improve the analysis accuracy and output quality in complex multimodal scenarios and has broad application prospects. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0025] Figure 1 This is a schematic diagram of a closed-loop self-learning system for multimodal media content analysis based on a contradictory sample pool, according to an embodiment of the present invention.

[0026] Figure 2 This is a flowchart of a closed-loop self-learning system for multimodal media content analysis based on a contradictory sample pool, according to an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0028] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar modules or modules having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. Rather, embodiments of the present invention include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0029] To address the following technical issues in current media content analysis systems: 1. Modal fragmentation: In traditional systems, multimodal modules such as ASR, OCR, image recognition, and face recognition are independent of each other and cannot work together to determine whether the same content is consistent, which can easily lead to label conflicts or omissions.

[0030] 2. Errors cannot be traced back: Once the model automatically generates labels and an error occurs, it cannot be detected or automatically corrected, and the system lacks an error correction mechanism.

[0031] 3. Difficulty in achieving data-driven closed-loop optimization: Traditional systems use static tag libraries and lack the ability to continuously optimize the model with new data.

[0032] 4. Inability to identify potentially conflicting samples: Existing models cannot automatically detect high-risk samples such as "modal inconsistency" and "semantic drift".

[0033] 5. Low annotation efficiency: Manual intervention points are scattered and lack specificity, failing to support precise focusing on weak areas of the model.

[0034] This invention proposes a closed-loop self-learning system and method for multimodal media content analysis based on a contradictory sample pool. It addresses at least one of the aforementioned technical problems. This invention can automatically detect inconsistencies in different modal information within media content such as videos, centrally manage these contradictory samples, and continuously optimize the model through manual correction, thereby achieving a closed-loop self-improvement of analytical capabilities. This improves the accuracy and consistency of multimodal content analysis results, reduces manual annotation workload, and produces standardized, unified tags for content retrieval and downstream applications.

[0035] Please refer to Figure 1The multimodal media content analysis closed-loop self-learning system based on the contradiction sample pool mainly includes a multimodal analysis module, a cross-modal consistency discrimination module, a contradiction sample pool, a model local update module, and a label output module. The multimodal analysis module is mainly used to acquire media content and segment it; and to extract multimodal features of each segment using a multimodal analysis model. The cross-modal consistency judgment module is mainly used to judge the intermodal consistency of the multimodal features of each segment. If any feature of any modality is inconsistent with other features, the segment is marked as a contradictory sample and stored in the contradictory sample pool. Otherwise, the tag of the segment is determined based on its multimodal features and output to the tag output module. The contradictory sample pool is mainly used to store contradictory samples and their identification information, multimodal features, and automatically identified preliminary tags. The manual correction module is mainly used to provide a manual review and annotation interface and receive correction tags for contradictory samples. The model local update module is mainly used to perform local updates to the multimodal analysis model based on contradictory samples and correction tags. The tag output module is mainly used to summarize the tags output by the cross-modal consistency judgment module or the correction tags by the manual correction module for each segment and generate a unified tag output for the media content.

[0036] Specifically, in the multimodal analysis module, the multimodal analysis model can extract multimodal features of media content segments. In this embodiment, the multimodal analysis model mainly includes a video frame image embedding extraction unit, a speech-to-text embedding extraction unit, and an optical character recognition (OCR) text embedding extraction unit. The video frame image embedding extraction unit extracts the video frame image embedding vector of the segment; the speech-to-text embedding extraction unit extracts the ASR audio-to-text embedding vector of the segment; the OCR text embedding extraction unit extracts the OCR text embedding vector of the segment; and the face feature embedding extraction unit extracts the face recognition embedding vector of the segment.

[0037] After acquiring multimodal features using the multimodal analysis module, the cross-modal consistency judgment module is needed to determine the intermodal consistency of the features, i.e., to compare whether there are semantic conflicts in the same segment of content across different modalities. In this embodiment, the multimodal embedding vectors are used as the core carrier of the basic semantic expression, and intermodal relationship modeling and semantic reasoning are performed in a unified embedding representation space; thereby determining whether there are differences between modalities. This embodiment also provides a method for determining intermodal consistency for a specific segment, which includes: (1) Similarity matching: The similarity score between the image frame embedding vector and the ASR text embedding vector is calculated using a pre-trained cross-modal model, and it is determined whether the preset threshold requirement is met. Generally, when the similarity score is less than the preset threshold, it indicates that the two are inconsistent.

[0038] (2) Semantic reference: Compare whether there are semantic differences between the ASR text embedding vector and the OCR text embedding vector. If there are differences, it means that the two are inconsistent.

[0039] (3) Identity matching: compare whether the identity in the face embedding vector matches the identity in the ASR text embedding vector. If they do not match, it means that they are inconsistent.

[0040] (4) Context-assisted judgment: Compare whether there are consistent faces in the previous and next shots of the video. When the reasoning result of any of the above judgment methods (1) to (4) is that there is a difference between modalities, it is considered that the feature of any modality of the segment is inconsistent with other features. At this time, it can be considered that there is a conflict in the multimodal features of the segment, and the segment is stored as a conflict sample in the conflict sample pool. If there is no conflict in the multimodal features of the segment, the label can determine the label of the segment and output it to the label output module. It should be noted that the embedding vector in this embodiment is only used as the basis for cross-modal semantic representation. The entire intermodal consistency judgment process is essentially a multi-factor reasoning process, rather than just a pure numerical similarity judgment. In practical applications, the aforementioned judgment methods can be added or reduced. In addition to the aforementioned methods, the consistency judgment between each modality can also be combined with text entity parsing (such as name extraction) and confidence analysis (such as OCR confidence threshold) for multi-factor judgment. In this embodiment, instead of directly relying on the recognition results of a single modality to generate labels, a unified multimodal vector space is constructed in the cross-modal consistency discrimination module. The geometric / semantic consistency between the embedding vectors of each modality is analyzed. When multiple modalities are highly coupled, they can be considered as common support for the semantic label, thus automatically identifying the label of the segment. Specifically, the multimodal features of the segment are inferred in the same embedding representation space to determine the label of the segment (e.g., scene name, character role). In some embodiments, the segment label generation process can be standardized by LLM or rule templates. For contradictory samples, an automatically recognized preliminary label is generated.

[0041] In this embodiment, the contradictory sample pool essentially corresponds to the set of blind spots in the model's cognitive space embedded in the feature space, and is used to focus on difficult samples that the model cannot currently process accurately. Specifically, the contradictory sample pool stores the identification information, multimodal features, automatically identified preliminary labels, and consistency discrimination information of each contradictory sample for subsequent manual correction and model updates. In one embodiment, the contradictory sample pool can organize and retrieve contradictory samples based on the embedded vector space.

[0042] Because the multimodal features of contradictory samples conflict, their labels are also inaccurate. This embodiment proposes a manual correction module to correct contradictory samples. Specifically, the manual correction module provides a manual review and annotation interface, which can receive correction labels or annotation feedback from manual correction. In practical applications, manual correction personnel call the manual review and annotation interface to read the original media content of the segment and the modal analysis results given by the model, correct the model analysis errors according to the actual situation, and obtain the corresponding correction labels or descriptive information for the segment.

[0043] After correcting contradictory samples, the multimodal analysis model needs to be locally updated based on the manual correction results. This ensures the updated model can correctly analyze similar content that previously showed contradictions. Using a local update approach enhances the model's ability to identify contradictory samples and similar samples, while largely maintaining the original model's performance on other samples.

[0044] Specifically, the present invention performs local updates to the multimodal analysis model through a local model update module. Specifically, based on contradictory samples and correction labels obtained by the manual correction module, at least a portion of the multimodal analysis model parameters in the multimodal analysis module are locally updated. That is, the selected layer is fine-tuned and trained on the original model with a pre-set small learning rate, using only contradictory samples and their correction labels (optionally including a small amount of original training data) for several iterations, thereby improving the model's ability to identify the category or scene to which the contradictory samples belong without significantly affecting the model's learning performance on other data. In one embodiment, the embedding representation space can also be directly optimized and adjusted.

[0045] Furthermore, the tag output module of this embodiment includes a tag aggregation unit and a large language model standardization unit. The tag aggregation unit aggregates the tags output by the cross-modal consistency discrimination module or the corrected tags by the manual correction module for each segment. The large language model standardization unit, by fusing and normalizing the tags from multiple modalities across all segments using a language model larger than a standard language model, generates a consistent and complete semantic description tag result for the media content.

[0046] In one embodiment, when the large language model generates standardized output based on multimodal information, a preset prompt template is used to ensure that it covers all modal elements and has a uniform format; if the output is a free text description, the sentences smoothly integrate visual and auditory information; if the output is a structured label, each field follows a predetermined standard and eliminates conflicts that may be caused by different modal analyses.

[0047] Large language model (LLM) can ensure the fusion of information from different modalities, preventing them from operating independently and providing them to users in a standardized, structured form (such as standard classification terms or descriptive phrases). For example, the system can integrate the identity of people recognized from images, the dialogue content obtained from ASR, and the text extracted from OCR, and use LLM to express them as a unified description or a set of standard labels, avoiding repetition and ambiguity.

[0048] In one embodiment, the system of this invention can be deployed in a private cloud server environment, utilizing cloud-based distributed computing resources to achieve real-time analysis and model updates of multimodal content, ensuring that media data does not leave the internal network to guarantee data security.

[0049] Please continue to refer to this. Figure 1 In this embodiment, the modules are connected to form a closed-loop system. The overall process includes: the multimodal analysis module outputs initial analysis results -> detects contradictory samples -> contradictory samples enter the contradictory sample pool -> manual correction provides correction labels -> local model updates improve capabilities -> the updated model is reused for subsequent analysis. When new media content arrives, the system repeats the above process continuously to complete the media content analysis.

[0050] The technical solution of this invention will be further described below in conjunction with typical use cases of intelligent media asset management systems for television stations. Those skilled in the art should understand that the following embodiments are used to illustrate the working process of this invention and do not constitute a limitation on the scope of protection. This invention is not only applicable to media asset management for television stations, but also applicable to any scenario requiring multimodal content analysis and self-learning, including but not limited to news organizations, video platforms, educational institutions, monitoring content management systems, and content review systems.

[0051] Specifically, the overall system is deployed in the TV station's private cloud environment and consists of a multimodal analysis module, a cross-modal consistency discrimination module, a contradictory sample pool, a manual correction module, a model local update module, and a label output module. Among these, The multimodal analysis module receives the program video to be processed and segments it, or directly receives the corresponding segments; it extracts the visual features, audio features, and OCR text features of the segments, and can also extract facial features for identity verification. The above features are represented in the form of embedding vectors, constituting the multimodal feature set of the segment.

[0052] The cross-modal consistency discrimination module performs semantic comparisons on multimodal features within the same time segment, such as comparing the semantic similarity between images and ASR text, and the consistency between OCR text and speech text. When at least one modality has a significant semantic conflict with other modalities, it is judged as "inconsistent," and the inconsistent segment is stored as a contradictory sample in the contradictory sample pool.

[0053] The conflict sample pool is used to centrally manage conflicting content segments for subsequent manual correction and model updates.

[0054] Manual Correction Module: Through the manual review and annotation interface of the manual correction module, TV station catalogers can manually review contradictory samples via the review interface in the media asset system. Catalogers can confirm whether there are indeed contradictions and correct, supplement, or delete tags. The correction results are written into the system as correction tags for that segment, and the sample status is updated.

[0055] The model local update module updates the parameters of the multimodal analysis model or consistency discrimination model within a small range based on manually corrected contradictory samples. The update process can employ fine-tuning, incremental learning, or embedded spatial calibration to gradually adapt the model to the characteristics of television program content, thereby improving multimodal consistency discrimination and label recognition capabilities.

[0056] The tag output module merges the tags from the multimodal analysis module with manually corrected tags, and outputs structured tags or natural language descriptions after being uniformly formatted by a large language model. These tags can be used for program cataloging, retrieval, or other business processes.

[0057] The system can complete closed-loop operation in the TV station's media asset business, that is, form a closed loop of "automatic analysis - contradiction detection - manual correction - model update", so that the model's capabilities can be continuously improved in actual business operations.

[0058] In scenarios such as news, feature, and program production, this invention can automatically process a large number of inbound programs. After automatically generating preliminary tags, catalogers only need to review conflicting samples, significantly reducing the workload of manual cataloging. At the same time, as the results of manual review are continuously fed back and used for local model updates, the system's adaptability to the station's program style, host characteristics, dialect accents, subtitle styles, etc., gradually increases, achieving continuous self-learning.

[0059] To more clearly illustrate the multimodal media content analysis closed-loop self-learning system based on contradictory sample pools of the present invention, this embodiment further provides a multimodal media content analysis closed-loop self-learning method based on contradictory sample pools, the specific steps of which are as follows: S1. Obtain the media content to be analyzed and segment the media content.

[0060] In this embodiment, the media content is segmented by time, and each segment is then processed.

[0061] S2. For any segment, use a multimodal analysis model to extract the multimodal features of the segment.

[0062] In this embodiment, the extraction of video frame image embedding vectors, audio ASR transcribed text embedding vectors, image OCR text embedding vectors, and human face embedding vectors is mainly completed through the multimodal analysis module.

[0063] S3. Perform intermodal consistency judgment on the multimodal features of the segment. If any feature of any modality is inconsistent with other features, the segment is marked as a contradictory sample. The contradictory sample and its identification information, multimodal features and preliminary labels automatically identified in the intermodal consistency judgment process are stored in the contradictory sample pool. Otherwise, generate a label for the segment based on the multimodal features of the segment.

[0064] Specifically, in the process of determining intermodal consistency, the embedding vectors of multiple modalities serve as the core carriers of semantic expression. Relationship modeling and semantic reasoning between modalities are performed within a unified embedding representation space; subsequently, it is determined whether differences exist between modalities. This embodiment provides the following determination methods: (1) Calculate the similarity score between the image frame embedding vector and the ASR text embedding vector using the pre-trained cross-modal model, and determine whether the preset threshold requirement is met; (2) Compare whether the ASR text embedding vector and the OCR text embedding vector satisfy semantic consistency; (3) Compare whether the identity in the face embedding vector matches the identity in the ASR text embedding vector; It should be noted that the above judgment methods can be added or reduced as needed. For example, consistency judgment between different modalities can also be combined with text entity parsing (such as name extraction), confidence analysis (such as OCR confidence threshold) and contextual information (such as continuity on the time axis) to make multi-factor judgments.

[0065] When the reasoning result of any intermodal judgment means that there is a difference between modalities, it is considered that the feature of any modality in the segment is inconsistent with other features, and thus the segment can be considered a contradictory sample.

[0066] For segments that are judged to be consistent across modalities, their multimodal features can be inferred within the same embedding representation space to determine the segment's label (e.g., scene name, character role). Segments judged to be inconsistent are treated as contradictory samples, and an initial label for automatic identification is generated.

[0067] S4. Manually correct the contradictory samples in the contradictory sample pool and generate correction labels.

[0068] In practical applications, professionals can view the original media content of the segment and the modal analysis results provided by the model, and assign the correct labels to the incorrect or inconsistent parts.

[0069] S5. Use the correction labels to locally update the multimodal analysis model.

[0070] In this embodiment, the multimodal analysis model is locally updated based on the manually corrected contradictory samples and correction labels, or the embedding space representation is directly adjusted.

[0071] S6. Repeat S2~S5 until all segments are labeled by the cross-modal consistency discrimination module or by the correction module, and generate a unified label output for the media content.

[0072] In this embodiment, the labels output by the cross-modal consistency discrimination module and the corrected labels by the manual correction module are combined, and the labels are semantically normalized and format standardized by the Large Language Model (LLM) to finally form a unified label set output.

[0073] Finally, the multimodal media content analysis closed-loop self-learning system and method based on contradictory sample pool proposed in this invention differs from existing technologies in the following ways: 1. Distinguishing from Traditional Active Learning: Many active learning approaches exist for manually labeling and optimizing models for uncertain samples. Typical methods include selecting samples based on entropy values ​​of model prediction probabilities and query-by-committee methods based on inconsistencies in voting among multiple models. However, these methods are primarily applied to single modalities or single tasks (such as difficult-to-classify samples in image classification). This invention extends the concept of "sample inconsistency" to a cross-modal dimension: utilizing semantic contradictions between images, text, and speech to discover model weaknesses. This cross-modal contradiction sampling strategy is not documented in existing active learning literature. Existing technologies do not teach or imply that analysis results from different modalities should be compared to select training samples, thus fundamentally different in guiding principle. Furthermore, this invention performs local model fine-tuning after discovering contradictions, unlike the common global retraining in active learning. Local updates ensure efficiency while avoiding forgetting, representing a specific optimization for online multimodal systems and a non-obvious improvement.

[0074] 2. Distinguishing from CLIP clustering and label refinement methods: Currently, some cross-modal pre-trained models (such as CLIP) are used to cluster unlabeled data, and then large language models are used to automatically name labels for each cluster. These methods (e.g., using CLIP embeddings to cluster massive amounts of images, and then GPT-4 generating descriptions based on cluster features) embody an unsupervised data processing approach. However, compared to this invention, there is a fundamental difference: First, it lacks a closed loop of human feedback—the clustering and label generation processes are completely automatic and offline, and the model parameters are not updated due to human intervention; second, it focuses on exploring the overall structure of the dataset, rather than correcting model errors. The contradictory sample pool of this invention does not select representative samples based on cluster density, but specifically captures cases where the model makes errors—a completely different sample selection criterion. Therefore, even though existing technologies know that CLIP+LLM can be used to generate labels, they have not considered using it for online model learning, let alone a solution to improve model performance through a semantic contradiction closed loop. This invention uses LLM for label normalization output, and its innovation lies in the preceding contradiction detection and model update stages. The overall solution is a non-trivial combination and is creative.

[0075] 3. Distinguishing from traditional media asset multimodal processing workflows: Traditional video content analysis or media asset management systems mostly adopt a pipeline architecture, with each module performing its own function. For example, image analysis is used to detect scenes and people, then speech recognition is used to obtain subtitles, and finally, a simple manual check is performed. Generally, no module verifies whether the image and speech descriptions are consistent. Furthermore, there is no mechanism to feed back errors discovered during runtime to the model. Some systems may have a human review process, but this review is only used for output quality control and does not affect the model. In contrast, this invention provides a closed loop: analysis -> verification -> manual error correction -> model optimization -> re-analysis, breaking the limitations of the linear, unidirectional workflow of general media asset systems and enabling the system to possess human-like learning capabilities. This architectural change is not a natural development as is customary in the industry, but rather a new approach that comprehensively utilizes machine learning's active learning and multimodal fusion, significantly different from any existing single-module improvement, and therefore possesses non-obviousness.

[0076] 4. New Applications of Private Cloud Combined with Human-Machine Collaboration: In engineering implementation, this invention integrates complex algorithmic processes into a private cloud environment and fully utilizes large language models for result processing. This combined approach is not a simple patchwork of existing technologies: while private clouds are primarily used for distributed processing and data security, this invention uniquely introduces a human-machine collaboration loop in the cloud, making manual correction—a traditionally offline process—an online component of the system, thereby improving model performance more efficiently. In patent and academic literature, we have not found any similar closed-loop learning system solutions for multimodal analysis. Even some technologies from Adobe that combine different modal embeddings for content analysis (such as concatenating image and text embeddings for predicting content metrics) fall under the category of static analysis and do not involve contradictory sample screening or adaptive model updates. Therefore, the specific technical means and overall framework proposed in this invention are unprecedented in existing technologies, solving a long-standing but unresolved technical pain point, demonstrating significant innovation.

[0077] Based on the same technical concept, embodiments of the present invention also provide an electronic device that can implement the closed-loop self-learning method for multimodal media content analysis based on a contradictory sample pool provided in the above embodiments of the present invention. In one embodiment, the electronic device can be a server, a terminal device, or other electronic devices. Figure 3 As shown, the electronic device may include: At least one processor and a memory connected to the at least one processor. In this embodiment of the invention, the specific connection medium between the processor and the memory is not limited. Figure 3 The example used is the connection between the processor and memory via a bus. The bus... Figure 3 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. Buses can be divided into address buses, data buses, control buses, etc., but for ease of representation, [the specific bus type is not shown here]. Figure 3 The processor is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, a processor can also be called a controller; there are no restrictions on the name.

[0078] In this embodiment of the invention, the memory stores instructions executable by at least one processor. By executing the instructions stored in the memory, the at least one processor can execute the aforementioned closed-loop self-learning method for multimodal media content analysis based on a contradictory sample pool. The processor can implement... Figure 3 The functions of each module in the device shown.

[0079] The processor is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory and calling data stored in memory, it can monitor the device's various functions and process data, thereby enabling overall monitoring of the device.

[0080] In an alternative design, the processor may include one or more processing units. The processor may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor. In some embodiments, the processor and memory may be implemented on the same chip; in some embodiments, they may also be implemented separately on separate chips.

[0081] The processor can be a general-purpose processor, such as a CPU, digital signal processor, application-specific integrated circuit, field-programmable array, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the multimodal media content analysis closed-loop self-learning method based on contradictory sample pools disclosed in the embodiments of this invention can be directly manifested as execution by a hardware processor, or as execution by a combination of hardware and software modules within the processor.

[0082] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. In embodiments of the present invention, memory can also be a circuit or any other device capable of implementing storage functions, used to store program instructions and / or data.

[0083] By designing and programming the processor, the code corresponding to the multimodal media content analysis closed-loop self-learning method based on contradictory sample pools described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the steps of the method described in the foregoing embodiments during runtime. How to design and program the processor is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0084] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform a closed-loop self-learning method for multimodal media content analysis based on a contradictory sample pool, as described above.

[0085] In some alternative embodiments, the present invention also provides a method for closed-loop self-learning of multimodal media content analysis based on contradictory sample pools, which can also be implemented as a program product including program code. When the program product is run on a device, the program code is used to cause the control device to perform the steps in the method for closed-loop self-learning of multimodal media content analysis based on contradictory sample pools according to various exemplary embodiments of the present invention as described in this specification.

[0086] It should be noted that although several units or sub-units of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units. Furthermore, although the operation of the method of the invention is described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0087] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a server, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0089] Program code for performing the operations of this invention can be written using any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0090] In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0093] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A closed-loop self-learning system for multimodal media content analysis based on a contradictory sample pool, characterized in that, include: The multimodal analysis module is used to acquire media content and segment it. And use a multimodal analysis model to extract the multimodal features of each segment; The cross-modal consistency discrimination module is used to judge the intermodal consistency of the multimodal features of each segment. If any feature of any modality is inconsistent with other features, the segment is marked as a contradictory sample and stored in the contradictory sample pool; otherwise, the label of the segment is determined based on the multimodal features of the segment and output to the label output module. The contradictory sample pool is used to store contradictory samples and their identification information, multimodal features, and preliminary labels for automatic identification. The manual correction module provides an interface for manual review and annotation, and receives correction labels for contradictory samples; The model local update module is used to locally update the multimodal analysis model based on contradictory samples and correction labels; The tag output module is used to summarize the tags output by the cross-modal consistency discrimination module or the correction tags by the manual correction module for each segment, and generate a unified tag output for the media content.

2. The multimodal media content analysis closed-loop self-learning system based on a contradictory sample pool according to claim 1, characterized in that, The multimodal analysis model includes: The video frame image embedding extraction unit is used to extract the video frame image embedding vector of a segment. The speech-to-text embedding extraction unit is used to extract the ASR audio-to-text embedding vector of a segment; An optical character recognition text embedding extraction unit is used to extract the OCR recognition text embedding vector of a segment; The face feature embedding extraction unit is used to extract the face recognition embedding vector of the segment.

3. The multimodal media content analysis closed-loop self-learning system based on a contradictory sample pool according to claim 2, characterized in that, The intermodal consistency judgment process in the cross-modal consistency judgment module includes: Based on the extracted multimodal features, intermodal relationship modeling and semantic reasoning are performed in a unified embedding representation space, and intermodal differences are inferred; the semantic reasoning includes similarity matching, semantic reference recognition, and context-assisted judgment. If the result of semantic reasoning between any two modalities indicates that there are differences between the two modalities, it is considered that the segment has features of any modality that are inconsistent with other features.

4. The multimodal media content analysis closed-loop self-learning system based on a contradictory sample pool according to claim 1, characterized in that, The specific working process of the model local update module includes: Based on the manually corrected contradictory samples, the multimodal analysis model is updated locally or the embedded representation space is directly optimized and adjusted.

5. The multimodal media content analysis closed-loop self-learning system based on a contradictory sample pool according to claim 4, characterized in that, The local parameter update includes: performing several iterative updates on the multimodal analysis model using only contradictory samples and correction labels.

6. The multimodal media content analysis closed-loop self-learning system based on a contradictory sample pool according to claim 1, characterized in that, The tag output module includes: The tag aggregation unit is used to aggregate the tags output by the cross-modal consistency discrimination module or the correction tags of the manual correction module for each segment; The large language model standardization unit is used to perform semantic normalization and format standardization on the tags summarized by the tag aggregation unit, and output standardized and unified tag results.

7. A closed-loop self-learning method for multimodal media content analysis based on a contradictory sample pool, characterized in that, include: S1. Obtain the media content to be analyzed and segment the media content. S2. For any given segment, use a multimodal analysis model to extract the multimodal features of the segment; S3. Perform intermodal consistency judgment on the multimodal features of the segment. If any feature of any modality is inconsistent with other features, the segment is marked as a contradictory sample, and the contradictory sample, its identification information, multimodal features and labels are stored in the contradictory sample pool; otherwise, generate the label of the segment based on the multimodal features of the segment. S4. Manually correct the contradictory samples in the contradictory sample pool and generate correction labels; S5. The multimodal analysis model is locally updated using the correction labels; S6. Repeat S2~S5 until all segments are labeled by the cross-modal consistency discrimination module or by the correction label of the manual correction module, and generate a unified label output for the media content.

8. The closed-loop self-learning method for multimodal media content analysis based on a contradictory sample pool as described in claim 7, characterized in that, In S2, the extracted multimodal features include video frame image embedding vectors, audio ASR transcription text embedding vectors, image OCR text embedding vectors, and human face embedding vectors.

9. The closed-loop self-learning method for multimodal media content analysis based on a contradictory sample pool as described in claim 8, characterized in that, In S3, the intermodal consistency judgment specifically includes: Based on the extracted multimodal features, intermodal relationship modeling and semantic reasoning are performed in a unified embedding representation space, and intermodal differences are inferred; the semantic reasoning includes similarity matching, semantic reference recognition, and context-assisted judgment. If the result of semantic reasoning between any two modalities indicates that there are differences between the two modalities, it is considered that the segment has features of any modality that are inconsistent with other features.

10. The closed-loop self-learning method for multimodal media content analysis based on a contradictory sample pool as described in claim 7, characterized in that, In step S5, the multimodal analysis model is locally updated using the correction label, specifically including: Based on the manually corrected contradictory samples and correction labels, the multimodal analysis model is updated locally, or the embedded representation space is directly optimized and adjusted.

Images