A construction method and system for multimodal archive review

By acquiring, unifying vectorization, and analyzing multimodal data, the challenges of content extraction and review of multimodal archives have been solved, achieving efficient data processing and review.

CN118760769BActive Publication Date: 2024-11-15SHANDONG BANNER INFORMATION CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411254622.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-09
Publication Date
2024-11-15
Estimated Expiration
2044-09-09

AI Technical Summary

Technical Problem

Extracting and reviewing the content of multimodal archives is challenging, as it requires considering the differences and relationships between different modal data, and existing technologies struggle to achieve efficient and unified processing.

Method used

The method employs multimodal data acquisition, unified vectorization processing, data analysis, and review, including separate vectorization processing of text, images, audio, and video, combined with speech recognition and keyframe extraction, utilizing distributed crawler technology to improve data collection efficiency, and conducting review through a large language model.

Benefits of technology

It achieves unified processing of multimodal data, improves processing efficiency, effectively extracts key content from different types of data, and ensures the accuracy and efficiency of data analysis through auditing rules.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118760769B_ABST
    Figure CN118760769B_ABST
Patent Text Reader

Abstract

A construction method and system for multimodal archive review, comprising the following steps: multimodal data acquisition, the multimodal data including image data; unified vectorization processing of the multimodal data; data analysis processing of the multimodal data processed by the unified vectorization; and review of the multimodal data processed by the data analysis. The present application performs unified vectorization processing on the multimodal data, and then performs data analysis processing on the processed data, so that the multimodal data can be uniformly handled to ensure processing efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to a method and system for constructing a multimodal archive review. Background Art

[0002] Multimodal data refers to a dataset that contains different types of data, such as text, images, videos, and audio. This type of dataset is characterized by containing multiple forms of heterogeneous data, each of which provides information in a unique way. For example, a video contains not only visual information, but may also include sound. The processing and analysis of multimodal data usually involves using these different modalities of data to work together to extract richer and more comprehensive information. For example, in human-computer interaction, speech recognition and facial expression analysis can be performed simultaneously to understand the user's intentions and emotional state. For multimodal archives, that is, a collection of multimodal data, due to the complexity of the data itself, it is difficult to extract and review its content. It is necessary to consider both the differences between different modal data and the connections between them. Therefore, it is necessary to establish a processing method different from that of single modal archives. Summary of the invention

[0003] In order to solve the above problems, the present application discloses a method for constructing a multimodal archive review, comprising the following steps:

[0004] Acquiring multimodal data, wherein the multimodal data includes image data;

[0005] Perform unified vectorization processing on multimodal data;

[0006] Perform data analysis and processing on the multimodal data that has been uniformly vectorized;

[0007] The multimodal data after data analysis and processing is reviewed. This application uniformly vectorizes the multimodal data, and then performs data analysis and processing on the processed data, so that the multimodal data can be uniformly processed to ensure processing efficiency.

[0008] Preferably, the multimodal data further includes text data, audio data and video data, and the unified vectorization processing is performed according to the following steps:

[0009] For text data, perform text vectorization to obtain text vectors;

[0010] For image data, image vectorization is performed to obtain image vectors;

[0011] For audio data, we obtain text through speech recognition, and then perform text vectorization to obtain audio vectors.

[0012] For video data, firstly, information extraction is performed to obtain video key frames and video audio; for video key frames, video key frame image vectorization is performed to obtain video image vectors, for video audio, speech recognition is performed to obtain video text, and then video text vectorization is performed to obtain video audio vectors, and the video image vectors and video audio vectors are integrated to obtain video vectors. The present application performs vectorization processing on text data, image data, audio data and video data respectively, and this method can find out the key contents of different types of data, which is also conducive to later data analysis and review.

[0013] Preferably, the video key frames are selected in the following manner:

[0014] For video data containing video and audio, proceed as follows:

[0015] Perform speech recognition on the video and audio to obtain video text, and then perform video text vectorization to obtain a video and audio vector. If the video and audio vector has only one dimension, locate the playback position of the audio that generates the vector according to the obtained video and audio vector, and then obtain the playback position of the video, and then take the middle position of the playback position and three video frames within ±100ms of the middle position as video key frames;

[0016] If the video and audio vector has multiple dimensions, the playback position of the audio that generates the vector dimension is located according to the obtained video and audio vector, and then the playback position of the video is obtained. Then, the middle position of the playback position and three video frames within ±100ms of the middle position are taken as video key frames. If the time distance between video key frames does not exceed 80ms, the video key frames at the rear are deleted, and the combing is repeated until the distance between adjacent video key frames exceeds 80ms.

[0017] For video data without video or audio, proceed as follows:

[0018] The video frame is obtained by extracting one frame at 20ms;

[0019] The video frames are analyzed. If the content change of 5 consecutive video frames does not exceed 30%, a frame in the middle of the time is taken as the video key frame. When there is voice in the video, the application obtains the key frame with the assistance of voice recognition. For the part without video audio, the video frame content change rate is analyzed to effectively extract the display content that exists in the video for a long time, thereby completing the extraction of the video key frame.

[0020] Preferably, the multimodal data acquisition is performed in the following manner:

[0021] To obtain text data, use Python's `requests` library or `Scrapy` to crawl the web, and combine `BeautifulSoup` or `lxml` to parse HTML; for DOC files, use `python-docx` to process, and PDF files use `PyPDF2` or `pdfminer` to process;

[0022] Image data is collected using Python's `requests` library or `Scrapy` for web crawling, and `Pillow` is used for image processing, including format conversion and basic processing;

[0023] Audio data collection uses the `requests` library for web crawling, and uses `pydub` or `librosa` for audio processing to ensure support for MP3 and WAV formats;

[0024] Video data collection uses the `requests` library for web crawling, and `moviepy` is used for video processing and format conversion.

[0025] Preferably, web crawling uses distributed crawler technology; use `Scrapy` combined with the `Scrapy-Redis` framework to implement distributed crawlers, implement task queue management through Redis, ensure deduplication and scheduling of crawler tasks, and use multiple crawler nodes to work in parallel to improve the efficiency and reliability of data collection;

[0026] Preferably, data preprocessing is performed after data processing, and the data preprocessing includes data deduplication and data noise reduction;

[0027] Data deduplication is performed using hash sets in Redis or databases. A unique hash value is generated for each piece of data. If the hash value already exists, the data is skipped.

[0028] For data denoising, the text data is cleaned using `nltk` or `spacy` to remove HTML tags and special character noise; the audio data is denoised using `pydub` or `librosa`; and the image data and video data are subjected to basic noise removal processing.

[0029] Preferably, the data analysis process includes content classification, information extraction, document summarization and relationship reasoning, and is performed in the following manner:

[0030] The content classification includes subject classification and importance rating:

[0031] The topic classification adopts predefined topic categories and few-shot learning method to classify archive content and category descriptions according to content similarity.

[0032] The importance rating is based on predefined rating standards and is evaluated and graded according to the criticality and impact scope of the content.

[0033] The information extraction utilizes a named entity recognition method to extract key information from the archives, and according to a prompt template, explicitly specifies the entity type to be extracted, and identifies and extracts the required information.

[0034] The document summary utilizes text generation capability, designs a specific prompt template, specifies the length and focus of the summary, and generates a document summary that meets the requirements.

[0035] The relational reasoning is used to analyze the association between different archives, and guides the analysis of the time sequence, subject relevance and character association between archives by combining the content and metadata of multiple related archives.

[0036] Preferably, the review is performed in the following manner:

[0037] Obtain multimodal data after data analysis and processing;

[0038] Load and manage audit rules;

[0039] Encapsulate and call multimodal large language models;

[0040] Conduct audits;

[0041] The audit results are stored.

[0042] Preferably, the multimodal data after data analysis and processing is obtained in the following manner:

[0043] Use the SQLAlchemy ORM framework to interact with PostgreSQL to achieve efficient file reading and batch acquisition of files to be reviewed, supporting paging and concurrency control;

[0044] Loading and managing audit rules is done as follows:

[0045] Use Redis to cache hotspot audit rules to increase the speed of rule loading and implement dynamic rule loading and update mechanisms, and support hot updates;

[0046] The encapsulation call for the multimodal large language model is performed as follows:

[0047] Use gRPC protocol to communicate with multimodal data and multimodal large models to implement asynchronous calling mechanism and avoid model calling blocking the main process;

[0048] The audit is performed as follows:

[0049] Use the rule engine to implement complex rule combinations and conditional judgments, and use Python's asyncio library to implement concurrent audits;

[0050] The audit results are stored as follows: Use the Jinja2 template engine to generate standardized audit reports and use batch processing technology to optimize database write performance.

[0051] On the other hand, the present application also discloses a system for constructing a multimodal archive review, including the following modules:

[0052] Data acquisition module, used for acquiring multimodal data;

[0053] Vectorization processing module, used for unified vectorization processing of multimodal data;

[0054] A data analysis module, used for performing data analysis on the multimodal data that has been uniformly vectorized;

[0055] The review module is used to review the multimodal data after data analysis and processing.

[0056] This application can bring the following beneficial effects:

[0057] 1. This application uniformly vectorizes multimodal data and then performs data analysis on the processed data, so that multimodal data can be uniformly processed to ensure processing efficiency.

[0058] 2. This application performs vectorization processing on text data, image data, audio data and video data respectively. This method can identify the key contents of different types of data, which is also conducive to subsequent data analysis and review.

[0059] 3. When there is voice in the video, this application obtains key frames with the assistance of voice recognition. For the part where there is no video or audio, it effectively extracts the display content that exists in the video for a long time by analyzing the change rate of the video frame content, thereby completing the extraction of video key frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0061] Figure 1 It is a schematic diagram of Example 1;

[0062] Figure 2 This is a schematic diagram of Example 2. DETAILED DESCRIPTION

[0063] In order to clearly illustrate the technical features of this solution, this application is elaborated in detail below through specific implementation methods.

[0064] In the first embodiment, if Figure 1 As shown, a method for constructing a multimodal archive review includes the following steps:

[0065] S101 multimodal data acquisition, the multimodal data including image data, text data, audio data and video data;

[0066] The multimodal data acquisition is performed as follows:

[0067] To obtain text data, use Python's `requests` library or `Scrapy` to crawl the web, and combine `BeautifulSoup` or `lxml` to parse HTML; for DOC files, use `python-docx` to process, and PDF files use `PyPDF2` or `pdfminer` to process;

[0068] Image data is collected using Python's `requests` library or `Scrapy` for web crawling, and `Pillow` is used for image processing, including format conversion and basic processing;

[0069] Audio data collection uses the `requests` library for web crawling, and uses `pydub` or `librosa` for audio processing to ensure support for MP3 and WAV formats;

[0070] Video data collection uses the `requests` library for web crawling, and `moviepy` is used for video processing and format conversion.

[0071] Web crawling uses distributed crawler technology; use `Scrapy` combined with the `Scrapy-Redis` framework to implement distributed crawlers, implement task queue management through Redis, ensure deduplication and scheduling of crawler tasks, and use multiple crawler nodes to work in parallel to improve the efficiency and reliability of data collection.

[0072] After the data processing, data preprocessing is also performed, and the data preprocessing includes data deduplication and data noise reduction;

[0073] Data deduplication is performed using hash sets in Redis or databases. A unique hash value is generated for each piece of data. If the hash value already exists, the data is skipped.

[0074] For data denoising, the text data is cleaned using `nltk` or `spacy` to remove HTML tags and special character noise; the audio data is denoised using `pydub` or `librosa`; and the image data and video data are subjected to basic noise removal processing.

[0075] S102 performs unified vectorization processing on the multimodal data;

[0076] Unified vectorization processing is performed in the following steps:

[0077] For text data, perform text vectorization to obtain text vectors;

[0078] For image data, image vectorization is performed to obtain image vectors;

[0079] For audio data, we obtain text through speech recognition, and then perform text vectorization to obtain audio vectors.

[0080] For video data, information extraction is first performed to obtain video key frames and video audio. For the video key frames, video key frame image vectorization is performed to obtain video image vectors. For the video audio, speech recognition is performed to obtain video text. Then, the video text is vectorized to obtain video audio vector. The video image vector and video audio vector are integrated to obtain the video vector.

[0081] The video key frames are selected in the following manner:

[0082] For video data containing video and audio, proceed as follows:

[0083] Perform speech recognition on the video and audio to obtain video text, and then perform video text vectorization to obtain a video and audio vector. If the video and audio vector has only one dimension, locate the playback position of the audio that generates the vector according to the obtained video and audio vector, and then obtain the playback position of the video, and then take the middle position of the playback position and three video frames within ±100ms of the middle position as video key frames;

[0084] If the video and audio vector has multiple dimensions, the playback position of the audio that generates the vector dimension is located according to the obtained video and audio vector, and then the playback position of the video is obtained. Then, the middle position of the playback position and three video frames within ±100ms of the middle position are taken as video key frames. If the time distance between video key frames does not exceed 80ms, the video key frames at the rear are deleted, and the combing is repeated until the distance between adjacent video key frames exceeds 80ms.

[0085] For video data without video or audio, proceed as follows:

[0086] The video frame is obtained by extracting one frame at 20ms;

[0087] For video frames, if the content of 5 consecutive video frames does not change by more than 30%, a frame in the middle in time is taken as the video key frame.

[0088] S103 performs data analysis on the multimodal data that has been uniformly vectorized;

[0089] The data analysis process includes content classification, information extraction, document summarization, and relationship reasoning, and is performed as follows:

[0090] The content classification includes subject classification and importance rating:

[0091] The topic classification adopts predefined topic categories and few-shot learning method to classify archive content and category descriptions according to content similarity.

[0092] The importance rating is based on predefined rating standards and is evaluated and graded according to the criticality and impact scope of the content.

[0093] The information extraction utilizes a named entity recognition method to extract key information from the archives, and according to a prompt template, explicitly specifies the entity type to be extracted, and identifies and extracts the required information.

[0094] The document summary utilizes text generation capability, designs a specific prompt template, specifies the length and focus of the summary, and generates a document summary that meets the requirements.

[0095] The relational reasoning is used to analyze the association between different archives, and guides the analysis of the time sequence, subject relevance and character association between archives by combining the content and metadata of multiple related archives.

[0096] S104 reviews the multimodal data after data analysis and processing.

[0097] The review is conducted as follows:

[0098] S1041 acquiring multimodal data after data analysis and processing;

[0099] Use the SQLAlchemy ORM framework to interact with PostgreSQL to achieve efficient file reading and batch acquisition of files to be reviewed, supporting paging and concurrency control;

[0100] S1042 Load and manage audit rules;

[0101] Use Redis to cache hotspot audit rules to increase the speed of rule loading and implement dynamic rule loading and update mechanisms, and support hot updates;

[0102] S1043 encapsulates and calls a multimodal large language model;

[0103] Use gRPC protocol to communicate with multimodal data and multimodal large models to implement asynchronous calling mechanism and avoid model calling blocking the main process;

[0104] S1044 Execution review;

[0105] Use the rule engine to implement complex rule combinations and conditional judgments, and use Python's asyncio library to implement concurrent audits;

[0106] S1045 stores the audit results:

[0107] Use Jinja2 template engine to generate standardized audit reports and adopt batch processing technology to optimize database write performance.

[0108] In the second embodiment, if Figure 2 As shown, a construction system for multimodal archive review includes the following modules:

[0109] A data acquisition module 201, used for acquiring multimodal data;

[0110] A vectorization processing module 202, used for performing unified vectorization processing on multimodal data;

[0111] The data analysis module 203 is used to perform data analysis on the multimodal data that has been uniformly vectorized;

[0112] The review module 204 is used to review the multimodal data after data analysis and processing.

[0113] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for constructing a multimodal archive review, characterized in that: The steps include: Acquiring multimodal data, wherein the multimodal data includes image data; Perform unified vectorization processing on multimodal data; Perform data analysis and processing on the multimodal data that has been uniformly vectorized; Review the multimodal data after data analysis and processing; The multimodal data also includes text data, audio data and video data, and the unified vectorization processing is performed according to the following steps: For text data, perform text vectorization to obtain text vectors; For image data, image vectorization is performed to obtain image vectors; For audio data, we obtain text through speech recognition, and then perform text vectorization to obtain audio vectors. For video data, firstly, information extraction is performed to obtain video key frames and video audio; for video key frames, video key frame image vectorization is performed to obtain video image vectors, for video audio, speech recognition is performed to obtain video text, and then video text vectorization is performed to obtain video audio vectors, and the video image vector and video audio vector are integrated to obtain the video vector; The data analysis process includes content classification, information extraction, document summarization, and relationship reasoning, and is performed as follows: The content classification includes subject classification and importance grading; The topic classification uses predefined topic categories and few-shot learning methods to classify archive content and category descriptions according to content similarity; The importance classification is based on predefined rating standards and is evaluated and classified according to the criticality and impact scope of the content; The information extraction uses a named entity recognition method to extract key information from the archives, and according to the prompt template, clearly specifies the entity type to be extracted, identifies and extracts the required information; The document summary utilizes text generation capability, designs a specific prompt template, specifies the length and focus of the summary, and generates a document summary that meets the requirements; The relational reasoning is used to analyze the association between different archives, and guides the analysis of the time sequence, subject relevance and character association between archives by combining the content and metadata of multiple related archives.

2. A method for constructing a multimodal archive review according to claim 1, characterized in that: The video key frames are selected in the following manner: For video data containing video and audio, proceed as follows: Perform speech recognition on the video and audio to obtain video text, and then perform video text vectorization to obtain a video and audio vector. If the video and audio vector has only one dimension, locate the playback position of the audio that generates the vector according to the obtained video and audio vector, and then obtain the playback position of the video, and then take the middle position of the playback position and three video frames within ±100ms of the middle position as video key frames; If the video and audio vector has multiple dimensions, the playback position of the audio that generates the vector dimension is located according to the obtained video and audio vector, and then the playback position of the video is obtained. Then, the middle position of the playback position and three video frames within ±100ms of the middle position are taken as video key frames. If the time distance between video key frames does not exceed 80ms, the video key frames at the rear are deleted, and the combing is repeated until the distance between adjacent video key frames exceeds 80ms. For video data without video or audio, proceed as follows: The video frame is obtained by extracting one frame at 20ms; For video frames, if the content of 5 consecutive video frames does not change by more than 30%, a frame in the middle in time is taken as the video key frame.

3. A method for constructing a multimodal archive review according to claim 1, characterized in that: The multimodal data acquisition is performed as follows: To obtain text data, use Python's `requests` library or `Scrapy` to crawl the web, and combine `BeautifulSoup` or `lxml` to parse HTML; for DOC files, use `python-docx` to process, and PDF files use `PyPDF2` or `pdfminer` to process; Image data is collected using Python's `requests` library or `Scrapy` for web crawling, and `Pillow` is used for image processing, including format conversion and basic processing; Audio data collection uses the `requests` library for web crawling, and uses `pydub` or `librosa` for audio processing, supporting MP3 and WAV formats; Video data collection uses the `requests` library for web crawling, and `moviepy` is used for video processing and format conversion.

4. A method for constructing a multimodal archive review according to claim 3, characterized in that: Web crawling uses distributed crawler technology; use `Scrapy` combined with the `Scrapy-Redis` framework to implement distributed crawlers, implement task queue management through Redis, ensure deduplication and scheduling of crawler tasks, and use multiple crawler nodes to work in parallel to improve the efficiency and reliability of data collection.

5. A method for constructing a multimodal archive review according to claim 3, characterized in that: After the data processing, data preprocessing is also performed, and the data preprocessing includes data deduplication and data noise reduction; Data deduplication is performed using hash sets in Redis or databases. A unique hash value is generated for each piece of data. If the hash value already exists, the data is skipped. For data denoising, the text data is cleaned using `nltk` or `spacy` to remove HTML tags and special character noise; the audio data is denoised using `pydub` or `librosa`; and the image data and video data are subjected to basic noise removal processing.

6. A method for constructing a multimodal archive review according to claim 1, characterized in that: The review is conducted as follows: Obtain multimodal data after data analysis and processing; Load and manage audit rules; Encapsulate and call multimodal large language models; Conduct audits; The audit results are stored.

7. A method for constructing a multimodal archive review according to claim 6, characterized in that: The multimodal data after data analysis and processing is obtained as follows: Use the SQLAlchemy ORM framework to interact with PostgreSQL to achieve efficient file reading and batch acquisition of files to be reviewed, supporting paging and concurrency control; Loading and managing audit rules is done as follows: Use Redis to cache hotspot audit rules to increase the speed of rule loading and implement dynamic rule loading and update mechanisms, and support hot updates; The encapsulation call for the multimodal large language model is performed as follows: Use gRPC protocol to communicate with multimodal data and multimodal large models to implement asynchronous calling mechanism and avoid model calling blocking the main process; The audit is performed as follows: Use the rule engine to implement complex rule combinations and conditional judgments, and use Python's asyncio library to implement concurrent audits; The audit results are stored as follows: Use the Jinja2 template engine to generate standardized audit reports and use batch processing technology to optimize database write performance.

8. A construction system for implementing the construction method of multimodal archive review according to any one of claims 1 to 7, characterized in that: Includes the following modules: Data acquisition module, used for acquiring multimodal data; Vectorization processing module, used for unified vectorization processing of multimodal data; A data analysis module, used for performing data analysis on the multimodal data that has been uniformly vectorized; The review module is used to review the multimodal data after data analysis and processing.