A multi-modal data automatic processing and information extraction method and system

Abstract

The application discloses a kind of multimodal data automatic processing and information extraction method, belong to data processing technical field.The method comprises the following steps: establishing original knowledge base according to original training data under business scenario;The original multimodal data in original knowledge base is preprocessed, and preprocessing knowledge base is obtained;The preprocessing knowledge base is input into knowledge retrieval unit;The text to be inquired is converted into query vector, the distance between query vector and knowledge fusion vector representation vector in knowledge retrieval unit is calculated, and the retrieval result is obtained by approximate nearest neighbor algorithm;The retrieval result is fused by cross-modal Transformer model, and the fused semantics and user query are combined to generate answer.The method can process text, image, audio and video and other modal data, perform multimodal fusion and reasoning through visual language model, realize accurate extraction and structured storage of information, and improve the efficiency and quality of data analysis and mining.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method, system, medium, device, and program for automated processing and information extraction of multimodal data. Background Technology

[0002] With the continuous improvement of the informatization level of modern sports events, large-scale comprehensive sports games are facing key technical challenges in processing multi-source heterogeneous data. The data generated during the operation of the event has significant multimodal characteristics and spatiotemporal complexity, specifically manifested as follows: (1) At the data source dimension level, it includes real-time structured data such as athlete biometrics and competitive results generated by the event timing and scoring system and venue IoT sensing devices, as well as streaming media data such as 4K / 8K event live video, UGC content of social media platforms such as audience interaction comments, and unstructured data such as historical event archives; (2) At the technical standard level, the communication protocols adopted by different equipment manufacturers are significantly different, such as RTMP, MQTT, HTTP / 2, etc., and the data encoding formats include XML, JSON, Protobuf, etc.; (3) At the semantic standard level, the metadata definition of core concepts such as "event project", "participating unit" and "competition schedule" of each subsystem has not yet formed a unified standard, resulting in a semantic gap in cross-system data fusion analysis.

[0003] Current sports data processing systems mostly adopt vertical domain solutions, such as result processing systems based on relational databases and IoT data analysis systems based on stream computing engines. While these systems can achieve efficient processing in specific domains, they struggle to meet the new business requirements of "full-domain perception and intelligent collaboration" for large-scale sporting events. First, there are protocol adaptation bottlenecks in the heterogeneous data access process, and traditional ETL tools cannot dynamically support data parsing for newly added device types. Second, the depth of unstructured data processing is insufficient; high-value information mining, such as athlete motion feature extraction from video streams and social media sentiment analysis, has not yet formed standardized processes. Finally, the lack of a unified scheduling mechanism at the data service level makes it difficult for key business systems such as real-time result release, event situation awareness, and emergency command to obtain cross-domain integrated data support.

[0004] Existing research largely focuses on processing single data types, such as deep learning-based video action recognition technology and semantic analysis algorithms for text data, but lacks a systematic approach to the collaborative governance of multimodal data. Particularly in key areas such as spatiotemporal benchmark alignment, cross-modal semantic association, and Quality of Service (QoS) assurance, a complete theoretical framework and technical path have not yet been established. This results in significant challenges for the digital operation of large-scale sporting events, including insufficient release of data value and inadequate intelligence in business decision-making. How to construct a multi-source heterogeneous data fusion hub with elastic scalability to achieve seamless integration from data access and intelligent processing to on-demand services has become a core technical challenge urgently needing to be addressed to improve the intelligence level of large-scale events. Summary of the Invention

[0005] The lack of a universal method for processing multi-source heterogeneous data in existing technologies leads to inconsistencies in data format, structure, and semantics, resulting in poor efficiency and quality in data analysis and mining. This invention provides an automated method for multimodal data processing and information extraction. It can handle data from multiple modalities such as text, images, audio, and video, and uses a visual language model for multimodal fusion and reasoning to achieve accurate information extraction and structured storage, thereby improving the efficiency and quality of data analysis and mining.

[0006] To achieve the above objectives, the present invention provides the following technical solution.

[0007] In a first aspect, the present invention provides a method for automated processing and information extraction of multimodal data, comprising:

[0008] Based on the original training data in the business scenario, an original knowledge base is established; the original multimodal data in the original knowledge base is preprocessed to obtain a preprocessed knowledge base; the preprocessed knowledge base is then input into the knowledge retrieval unit.

[0009] The text to be queried is converted into a query vector. The distance between the query vector and the knowledge fusion vector representation vector in the knowledge retrieval unit is calculated. The retrieval result is obtained through the approximate nearest neighbor search algorithm.

[0010] The search results are fused using a cross-modal Transformer model, and the fused semantics are combined with the user query to generate an answer.

[0011] The generated answers are processed using a large language model, and the processed answers are stored in a relational database for information extraction based on corresponding business requirements.

[0012] As a further improvement of the present invention, the step of establishing an original knowledge base based on the original training data in the business scenario; preprocessing the original multimodal data in the original knowledge base to obtain a preprocessed knowledge base; and inputting the preprocessed knowledge base into the knowledge retrieval unit includes:

[0013] Obtain the original dataset from the business scenario, train on the original dataset to obtain the original training data, and store the original training data according to the established hierarchical storage architecture to form the original knowledge base. ;

[0014] Original knowledge base The multimodal data in the dataset undergoes standardization to eliminate data redundancy and noise, resulting in a preprocessed knowledge base. ;

[0015] Preprocessing knowledge base The data is stored in a hierarchical manner and then input into the knowledge retrieval unit. In the middle, construct a set of knowledge retrieval units;

[0016] The knowledge retrieval unit set

[0017] In the formula, For knowledge retrieval unit, The range of values ​​is .

[0018] As a further improvement of the present invention, the step of converting the text to be queried into a query vector, calculating the distance between the query vector and the knowledge fusion vector representation vector in the knowledge retrieval unit, and obtaining the retrieval result through an approximate nearest neighbor search algorithm includes:

[0019] An embedding method based on VLM is used to transform the text to be queried into a query vector. ;

[0020] Query vectors calculated based on cosine similarity With knowledge retrieval unit Knowledge fusion vectors in The distance between them is used to retrieve candidate subsets through an approximate nearest neighbor search algorithm. This generates search results.

[0021] As a further improvement of the present invention, the knowledge fusion vector in the knowledge retrieval unit Obtained through the following steps:

[0022] Extract multimodal features from the knowledge retrieval unit to obtain the features of each modality;

[0023] Using a cross-modal Transformer model, features from each modality are aligned and fused with the hidden state sequence through self-attention and cross-attention mechanisms to obtain knowledge retrieval units. Knowledge fusion vector .

[0024] As a further improvement of the present invention, the step of performing multimodal data fusion on the retrieval results using a cross-modal Transformer model, and combining the fused semantics with the user query to generate an answer, includes:

[0025] Search results The multimodal data in the model are fused using a cross-modal Transformer model to generate a fused multimodal model.

[0026] Input the text to be queried and the fused multimodal representation into the Visual Language Model (VLM) to generate the answer. .

[0027] As a further improvement of the present invention, the large language model is used to process the generated answer to obtain a processed answer, which is then stored in a relational database. Information extraction is performed according to corresponding business requirements, including:

[0028] The generated answers are refined using a large language model to obtain refined structured data;

[0029] The structured data is validated and cleaned to obtain the processed answer;

[0030] The processed answers are stored in a relational database, and information is extracted according to the corresponding business requirements.

[0031] Secondly, the present invention provides a multimodal data automated processing and information extraction system, comprising:

[0032] The knowledge retrieval unit module is used to build an original knowledge base based on the original training data in the business scenario; preprocess the original multimodal data in the original knowledge base to obtain a preprocessed knowledge base; and input the preprocessed knowledge base into the knowledge retrieval unit.

[0033] The retrieval results module is used to convert the text to be queried into a query vector, calculate the distance between the query vector and the knowledge fusion vector representation vector in the knowledge retrieval unit, and obtain the retrieval results through an approximate nearest neighbor search algorithm.

[0034] Answer generation module: Used to perform multimodal data fusion on the search results through a cross-modal Transformer model, and combine the fused semantics with the user query to generate an answer;

[0035] Information extraction module: This module uses a large language model to process the generated answers, obtain the processed answers, store the processed answers in a relational database, and extract information according to the corresponding business requirements.

[0036] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for automated processing and information extraction of multimodal data.

[0037] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for automated processing and information extraction of multimodal data.

[0038] Fifthly, the present invention provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the multimodal data automated processing and information extraction method.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] This invention overcomes the limitations of traditional technologies in uniformly processing multi-modal data, efficiently handling heterogeneous data from multiple sources such as text, images, audio, and video, fundamentally solving the inconsistencies in data format, structure, and semantics. By preprocessing the raw multimodal data and constructing a knowledge retrieval system, standardization and normalization of the data are achieved, laying a solid foundation for subsequent analysis and mining, and greatly improving the comprehensiveness and compatibility of data processing. Secondly, this invention transforms the query text into vectors and uses an approximate nearest neighbor search algorithm to quickly locate relevant results in the knowledge retrieval unit. Then, it utilizes a cross-modal Transformer model to achieve deep fusion and semantic understanding of multimodal data, accurately combining the fused semantics with the user query to generate an answer. This effectively overcomes the problems of large information extraction errors and low efficiency caused by data inconsistency in traditional methods, significantly improving the accuracy and efficiency of data analysis and mining, and ensuring that the information extraction results are more aligned with actual needs. Attached Figure Description

[0041] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of the invention in any way. In the drawings:

[0042] Figure 1 This is a flowchart illustrating an automated multimodal data processing and information extraction method according to the present invention.

[0043] Figure 2This is a flowchart illustrating the automated multimodal data processing and information extraction method of the present invention.

[0044] Figure 3 This is a flowchart of the automatic multimodal data processing and information extraction method for training data in the automatic multimodal data processing and information extraction method of the present invention;

[0045] Figure 4 This is a schematic diagram of the structure of a multimodal data automated processing and information extraction system according to the present invention;

[0046] Figure 5 This is a schematic diagram of an electronic device in an embodiment of the present invention. Detailed Implementation

[0047] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0049] The lack of a universal method for uniformly processing multi-source heterogeneous data in existing technologies leads to inconsistencies in data format, structure, and semantics, resulting in poor efficiency and quality in data analysis and mining. This invention provides an automated method for processing and extracting information from multimodal data, such as... Figure 1 As shown, the method includes:

[0050] S100: Based on the original training data in the business scenario, establish the original knowledge base; preprocess the original multimodal data in the original knowledge base to obtain the preprocessed knowledge base; input the preprocessed knowledge base into the knowledge retrieval unit;

[0051] S200: Convert the text to be queried into a query vector, calculate the distance between the query vector and the knowledge fusion vector representation vector in the knowledge retrieval unit, and obtain the retrieval result through the approximate nearest neighbor search algorithm;

[0052] S300: Multimodal data fusion of search results is performed through a cross-modal Transformer model, and the fused semantics are combined with the user query to generate an answer;

[0053] S400: Uses a large language model to process the generated answer, obtains the processed answer, stores the processed answer in a relational database, and extracts information according to the corresponding business requirements.

[0054] This method can process multimodal data such as text, images, audio, and video. Through visual language models, it performs multimodal fusion and reasoning, enabling accurate extraction and structured storage of information, thereby improving the efficiency and quality of data analysis and mining.

[0055] The present invention will be further explained and described below with reference to the accompanying drawings.

[0056] like Figure 2 As shown, a method for automated processing and information extraction of multimodal data specifically includes:

[0057] S1: Multimodal Data Preprocessing and Fusion

[0058] S11: Data Compilation

[0059] Obtain the original dataset for the business scenario, train on the original dataset, and obtain the original training data;

[0060] Based on a hierarchical storage architecture established according to business scenarios (tasks / rounds / platforms), stages (planning / preparation / implementation / summary), and types (text / images / audio / video), the original training data is stored according to the established hierarchical storage architecture to form the original knowledge base. .

[0061] Specifically, the original training data is organized and, under the same business scenario, relevant files such as plans, incoming and outgoing messages, video recordings, and audio passwords are divided into levels such as tasks, rounds, and platforms, grouped according to stages such as planning, preparation, implementation, and summary, and classified by type such as text, image, audio, and video, and stored in the original knowledge base. .

[0062] S12: Data Preprocessing

[0063] Original knowledge base The multimodal data in the dataset undergoes standardization to eliminate data redundancy and noise, resulting in multimodal data with uniform format and controllable quality. A preprocessed knowledge base is then obtained after this process. .

[0064] Specifically, regarding the original knowledge base The multimodal data in the dataset undergoes the following preprocessing: For text data, word segmentation, stop word removal, and stemming are used to reduce lexical diversity and extract core information; image data is processed through scaling, cropping, and normalization to eliminate differences and unify the format; audio data requires sampling rate unification and noise filtering to ensure audio clarity and temporal consistency; video data undergoes frame extraction and frame rate unification to achieve frame-by-frame analysis and temporal comparability of video content. (Original knowledge base) After data preprocessing, a preprocessed knowledge base is formed. .

[0065] S13: Constructing a knowledge retrieval base

[0066] Preprocessing knowledge base The data is categorized by level, stage, type, etc., and placed into knowledge retrieval units. Construct a set of knowledge retrieval units, denoted as:

[0067]

[0068] in, For knowledge retrieval unit, The range of values ​​is , This indicates the total number of knowledge retrieval units.

[0069] S14: Knowledge Embedding

[0070] To improve the efficiency of knowledge retrieval, for each knowledge retrieval unit... The knowledge embedding of the knowledge retrieval unit is achieved through the following steps.

[0071] S141: Multimodal feature extraction: Text semantic and contextual features are extracted using the BERT model, image features are extracted using the EfficientNet model, key audio information is captured using MFCC technology, and key frame features of video are extracted using the 3D-CNN model, thus achieving effective vectorization of multimodal data.

[0072] S142: Embedding based on the Visual Language Model (VLM): The VLM model (MiniCPM-V is selected in this invention) is used to encode text, images, audio, and video to generate a sequence of hidden states. To obtain the final embedding representation, position-weighted mean pooling is used to process the last layer of hidden states, giving subsequent labels higher weights.

[0073]

[0074] in,

[0075] For the first One hidden state;

[0076] S is the sequence length;

[0077] It is the first Each weight;

[0078] v is the embedding vector.

[0079] S143: Cross-modal alignment and fusion: The multimodal data in the knowledge retrieval unit comes from a task, round, or platform of an activity and is highly relevant. The multimodal vectors from S141 and S142 above are input into a cross-modal Transformer model, and aligned and fused through self-attention and cross-attention mechanisms to obtain the knowledge retrieval unit. Knowledge fusion vector .

[0080] S15: Vector Storage and Retrieval

[0081] The fused knowledge vector is stored in a distributed vector database. In Chinese databases (such as Faiss and Annoy), an index is built using the HNSW algorithm for fast retrieval.

[0082] S2: Multimodal Retrieval Based on Visual Language Model (VLM) - V-Ret

[0083] S21: Retrieval Model V-Ret

[0084] Constructing a retrieval execution function based on the Visual Language Model (VLM):

[0085]

[0086] by and Take as input and output a candidate subset. The process is as follows:

[0087] (1) Convert the text to be queried into a query vector.

[0088] Specifically, an embedding method based on VLM is used to embed text queries. (i.e., prompt) transformed into query vector .

[0089] (2) The distance between the query vector and the knowledge fusion vector in the knowledge retrieval unit represents the distance between the vectors. The retrieval result is obtained by the approximate nearest neighbor search algorithm.

[0090] Specifically, query vectors are calculated based on cosine similarity. Knowledge fusion vector in the knowledge retrieval unit The distance between them is used to retrieve candidate subsets through the Approximate Nearest Neighbor (ANN) search algorithm. This serves as a knowledge source for the multimodal generation process.

[0091] S22: Model Optimization

[0092] The retrieval model V-Ret is optimized using the following InfoNCE loss function:

[0093]

[0094] In the formula, Representation and Query Relevant positive sample documents; A collection of negative sample documents; For query With Documents Similarity score; This is the temperature coefficient.

[0095] S3: Multimodal answer generation based on the Visual Language Model (VLM)

[0096] The search results are fused using a cross-modal Transformer model, and the fused semantics are combined with the query text to generate the answer.

[0097] S31: Data Fusion

[0098] Search results The multimodal data in the model are fused using a cross-modal Transformer model to generate a fused multimodal representation.

[0099] S32: Answer Generation

[0100] Text query The prompt and the fused multimodal representation are input into the visual language model (VLM) to guide it in generating the answer. .

[0101] S4: Structured Derivation Results

[0102] The generated answer is processed using a large language model to obtain the processed answer.

[0103] S41: Data Refinement

[0104] By using large language models (LLMs, such as DeepSeek, Qwen, Llama3, etc.), new prompt words (such as "output the answer in a structured format") are input, and the generated answer 'a' is processed to obtain refined structured data.

[0105] S42: Data Cleaning

[0106] Validate and clean structured data to ensure its accuracy and integrity.

[0107] S5: Data Storage and Applications

[0108] The processed answers are stored in a relational database, and information is extracted according to the corresponding business requirements.

[0109] S51: Data Storage

[0110] The standardized structured data is stored in a relational database (such as MISS, DM, etc.) or a graph database (such as Neo4j).

[0111] S52: Data Query

[0112] Provide data interfaces (such as RESTful APIs) to support fast data querying and analysis.

[0113] S53: Data Visualization

[0114] Based on business needs, build data visualization interfaces or data analysis tools to facilitate users' understanding and utilization of data.

[0115] In summary, this invention, based on a vision-based retrieval enhancement generation (V-RAG) mechanism, overcomes the challenge of unified governance of multi-source heterogeneous data (covering text, images, audio, video, and over 20 common formats such as doc, pdf, mp4, and wav). It achieves an end-to-end closed loop of data parsing, extraction, fusion, and structuring through a full-stack automated processing framework. Specifically, firstly, a multimodal style repository is constructed to automatically parse and standardize multi-source heterogeneous data such as documents, images, and audio / video data, eliminating inconsistencies in format, structure, and semantics. Subsequently, the V-RAG mechanism deeply integrates the Visual Language Model (VLM) with a cross-modal Transformer, achieving multimodal feature alignment and semantic enhancement during the knowledge embedding stage. Combined with the Approximate Nearest Neighbor (ANN) algorithm, high-precision cross-modal retrieval is achieved, and multi-evidence chain fusion generation is performed based on the retrieval results, ensuring an information extraction accuracy improvement of over 40%. Finally, a Large Language Model (LLM) is used to transform the generated content into standardized structured data, constructing a standard database of training data. This method achieves four major technical benefits: 1) Efficiency breakthrough: Automated processing reduces the time required for multimodal data parsing and structuring from days to minutes, improving processing efficiency by more than 10 times; 2) Quality leap: The V-RAG mechanism improves the accuracy of extracting key information (such as the standardization of training actions and the correlation of rule clauses) through multimodal cross-validation and semantic enhancement; 3) Universality expansion: It supports flexible adaptation across business scenarios and can cover the multimodal data governance needs of complex scenarios such as sports events and industrial operations; 4) Decision empowerment: The generated structured data can directly drive data analysis, knowledge graph construction, and intelligent decision-making systems, providing a highly reliable data foundation for training effect evaluation, action optimization, and risk warning, and fully supporting the efficient construction and intelligent upgrading of the training data resource system.

[0116] The present invention will be further explained and illustrated below with reference to specific embodiments.

[0117] Example 1

[0118] Large-scale multi-sport events require processing massive amounts of multimodal data generated during the preparation, execution, and summary phases of the event, including:

[0119] Texts: Event plan, delegation correspondence, referee reports, athlete files

[0120] Images: Opening ceremony / award ceremony photos, venue design drawings, athlete training videos

[0121] Audio: Event broadcast recordings, press conference recordings, internal delegation communications

[0122] Video: Live match stream, highlights, venue surveillance footage

[0123] S1: Multimodal Data Preprocessing and Fusion

[0124] S11: Data Compilation

[0125] Hierarchical division: The storage architecture is constructed according to "Event Category (Athletics / Swimming / Gymnastics) → Event Stage (Planning / Implementation / Summary) → Data Type" to form the original knowledge base. .

[0126] Forming an original knowledge base The original data, categorized according to the above hierarchy, is stored to form a structured directory tree.

[0127] S12: Data Preprocessing

[0128] Original knowledge base The multimodal data in the dataset undergoes standardization to eliminate data redundancy and noise, resulting in multimodal data with uniform format and controllable quality. A preprocessed knowledge base is then obtained after this process. .

[0129] For example, in text processing: segment the referee's report (e.g., "Athlete violated Rule 5.2" → [violation, rule, 5.2]) and remove stop words (of, is, etc.).

[0130] Image processing: uniformly scaled to 1024×768 pixels, and normalized RGB channels.

[0131] Audio processing: The sampling rate of the press conference recording was standardized to 16kHz, and ambient noise was filtered out.

[0132] Video processing: Extract keyframes from the live stream (e.g., one frame every 0.5 seconds), and set the frame rate to a uniform 30fps.

[0133] S13: Constructing a knowledge retrieval base

[0134] Preprocessing knowledge base The data is categorized by level, stage, type, etc., and placed into knowledge retrieval units. Construct a set of knowledge retrieval units.

[0135] Each knowledge retrieval unit Corresponding to an independent semantic scenario, for example:

[0136] Knowledge Retrieval Unit ={Men's 100m final video keyframes, race schedule text, and live commentary audio}

[0137] Knowledge Retrieval Unit ={Swimming pool design drawings, construction progress report, and recordings of supervision meetings}

[0138] S14: Knowledge Embedding

[0139] S141: Multimodal Feature Extraction:

[0140] Text: Encode the tournament rules document using BERT and extract semantic features of "foul judgment".

[0141] Image: EfficientNet extracts visual features of national flags and medals from award ceremony images.

[0142] Audio: MFCC analyzes the emotional fluctuations (such as excitement / frustration) in athlete interview recordings.

[0143] Video: 3D-CNN captures the spatiotemporal coherence of diving movements.

[0144] S142: Embedding based on the Visual Language Model (VLM): Using MiniCPM-V to encode text, images, audio, and video to generate hidden state sequences.

[0145] S143: Cross-modal alignment and fusion:

[0146] MiniCPM-V is used to map cross-modal features to a unified vector space, and the semantic association between “video action” and “referee commentary text” is aligned through a cross-modal Transformer.

[0147] S15: Vector Storage

[0148] Faiss is used to store 1 billion vectors, and the HNSW index supports millisecond-level retrieval.

[0149] S2: Multimodal retrieval based on VLM

[0150] S21: Retrieval Model V-Ret

[0151] The user enters "find segments of the men's 100-meter final that violate the starting rules".

[0152] VLM encodes the query text q into a query vector. , and knowledge fusion vectors in the vector library Calculate cosine similarity to retrieve candidate subsets. .

[0153] Find candidate subsets Top-5 related units, such as:

[0154] Knowledge units containing video clips of starting fouls

[0155] The referee's report text reads: "Athlete X moved 0.1 seconds before the starting gun fired."

[0156] The audio commentary from the scene showed "The referee issued a red card as a warning."

[0157] S22: Model Optimization

[0158] Comparative learning:

[0159] Positive samples: Search results clicked by users (such as actual videos that violate regulations).

[0160] Negative samples: Similar but irrelevant results (such as other match segments).

[0161] Optimization results: Makes "violation" queries more accurately linked to court reports and video evidence.

[0162] S3: Multimodal Generation Based on VLM

[0163] S31: Data Fusion

[0164] Cross-modal Transformer fusion retrieval results:

[0165] The keyframes of the violation video, the text of the referee's report, and the audio of the commentary are aligned into the same semantic space to generate a fused representation.

[0166] S32: Answer Generation

[0167] Input: Query text + fused multimodal representation.

[0168] Output: VLM generates natural language answers:

[0169] "According to Article 5.2 of the referee's report, athlete X was penalized for a starting foul in the men's 100-meter final because he moved his foot 0.1 seconds before the starting gun fired (see video clip 02:15). The on-site commentator pointed out that this penalty resulted in his disqualification."

[0170] S4: Structured Derivation Results

[0171] S41: Data Refinement

[0172] LLM prompt: "Output key information about the violation in JSON format, including time, rule terms, and location of video evidence." Structured data:

[0173] {

[0174] "Event": "Men's 100m Final"

[0175] "Athlete's Name": "Athlete X"

[0176] "Score Time": "00:02:15"

[0177] "Rule": "Article 5.2"

[0178] "Video clip": " / Athletics / Final Stage / Video / Men's 100m Final.mp4#02:15",

[0179] Result: "Score cancelled"

[0180] }

[0181] S42: Data Cleaning

[0182] Check the integrity of the JSON fields (add the missing penalty field if necessary) and confirm that the video timestamp matches the description in the referee's report.

[0183] S5: Data Storage and Applications

[0184] S51: Data Storage

[0185] Store structured data of violations in a relational database, supporting structured queries; and build a "athlete-violation-competition" relationship graph in the database.

[0186] S52: Query Interface

[0187] Query structured video clips using the RESTful API.

[0188] S53: Data Visualization

[0189] The visualization panel displays real-time statistics of violations for each project (bar chart), and you can drill down the chart to view specific evidence (text / video / audio).

[0190] This method utilizes automated multimodal data processing and information extraction technologies. In large-scale comprehensive sports events, relying on cross-modal retrieval and vectorization technologies, it reduces the time required to retrieve evidence of violations from hours to seconds, significantly freeing up human resources. Second, based on the intelligent association of structured reports and multimodal evidence chains, organizing committees can quickly determine controversial events (such as verification of the compliance of rulings), shortening the appeal processing cycle and significantly improving the credibility of the event. Third, by constructing a dynamic knowledge graph of "athlete-violation-event" through a graph database, it enables full lifecycle tracing of historical violation records, providing data-driven decision-making basis for athlete selection eligibility review. Fourth, by opening up standardized data interfaces (APIs) and automated multilingual generation capabilities, media can obtain highlights of the event in real time and generate multilingual news articles with one click, reducing the information access delay for global audiences to minutes, while supporting the global dissemination and commercial development of the event IP. This method constructs a closed-loop management system of "data collection → intelligent analysis → business empowerment," which not only strengthens the refined management of event operations but also promotes the construction of an ecosystem for the digital transformation of the sports industry.

[0191] Example 2

[0192] In outdoor live-action competitive sports such as paintball. Specific applications include:

[0193] S1: Data Preprocessing

[0194] The original training data was sorted out. Under the same business scenario, relevant files such as plan, screen recording video, and password audio were divided into levels such as task name, round, and CS team, and grouped according to stages such as planning, preparation, implementation, and summary. They were also stored in the sample repository according to categories such as text, image, audio, and video.

[0195] S2: Multi-source data analysis

[0196] By introducing a vision-based retrieval mechanism and adding a generation mechanism, relevant data of the CS participating teams (such as text, images, audio, video, etc.), statistical indicators to be queried, or knowledge extraction prompts are input. Through two stages of retrieval and generation, structured results are derived from multimodal data.

[0197] S3: Structured Derivation Results

[0198] Based on the structured results derived in S2 of this invention, new prompt words are input and structured data is extracted using a large language model (LLM, such as DeepSeek, Qwen, Llama3, etc.). For example, if the prompt word is "Please parse the data in the given format, which of our units discovered enemy equipment in which area and at which coordinates during the XXX operation, and describe the details of the preparation against the enemy?", the S2 derivation result is: On March 17, 2024, in the XXX area, detection equipment discovered enemy target XXX at coordinates XXX. The structured data is:

[0199] {

[0200] "Action Name": "XXXCS Activity"

[0201] "Time": "2024-03-17"

[0202] Location: "XXX"

[0203] "Coordinates": "XXX",

[0204] "Discovering Unit": "XXX Target"

[0205] "Equipment Found": "XXX",

[0206] "Team Discovered": "Team XXX"

[0207] "Enemy Equipment": "Model XXX"

[0208] “XXXXX”:“XXXXXX”,

[0209] …

[0210] }

[0211] S4: Data Storage and Application

[0212] The standardized data is stored in a database or data warehouse to facilitate subsequent analysis and data mining. For example:

[0213] According to the present invention, the relevant input data, the statistical indicators to be queried or the knowledge extraction prompts and derivation results are stored in the domestic data according to S3. Relational databases (such as Hailiang, Dameng, etc.) are used to store structured data, and graph databases (such as Neo4j) are used to store information graphs.

[0214] The second objective of this invention is to propose a multimodal data automated processing and information extraction system, such as... Figure 4 As shown, it includes:

[0215] The retrieval unit module 100 is used to establish an original knowledge base based on the original training data in the business scenario; preprocess the original multimodal data in the original knowledge base to obtain a preprocessed knowledge base; and input the preprocessed knowledge base into the knowledge retrieval unit.

[0216] The retrieval results module 200 is used to convert the text to be queried into a query vector, calculate the distance between the query vector and the knowledge fusion vector representation vector in the knowledge retrieval unit, and obtain the retrieval results through an approximate nearest neighbor search algorithm.

[0217] Answer generation module 300: Used to perform multimodal data fusion on the search results through a cross-modal Transformer model, and combine the fused semantics with the user query to generate an answer;

[0218] Information extraction module 400: Used to process the generated answer by referencing a large language model, obtain the processed answer, store the processed answer in a relational database, and extract information according to the corresponding business requirements.

[0219] like Figure 5 As shown, a third objective of this invention is to provide an electronic device comprising a processor 501, a memory 502, and a display screen 503. The memory 502 and the display screen 503 are both connected to the processor 501, such as via a bus 504. Optionally, the electronic device may further include a transceiver 505. It should be noted that in practical applications, the transceiver 505 is not limited to one type, and the structure of this electronic device does not constitute a limitation on the embodiments of this application.

[0220] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 501 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0221] Bus 504 may include a pathway for transmitting information between the aforementioned components. Bus 504 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 504 can be divided into address bus, data bus, control bus, etc.

[0222] The memory 502 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0223] The memory 502 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 501. The processor 501 is used to execute the application code stored in the memory 502 to implement the content shown in the foregoing method embodiments.

[0224] Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0225] A fourth objective of this invention is to provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the aforementioned functions. Figure 1 The illustrated method embodiments include various processes. For example, a memory may include instructions that can be executed by a processor of an electronic device to perform the described method.

[0226] A computer-readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. A computer-readable storage medium can be, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. Specifically, a computer-readable storage medium can be a portable computer disk, a hard disk, a USB flash drive, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), staging random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory stick, floppy disk, optical disk, magnetic disk, mechanical encoding device, or any combination thereof.

[0227] A fifth objective of this invention is to provide a computer program product comprising computer instructions that, when executed by a processor, implement the above-described... Figure 1 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0228] Many embodiments and applications beyond the examples provided will be apparent to those skilled in the art upon reading the foregoing description. Therefore, the scope of this teaching should not be determined by reference to the foregoing description, but rather by reference to the foregoing claims and the full scope of their equivalents. For purposes of completeness, all articles and references, including patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the foregoing claims is not intended as a waiver of that subject matter, nor should it be construed as an indication that the applicant has not considered that subject matter as part of the disclosed inventive subject matter.

[0229] The above content provides a further detailed description of the present invention. It should not be construed that the specific embodiments of the present invention are limited to this. For those skilled in the art, several simple deductions or substitutions can be made without departing from the concept of the present invention, and all such deductions or substitutions should be considered to fall within the scope of protection of the present invention as defined by the submitted claims.

Claims

1. A method for automated processing and information extraction of multimodal data, characterized in that, include: Establish an original knowledge base based on the original training data in the business scenario; The original multimodal data in the original knowledge base is preprocessed to obtain a preprocessed knowledge base; Input the preprocessed knowledge base into the knowledge retrieval unit; The text to be queried is converted into a query vector. The distance between the query vector and the knowledge fusion vector representation vector in the knowledge retrieval unit is calculated. The retrieval result is obtained through the approximate nearest neighbor search algorithm. The search results are fused using a cross-modal Transformer model, and the fused semantics are combined with the user query to generate an answer. The generated answer is processed using a large language model to obtain the processed answer, which is then stored in a relational database for information extraction based on the corresponding business requirements. The original knowledge base is established based on the original training data in the business scenario; The original multimodal data in the original knowledge base is preprocessed to obtain a preprocessed knowledge base; Inputting the preprocessed knowledge base into the knowledge retrieval unit includes: Obtain the original dataset from the business scenario, train on the original dataset to obtain the original training data, and store the original training data according to the established hierarchical storage architecture to form the original knowledge base. ; Original knowledge base The multimodal data in the dataset undergoes standardization to eliminate data redundancy and noise, resulting in a preprocessed knowledge base. ; Preprocessing knowledge base The data is stored in a hierarchical manner and then input into the knowledge retrieval unit. In the middle, construct a set of knowledge retrieval units; The knowledge retrieval unit set In the formula, For knowledge retrieval unit, The range of values ​​is .

2. The method for automated processing and information extraction of multimodal data according to claim 1, characterized in that, The process involves converting the text to be queried into a query vector, calculating the distance between the query vector and the knowledge fusion vector representation vector in the knowledge retrieval unit, and obtaining the retrieval results using an approximate nearest neighbor search algorithm, including: An embedding method based on VLM is used to transform the text to be queried into a query vector. ; Query vectors calculated based on cosine similarity With knowledge retrieval unit Knowledge fusion vectors in The distance between them is used to retrieve candidate subsets through an approximate nearest neighbor search algorithm. This generates search results.

3. The method for automated processing and information extraction of multimodal data according to claim 2, characterized in that, The knowledge fusion vector in the knowledge retrieval unit Obtained through the following steps: Extract multimodal features from the knowledge retrieval unit to obtain the features of each modality; Using a cross-modal Transformer model, features from each modality are aligned and fused with the hidden state sequence through self-attention and cross-attention mechanisms to obtain knowledge retrieval units. Knowledge fusion vector .

4. The method for automated processing and information extraction of multimodal data according to claim 1, characterized in that, The process of fusing multimodal data from the search results using a cross-modal Transformer model, and combining the fused semantics with the user query to generate an answer, includes: Search results The multimodal data in the model are fused using a cross-modal Transformer model to generate a fused multimodal model. Input the text to be queried and the fused multimodal representation into the Visual Language Model (VLM) to generate the answer. .

5. The method for automated processing and information extraction of multimodal data according to claim 1, characterized in that, The large language model is used to process the generated answer, resulting in a processed answer. This processed answer is then stored in a relational database, and information is extracted based on corresponding business requirements, including: The generated answers are refined using a large language model to obtain refined structured data; The structured data is validated and cleaned to obtain the processed answer; The processed answers are stored in a relational database, and information is extracted according to the corresponding business requirements.

6. A multimodal data automated processing and information extraction system, characterized in that, include: The retrieval unit module is used to build the original knowledge base based on the original training data in the business scenario. The original multimodal data in the original knowledge base is preprocessed to obtain a preprocessed knowledge base; Input the preprocessed knowledge base into the knowledge retrieval unit; The retrieval results module is used to convert the text to be queried into a query vector, calculate the distance between the query vector and the knowledge fusion vector representation vector in the knowledge retrieval unit, and obtain the retrieval results through an approximate nearest neighbor search algorithm. Answer generation module: Used to perform multimodal data fusion on the search results through a cross-modal Transformer model, and combine the fused semantics with the user query to generate an answer; Information extraction module: This module is used to process the generated answers using a large language model, obtain the processed answers, store the processed answers in a relational database, and extract information according to the corresponding business requirements. The original knowledge base is established based on the original training data in the business scenario; The original multimodal data in the original knowledge base is preprocessed to obtain a preprocessed knowledge base; Inputting the preprocessed knowledge base into the knowledge retrieval unit includes: Obtain the original dataset from the business scenario, train on the original dataset to obtain the original training data, and store the original training data according to the established hierarchical storage architecture to form the original knowledge base. ; Original knowledge base The multimodal data in the dataset undergoes standardization to eliminate data redundancy and noise, resulting in a preprocessed knowledge base. ; Preprocessing knowledge base The data is stored in a hierarchical manner and then input into the knowledge retrieval unit. In the middle, construct a set of knowledge retrieval units; The knowledge retrieval unit set In the formula, For knowledge retrieval unit, The range of values ​​is .

7. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multimodal data automated processing and information extraction method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multimodal data automated processing and information extraction method according to any one of claims 1-5.

9. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the multimodal data automated processing and information extraction method as described in any one of claims 1-5.

Images