A multi-modal information automatic collection method and system fusing AI content recognition
By constructing an AI content recognition model and a text information association graph, the problem of low storage efficiency of multimodal information in traditional systems is solved, and accurate mining and intelligent compressed storage of multimodal information are achieved, improving storage efficiency and recognition capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING ZHIYI SHUPU DATA SERVICE CO LTD
- Filing Date
- 2026-02-13
- Publication Date
- 2026-06-05
AI Technical Summary
Traditional data acquisition systems struggle to identify and store multimodal information, resulting in low storage efficiency and difficulty in supporting in-depth text information retrieval.
By constructing an AI content recognition model, multi-dimensional text features of multimodal information are extracted, and a text information association graph is constructed and compressed for storage. This includes converting audio and video information into text information, constructing an initial model for unsupervised pre-training, iterative optimization, screening similar text features and node connections, and compression for storage.
It enables accurate mining and intelligent compressed storage of multimodal information, improving the efficiency of association recognition and storage of multimodal information.
Smart Images
Figure CN122153070A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a method and system for automatic acquisition of multimodal information that integrates AI content recognition. Background Technology
[0002] Industry news and announcements typically exist in various formats, including text, images, PDF documents, and videos. Traditional data collection systems are mostly limited to text data, lacking the ability to identify and store multimodal content, making it difficult to support in-depth text information retrieval, and resulting in low storage efficiency.
[0003] Therefore, how to achieve the association identification and storage processing of multimodal information, accurately mine the association relationship of multimodal information and perform intelligent compression storage are technical problems that urgently need to be solved by those skilled in the art. Summary of the Invention
[0004] This invention provides a method and system for automatic collection of multimodal information that integrates AI content recognition, in order to realize the association recognition and storage processing of multimodal information, accurately mine the association relationship of multimodal information and perform intelligent compression storage.
[0005] On one hand, the present invention provides a method for automatic collection of multimodal information integrating AI content recognition, which includes: Acquire multimodal information to be collected, including text information and audio / video information; convert audio / video information into text information; construct an AI content recognition model; extract multi-dimensional text features of text information based on the AI content recognition model; construct a text information association graph based on the multi-dimensional text features; and compress and store the text information association graph.
[0006] Furthermore, the audio and video information is converted into text information, including: extracting keyframes from the audio and video information to obtain audio keyframes and video keyframes respectively; performing speech recognition on the audio keyframes and image understanding on the video keyframes to convert the audio and video information into text information.
[0007] Furthermore, an AI content recognition model is constructed, and multi-dimensional text features of text information are extracted based on the AI content recognition model, including: acquiring training data, wherein the training data includes training text information and corresponding multi-dimensional text features; constructing an AI content recognition model based on the training data; inputting text information into the AI content recognition model; and outputting multi-dimensional text features corresponding to the text information.
[0008] Furthermore, an AI content recognition model is constructed based on the training data, including: building an initial AI content recognition model; performing unsupervised pre-training on large-scale general data to establish basic feature extraction capabilities; using the training data to perform targeted training on the pre-trained AI content recognition model, continuously iterating and optimizing the loss function to adjust the model parameters towards task requirements, and gradually improving the prediction accuracy in specific scenarios; using an independent test set not involved in training to test the model performance, and quantitatively evaluating it through accuracy and recall metrics; if the quantitative evaluation effect does not meet expectations, improvements are made by increasing training data, adjusting hyperparameters, or optimizing the model structure until the quantitative evaluation effect of accuracy and recall metrics reaches expectations.
[0009] Furthermore, a text information association graph is constructed based on multi-dimensional text features, including: setting each piece of text information as a text node, obtaining multi-dimensional text features between any two nodes; calculating the cosine similarity of the multi-dimensional text features between any two nodes, and filtering out similar text features with a cosine similarity greater than a preset similarity threshold; determining the association type between two nodes based on the filtered similar text features, and statistically analyzing the association types between all nodes; connecting nodes based on the association types between all nodes to obtain several node connection subgraphs, and constructing a text information association graph based on the node connection subgraphs.
[0010] Furthermore, the association type between the two nodes is determined based on the selected similar text features, including: extracting keywords from the similar text features to obtain similar feature keywords; calculating the average cosine similarity of all similar feature keywords between the two nodes, and selecting similar features whose average cosine similarity of keywords is greater than a preset association threshold; establishing a similar feature set between the two nodes based on the similar features whose average cosine similarity of keywords is greater than the preset association threshold, and determining the association type between the two nodes based on the similar feature set.
[0011] Furthermore, the text information association graph is compressed and stored, including: obtaining the number of text nodes in the node connection subgraph; if the number of text nodes is less than or equal to the preset capacity, the corresponding node connection subgraph is compressed and stored; if the number of text nodes is greater than the preset capacity, the degree of each text node in the node connection subgraph is counted, text nodes with a degree greater than the preset allowable degree are selected, and the node connection subgraph composed of the selected text nodes is compressed and stored.
[0012] On the other hand, the present invention also provides a multimodal information automatic acquisition system integrating AI content recognition, comprising: The conversion module is used to acquire the multimodal information to be collected, which includes text information and audio / video information, and to convert the audio / video information into text information; the construction module is used to construct an AI content recognition model and extract multi-dimensional text features of the text information based on the AI content recognition model; the storage module is used to construct a text information association graph based on the multi-dimensional text features and to compress and store the text information association graph.
[0013] On the other hand, the present invention also provides an electronic device, the electronic device including a memory and a processor, the memory being used to store a computer program, the processor running the computer program to cause the electronic device to perform the multimodal information automatic acquisition method fused with AI content recognition as described in any one of claims 1-7.
[0014] On the other hand, the present invention also provides a readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the multimodal information automatic acquisition method fused with AI content recognition as described in any one of claims 1-7.
[0015] This invention provides an automatic multimodal information acquisition method and system integrating AI content recognition. By acquiring multimodal information to be collected, including text and audio / video information, the method converts the audio / video information into text; constructs an AI content recognition model; extracts multi-dimensional text features from the text information based on the AI content recognition model; constructs a text information association graph based on the multi-dimensional text features; and compresses and stores the text information association graph. This method can accurately mine the association relationships of multimodal information and intelligently compress and store it, thus realizing the automatic and intelligent acquisition of multimodal information. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0017] Figure 1 This is a flowchart illustrating an automatic multimodal information acquisition method integrating AI content recognition provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a multimodal information automatic acquisition system that integrates AI content recognition, provided in an embodiment of the present invention. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0019] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0020] Figure 1 This is a flowchart illustrating an automatic multimodal information acquisition method integrating AI content recognition, provided by an embodiment of the present invention.
[0021] like Figure 1 As shown in the figure, the execution subject of the multimodal information automatic collection method integrating AI content recognition provided by this embodiment of the invention can be an electronic device, and the method mainly includes the following steps: S101, acquire the multimodal information to be collected, the multimodal information includes text information and audio / video information, and convert the audio / video information into text information; In some embodiments of this application, the conversion of audio and video information into text information includes: extracting keyframes from the audio and video information to obtain audio keyframes and video keyframes respectively; performing speech recognition on the audio keyframes and image understanding on the video keyframes to convert the audio and video information into text information.
[0022] In this embodiment, silence detection and energy analysis are performed on the audio information to extract audio keyframes containing effective speech segments. Scene segmentation and keyframe extraction are performed on the video information to obtain representative video keyframes. Automatic speech recognition technology is used to convert the audio keyframes into text, and image understanding technology is used to generate text descriptions of the scene content, objects, actions, and environment from the video keyframes. Finally, the original audio and video information is converted into structured text information.
[0023] S102, Construct an AI content recognition model and extract multi-dimensional text features of text information based on the AI content recognition model; In some embodiments of this application, an AI content recognition model is constructed, and multi-dimensional text features of text information are extracted based on the AI content recognition model. This includes: acquiring training data, which includes training text information and corresponding multi-dimensional text features; constructing an AI content recognition model based on the training data; inputting text information into the AI content recognition model; and outputting multi-dimensional text features corresponding to the text information.
[0024] In some embodiments of this application, constructing an AI content recognition model based on training data includes: constructing an initial AI content recognition model; performing unsupervised pre-training on large-scale general data to establish basic feature extraction capabilities; using training data to perform targeted training on the pre-trained AI content recognition model, continuously iterating and optimizing the loss function to adjust the model parameters towards task requirements, and gradually improving the prediction accuracy in specific scenarios; using an independent test set not involved in training to test the model performance, and quantitatively evaluating it through accuracy and recall metrics; if the quantitative evaluation effect does not meet expectations, improvements are made by increasing training data, adjusting hyperparameters, or optimizing the model structure until the quantitative evaluation effect of accuracy and recall metrics reaches expectations.
[0025] In this embodiment, the training data includes rich training text information and its manually or semi-automatically annotated multi-dimensional text feature labels. The feature dimensions include topic category, named entity, sentiment polarity, topic words, and writing style vector. The AI content recognition model is trained using the training data, and the multi-dimensional text features of each text information are output based on the trained AI content recognition model.
[0026] S103, construct a text information association graph based on multi-dimensional text features, and compress and store the text information association graph.
[0027] In some embodiments of this application, constructing a text information association graph based on multi-dimensional text features includes: setting each piece of text information as a text node, obtaining multi-dimensional text features between any two nodes; calculating the cosine similarity of the multi-dimensional text features between any two nodes, and filtering out similar text features with a cosine similarity greater than a preset similarity threshold; determining the association type between two nodes based on the filtered similar text features, and statistically analyzing the association types between all nodes; connecting nodes based on the association types between all nodes to obtain several node connection subgraphs, and constructing a text information association graph based on the node connection subgraphs.
[0028] In this embodiment, similar text features of any two text nodes are extracted by setting a similarity threshold, thereby obtaining the association type of the two text nodes. The two nodes with similar text features are connected by edges, and the corresponding edges are labeled by association type. The edge connection relationships of all nodes are counted to obtain several node connection subgraphs, thereby constructing a text information association graph.
[0029] In some embodiments of this application, determining the association type of two nodes based on the selected similar text features includes: extracting keywords from the similar text features to obtain similar feature keywords; calculating the average cosine similarity of all similar feature keywords between the two nodes, and selecting similar features whose average cosine similarity of keywords is greater than a preset association threshold; establishing a similar feature set of the two nodes based on the similar features whose average cosine similarity of keywords is greater than the preset association threshold, and determining the association type of the two nodes based on the similar feature set.
[0030] In this embodiment, keywords are extracted based on the similar text feature dimension to obtain a set of similar feature keywords representing the core semantics of the text feature in that dimension. The average cosine similarity between two nodes on the keyword set is calculated. A similar feature set is established by setting a preset association threshold. The feature dimensions in the similar feature set are combined to form the similar feature set of the node pair. The association type between nodes is defined based on the content of this feature set.
[0031] In some embodiments of this application, compressing and storing the text information association graph includes: obtaining the number of text nodes in the node connection subgraph; if the number of text nodes is less than or equal to a preset capacity, then compressing and storing the corresponding node connection subgraph; if the number of text nodes is greater than the preset capacity, then counting the degree of each text node in the node connection subgraph, filtering out text nodes with a degree greater than a preset allowable degree, and compressing and storing the node connection subgraph composed of the filtered text nodes.
[0032] In this embodiment, a preset capacity value is set. If the number of text nodes in a subgraph is less than or equal to the preset capacity, the subgraph is considered to have a compact structure and directly enters the compression and storage process. If the number of text nodes in a subgraph is greater than the preset capacity, it is pruned. The degree of each text node in the subgraph is calculated, and key nodes with a degree greater than a preset allowable degree are selected. These key nodes and the edges between them are retained to form a new, scaled-down core subgraph, which is then compressed and stored.
[0033] Figure 2 This is a schematic diagram of the structure of a multimodal information automatic acquisition system that integrates AI content recognition, provided in an embodiment of the present invention.
[0034] like Figure 2As shown in the figure, an embodiment of the present invention provides a multimodal information automatic acquisition system integrating AI content recognition, comprising: The conversion module is used to acquire the multimodal information to be collected, which includes text information and audio / video information, and to convert the audio / video information into text information; the construction module is used to construct an AI content recognition model and extract multi-dimensional text features of the text information based on the AI content recognition model; the storage module is used to construct a text information association graph based on the multi-dimensional text features and to compress and store the text information association graph.
[0035] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the multimodal information automatic acquisition method integrating AI content recognition as described in this application.
[0036] This application also provides a computer-readable storage medium storing computer program instructions, which are read and executed by a processor to perform the multimodal information automatic acquisition method integrating AI content recognition in this application.
[0037] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0038] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for automatically collecting multimodal information integrating AI content recognition, characterized in that, include: Acquire the multimodal information to be collected, which includes text information and audio / video information, and convert the audio / video information into text information; Build an AI content recognition model and extract multi-dimensional text features of text information based on the AI content recognition model; A text information association graph is constructed based on multi-dimensional text features, and the text information association graph is compressed and stored.
2. The method for automatic acquisition of multimodal information integrating AI content recognition according to claim 1, characterized in that, Converting audio and video information into text information, including: Extract keyframes from audio and video information to obtain audio keyframes and video keyframes respectively; Speech recognition is performed on audio keyframes, and image understanding is performed on video keyframes to convert audio and video information into text information.
3. The method for automatic acquisition of multimodal information integrating AI content recognition according to claim 1, characterized in that, Construct an AI content recognition model, and extract multi-dimensional text features from the text information based on the AI content recognition model, including: Acquire training data, which includes training text information and corresponding multi-dimensional text features, and construct an AI content recognition model based on the training data; Input text information into the AI content recognition model and output multi-dimensional text features corresponding to the text information.
4. The method for automatic acquisition of multimodal information integrating AI content recognition according to claim 3, characterized in that, An AI content recognition model is built based on the training data, including: Build an initial AI content recognition model, perform unsupervised pre-training on large-scale general data to establish basic feature extraction capabilities; The pre-trained AI content recognition model is trained using training data. By continuously iterating and optimizing the loss function, the model parameters are adjusted to meet the requirements of the task, thereby gradually improving the prediction accuracy in specific scenarios. The model performance was tested using an independent test set that was not used in the training, and quantitatively evaluated using accuracy and recall metrics. If the quantitative evaluation results do not meet expectations, improvements can be made by increasing training data, adjusting hyperparameters, or optimizing the model structure until the quantitative evaluation results of accuracy and recall meet expectations.
5. The method for automatic acquisition of multimodal information integrating AI content recognition according to claim 1, characterized in that, Construct a text information association graph based on multi-dimensional text features, including: Each piece of text information is set as a text node, and the multi-dimensional text features of any two nodes are obtained. Calculate the cosine similarity of multi-dimensional text features between any two nodes, and filter out similar text features whose cosine similarity is greater than a preset similarity threshold; The association type between two nodes is determined based on the selected similar text features, and the association types between all nodes are counted. Connect nodes according to the association types between all nodes to obtain several node connection subgraphs, and construct a text information association graph based on the node connection subgraphs.
6. The method for automatic acquisition of multimodal information integrating AI content recognition according to claim 5, characterized in that, The association type between two nodes is determined based on the selected similar text features, including: Keyword extraction is performed on similar text features to obtain similar feature keywords; Calculate the mean cosine similarity of all similar keywords between two nodes, and filter out similar features whose mean cosine similarity is greater than a preset association threshold; A similarity feature set for two nodes is established based on the similarity features where the average cosine similarity of the keywords is greater than a preset association threshold. The association type between the two nodes is then determined based on the similarity feature set.
7. The method for automatic acquisition of multimodal information integrating AI content recognition according to claim 5, characterized in that, Compress and store the text information association graph, including: Get the number of text nodes in the node connection subgraph. If the number of text nodes is less than or equal to the preset capacity, compress and store the corresponding node connection subgraph. If the number of text nodes exceeds the preset capacity, the degree of each text node in the node connection subgraph is counted, and text nodes with a degree greater than the preset allowable degree are filtered out. The node connection subgraph composed of the filtered text nodes is then compressed and stored.
8. A multimodal information automatic acquisition system integrating AI content recognition, characterized in that, include: The conversion module is used to acquire the multimodal information to be collected, which includes text information and audio / video information, and to convert the audio / video information into text information. The building module is used to build an AI content recognition model and extract multi-dimensional text features of text information based on the AI content recognition model. The storage module is used to construct a text information association graph based on multi-dimensional text features and to compress and store the text information association graph.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the multimodal information automatic acquisition method fused with AI content recognition as described in any one of claims 1-7.
10. A readable storage medium, characterized in that, The readable storage medium stores computer program instructions, which are read and executed by a processor to perform the multimodal information automatic acquisition method fused with AI content recognition as described in any one of claims 1-7.