Multi-modal data processing method and device based on large model, equipment and medium

Abstract

The invention discloses a multi-modal data processing method and device based on a large model, equipment and a medium. Performing feature extraction and association mining on the multi-modal data to obtain multi-modal data features; according to a retrieval sequence of the multiple text matching methods, sequentially using the multiple text matching methods to perform text retrieval on the multi-modal data features in a pre-constructed knowledge base to obtain a first associated sub-graph; based on the semantic similarity between the multi-modal data features and multi-modal data in a pre-constructed knowledge base and a similarity threshold value, determining a second associated sub-graph; performing standard evaluation on the plurality of candidate results in the first associated sub-graph and the second associated sub-graph, and sorting the plurality of candidate results according to an evaluation result to obtain a retrieval result; inputting the retrieval result and the multi-modal data features into a large model for information analysis to obtain prompt information and a target tool; and sending the prompt information and the target tool to the terminal, and receiving a data processing result of the terminal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multi-modal data processing method and device based on a large model, equipment and medium. BACKGROUND

[0002] Current multi-modal interaction systems, especially those involving voice interaction, are generally not end-to-end model implementation solutions and require the integration of multiple types of models, such as large language models, speech recognition models, speech synthesis models, and visual language multi-modal understanding models.

[0003] For a voice input, speech recognition transcription is first performed, then input into a large model for understanding and generation, and after the generation result, speech synthesis is performed, and finally feedback is given to the user. This approach often results in high interaction delays, as each step requires the completion of the previous step before it can be executed, especially in cases where the inference performance of the large model is not high.

[0004] At the same time, most current RAG (Retrieval-augmented Generation) systems still remain in single-modal retrieval and generation, and even multi-modal RAGs only perform retrieval on single-modal data, which cannot perform multi-modal retrieval, resulting in low recall efficiency of multi-modal data and insufficient relevance and coverage of the recall results.

[0005] Therefore, in the processing of multi-modal data, the recall efficiency of multi-modal interaction systems and multi-modal retrieval are current exploration directions. SUMMARY

[0006] The embodiments of the present application provide a multi-modal data processing method and device based on a large model to solve the problem that the prior art cannot perform multi-modal retrieval, resulting in low recall efficiency of multi-modal data and insufficient relevance and coverage of the recall results.

[0007] In a first aspect, the present application provides a multi-modal data processing method based on a large model, comprising:

[0008] receiving a data processing request of multi-modal data; performing feature extraction and association mining on the multi-modal data to obtain multi-modal data features; the multi-modal data features represent semantic relationships and logical relationships between the multi-modal data;

[0009] The multi-modal data features are sequentially subjected to text retrieval in a pre-constructed knowledge base according to a retrieval order of a plurality of text matching methods, to obtain a first associated subgraph; a second associated subgraph is determined based on semantic similarity between the multi-modal data features and multi-modal data in the pre-constructed knowledge base, and a similarity threshold value;

[0010] The plurality of candidate results in the first associated subgraph and the second associated subgraph are subjected to standard evaluation, and the plurality of candidate results are sorted according to the evaluation results, to obtain a retrieval result; the retrieval result includes candidate domain knowledge corresponding to the multi-modal data and candidate tool information required for processing the multi-modal data;

[0011] The retrieval result and the multi-modal data features are input into a large model for information analysis, to obtain prompt information and a target tool; the prompt information and the target tool are sent to a terminal, so that the terminal calls the target tool to process modal data corresponding to the prompt information, to obtain a data processing result.

[0012] In a second aspect, the present application provides a multi-modal data processing device based on a large model, comprising:

[0013] A feature processing module is configured to receive a data processing request of multi-modal data; extract features and perform association mining on the multi-modal data, to obtain multi-modal data features; the multi-modal data features represent semantic relationships and logical relationships between the multi-modal data;

[0014] A multi-modal retrieval module is configured to sequentially use a plurality of text matching methods to perform text retrieval on the multi-modal data features in a pre-constructed knowledge base according to a retrieval order of the plurality of text matching methods, to obtain a first associated subgraph; a second associated subgraph is determined based on semantic similarity between the multi-modal data features and multi-modal data in the pre-constructed knowledge base, and a similarity threshold value;

[0015] The multi-modal retrieval module is further configured to perform standard evaluation on a plurality of candidate results in the first associated subgraph and the second associated subgraph, and sort the plurality of candidate results according to the evaluation results, to obtain a retrieval result; the retrieval result includes candidate domain knowledge corresponding to the multi-modal data and candidate tool information required for processing the multi-modal data;

[0016] A data processing module is configured to input the retrieval result and the multi-modal data features into a large model for information analysis, to obtain prompt information and a target tool; the prompt information and the target tool are sent to a terminal, so that the terminal calls the target tool to process modal data corresponding to the prompt information, to obtain a data processing result.

[0017] In a third aspect, the present application also provides an electronic device, comprising a processor, wherein the processor is configured to implement the steps of the method for processing multi-modal data based on a large model according to any one of the above aspects when executing a computer program stored in a memory.

[0018] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program, wherein the computer program is configured to implement the steps of the method for processing multi-modal data based on a large model according to any one of the above aspects when executed by a processor.

[0019] In a fifth aspect, the present application provides a computer program product, comprising a computer program, wherein the computer program is configured to implement the method for processing multi-modal data based on a large model according to the first aspect when executed by a processor.

[0020] The embodiments of the present application provide at least the following beneficial effects:

[0021] In the embodiments of the present application, when searching based on multi-modal data, the multi-modal data is first subjected to feature extraction and correlation mining to obtain multi-modal data features, then a plurality of text matching methods are used in turn to perform text search on the multi-modal data features in a pre-constructed knowledge base according to the search order of the plurality of text matching methods, and semantic search is performed based on the semantic similarity between the multi-modal data features and the multi-modal data in the pre-constructed knowledge base, finally, the results of the text search and the semantic search are used for standard evaluation and sorting to obtain the final search results, thereby improving the accuracy of cross-modal search, solving the problem of single-modal semantic loss in the traditional RAG scheme, and improving the recall efficiency of multi-modal data and the relevance and coverage of the recall results.

[0022] Other features and advantages of the present application will be described in the following description, and some will become apparent from the description, or will be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the description. Obviously, the drawings in the following description are only some embodiments of the present application.

[0024] Figure 1 An application system schematic diagram of the multi-modal data processing method based on a large model provided by the embodiments of the present application;

[0025] Figure 2A flowchart of a multi-modal data processing method based on a large model provided by an embodiment of the present application is shown in FIG. 1.

[0026] Figure 3 A flowchart of a multi-modal data feature acquisition method provided by an embodiment of the present application is shown in FIG. 2.

[0027] Figure 4 A flowchart of a first associated subgraph acquisition method provided by an embodiment of the present application is shown in FIG. 3.

[0028] Figure 5 A flowchart of a second associated subgraph acquisition method provided by an embodiment of the present application is shown in FIG. 4.

[0029] Figure 6 A flowchart of a fine-grained sorting method provided by an embodiment of the present application is shown in FIG. 5.

[0030] Figure 7 An interaction diagram between a server and a terminal device provided by an embodiment of the present application is shown in FIG. 6.

[0031] Figure 8 A flowchart of constructing a knowledge base provided by an embodiment of the present application is shown in FIG. 7.

[0032] Figure 9 A structure diagram of a multi-modal data processing device based on a large model provided by an embodiment of the present application is shown in FIG. 8.

[0033] Figure 10 An electronic device structure diagram provided by an embodiment of the present application is shown in FIG. 9. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0035] It should be noted that the terms "include" and "have" and their variants as involved in the documents of the present application are intended to cover the inclusions without being exclusive, for example, a process, method, system, product or device containing a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0036] The terms "first", "second", etc. are used only for the purpose of description and are not to be interpreted in a relative manner unless otherwise specified. Thus, features defined with "first", "second" can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0037] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0038] Current multi-modal interaction systems, especially those involving voice interaction, are generally not end-to-end model implementation solutions, and require the integration of multiple types of models, such as large language models, speech recognition models, speech synthesis models, visual language multi-modal understanding models, etc.

[0039] For a voice input, speech recognition transcription needs to be performed first, and then input into a large model for understanding and generation. After the generation result, speech synthesis is performed, and finally feedback is given to the user. This approach often results in high interaction delay, as each step needs to be completed before the next step can be executed, especially in the case of low inference performance of the large model itself.

[0040] At the same time, most current RAG (Retrieval-augmented Generation) systems still remain in single-modal retrieval and generation, and even multi-modal RAGs only perform retrieval on single-modal data, and cannot perform multi-modal retrieval, resulting in low recall efficiency of multi-modal data and insufficient relevance and coverage of the recall results.

[0041] Therefore, in the processing of multi-modal data, the recall efficiency of multi-modal interaction systems and multi-modal retrieval are a current exploration direction.

[0042] Therefore, in the processing of multi-modal data, the recall efficiency of multi-modal interaction systems and multi-modal retrieval are a current exploration direction.

[0043] The inventive concept of the embodiments of the present application can be summarized as follows: receiving a data processing request of multi-modal data; performing feature extraction and correlation mining on the multi-modal data to obtain multi-modal data features; the multi-modal data features represent semantic relationships and logical relationships between the multi-modal data; according to the retrieval order of a plurality of text matching methods, the plurality of text matching methods are used in turn to perform text retrieval on the multi-modal data features in a pre-constructed knowledge base to obtain a first correlation subgraph; based on the semantic similarity between the multi-modal data features and the multi-modal data in the pre-constructed knowledge base, and a similarity threshold, a second correlation subgraph is determined; a plurality of candidate results in the first correlation subgraph and the second correlation subgraph are subjected to standard evaluation, and the plurality of candidate results are sorted according to the evaluation results to obtain a retrieval result; the retrieval result includes candidate domain knowledge corresponding to the multi-modal data and candidate tool information required for processing the multi-modal data; the retrieval result and the multi-modal data features are input into a large model for information analysis to obtain prompt information and a target tool; the prompt information and the target tool are sent to a terminal to enable the terminal to call the target tool to process the modal data corresponding to the prompt information to obtain a data processing result.

[0044] The multi-modal data processing method based on a large model provided in the present application is applied to an electronic device, which can be a PC, a mobile terminal, a terminal device, a server, etc. The multi-modal data processing method based on a large model provided in the present application can be applied to a distributed software platform represented by a blockchain.

[0045] After introducing the main inventive idea of the embodiments of the present application, the embodiments of the present application will be described in detail in combination with the accompanying drawings. Figure 1 The application system of the multi-modal data processing method based on a large model provided in the present application is described.

[0046] As shown in Figure 1 The application system of the multi-modal data processing method based on a large model in the present application includes the following modules:

[0047] Real-time interaction module: To achieve real-time communication of multi-modal, the application designs a high-performance real-time interaction module, which has the ability of real-time inference across text, audio and video, realizes smooth call experience, and supports users to interrupt AI in real time. As an interface that allows continuous data flow between applications, it does not need to make repeated requests. Unlike traditional HTTP request APIs, the real-time interaction module uses WebSocket to establish and maintain connections, and triggers data transmission through events (messages or notifications). When an event occurs, the server pushes relevant data to the connected terminal device, ensuring real-time updates. It realizes bidirectional and persistent communication between terminal devices and servers, so that data can be received immediately, providing real-time updates and seamless user interaction experience. The module integrates voice activity detection model (VAD), multi-modal key information retrieval and intelligent agent tool invocation capabilities during communication, realizes full-modal audio and video interaction function, supports interrupting and continuing at any time during interaction, invokes external tools to flexibly expand system capabilities, and greatly reduces the input cost during interaction of large models.

[0048] Among them, the voice activity detection model VAD detects whether the user is speaking, which is composed of preprocessing (responsible for audio normalization / framing), lightweight VAD model based on ONNX Runtime (sampling rate adaptive), context-aware decision engine (dynamic adjustment of silence threshold), realizes audio stream processing through a ring buffer, supports hot loading of model configurations in different scenarios, and outputs JSON events containing speaking start and end timestamps, confidence and interruption probability, so as to realize intelligent speaker detection and interruption judgment.

[0049] The VAD model runs on the fastest engine (ONNX Runtime) with the lightest and most efficient model, processes continuously like a pipeline, does not accumulate data, and uses context and dynamic threshold to make faster and more accurate judgments, reduces hesitation time, and controls the detection delay of audio and video to be small enough to be almost imperceptible.

[0050] Knowledge base construction module: Based on multi-modal data, this module processes and represents multi-modal data according to the dimensions of single-modal information processing, single-modal summary generation, mixed-modal information fusion, mixed-modal summary generation, and mixed-modal correlation mining, establishes a cross-modal semantic relationship graph, and obtains a constructed multi-modal knowledge base. The semantic relationship graph helps to improve the efficiency of multi-modal retrieval, and the semantic correlation and alignment across modalities also improve the accuracy of cross-modal retrieval and improve the problem of single-modal semantic loss in traditional RAG solutions.

[0051] Multi-modal retrieval module: after obtaining the data processing request of receiving multi-modal data through the real-time interaction module, the module is used to retrieve in the multi-modal knowledge base constructed by the knowledge base construction module to obtain the retrieval result. The module aims to align the input query and the multi-modal knowledge base. The module contains three retrieval optimization strategies of chain retrieval text recall, multi-modal embedding based cross recall and multi-modal data mixed fine ranking, which maximizes the use of multi-modal semantic information to provide high-quality and high-precision information retrieval capability. According to the actual measurement, the above retrieval scheme can effectively reduce the recall RAG size under the premise of maintaining the accuracy in the RAG system and the agent system under the large-scale tool set, thereby reducing the input cost of interaction with the large model and improving the model generation efficiency.

[0052] Therefore, through the above several modules, the application can be widely applied to traffic management field scenarios such as scene command, road hidden danger investigation, intelligent data analysis, etc., to improve the business knowledge retrieval accuracy and efficiency, and also provide a more friendly and natural interaction experience.

[0053] To further illustrate the technical solutions provided by the embodiments of the application, the following will be described in detail in conjunction with the drawings and specific embodiments. Although the embodiments of the application provide the method operation steps as described in the following embodiments or drawings, more or fewer operation steps can be included in the method based on conventional or non-creative labor. The execution order of these steps is not limited to the execution order provided by the embodiments of the application in the logical sense.

[0054] Referring to Figure 2 , a flowchart of a multi-modal data processing method based on a large model according to an embodiment of the application, the method can be specifically executed as the steps shown in Figure 2 .

[0055] In step S201, a data processing request of receiving multi-modal data is received; the multi-modal data is subjected to feature extraction and association mining to obtain multi-modal data features; the multi-modal data features represent the semantic relationship and logical relationship between the multi-modal data.

[0056] In the present application, a websocket connection is established between the terminal device and the server, and the server creates basic session information; then the terminal device completes session information management and setting, and the server updates the session information according to the session information management and setting and notifies the terminal device to enter the data transmission stage.

[0057] Then the terminal device appends text, image, audio and video multi-modal data to the ring buffer, and the server can receive the data processing request of the multi-modal data. The data processing request includes multi-modal data.

[0058] In a possible implementation, after receiving the data processing request of the multi-modal data, the voice activation detection (VAD) model is used to detect the audio and video data in the multi-modal data, to determine the valid speech segment of the audio and video data and the video frame corresponding to the valid speech segment.

[0059] In specific implementation, after receiving the data processing request, the server uses the VAD model to detect the audio and video data in the multi-modal data, to mark the start and end of the audio data in the audio and video data, that is, to obtain the valid speech segment, and to mark the video frame data in the time period corresponding to the valid speech segment.

[0060] In specific implementation, when uploading the video data, the terminal device disassembles the video into pictures frame by frame, adds accurate timestamps to each video frame, and then puts the video frames into the ring buffer in sequence.

[0061] In a possible implementation, in the actual interaction process of the present application, the user provides input data in multiple modalities, such as providing voice instructions and image data, and the multi-modal data needs to be first characterized and processed, to convert the multi-modal data into a semantic space similar to the pre-constructed knowledge base, so that the data can be accurately retrieved in the subsequent retrieval link. Therefore, the server extracts and correlates the features of the multi-modal data to obtain the multi-modal data features, which can be specifically implemented as shown in the following steps: Figure 3

[0062] In step S301, the multi-modal data is subjected to content analysis and feature extraction respectively to obtain the modal features corresponding to the multi-modal data respectively.

[0063] In specific implementation, the data of each modality, such as text, image, audio, and video, is subjected to independent feature extraction and content analysis, for example, text is subjected to word segmentation and semantic encoding, image is subjected to visual feature extraction, audio is subjected to sound spectrum analysis and speech-to-text conversion, and video is subjected to key frame extraction and dynamic behavior recognition, to obtain the modal features of the single-modal data.

[0064] In step S302, the multi-modal data is subjected to content analysis and feature extraction respectively to obtain the independent abstract features corresponding to the multi-modal data respectively.

[0065] In specific implementation, a compact abstract representation is generated for the data of a single modality, for example, text is subjected to abstract extraction, image is subjected to saliency region description, audio is subjected to key segment marking, and video is subjected to plot summary generation, to obtain the independent abstract features of the single-modal data, so as to reduce storage overhead and improve retrieval efficiency.

[0066] In step S303, the multi-modal data is subjected to information fusion to obtain the semantic correlation features between different modal data. ​

[0067] In specific implementation, the modal encoder of the multi-modal large model is extracted through the cross-modal alignment technology, the data of different modalities are associated at the semantic level, information fusion is performed, and the semantic association features between different modal data are obtained. For example, matching objects in images with text descriptions, combining dialogues in videos with audio emotion analysis to form a unified representation space.

[0068] In addition, for video and audio data, a time axis alignment needs to be established to realize a synchronization mechanism of video and audio input, and audio and video frames are organized into an interleaved structure to ensure that the audio and visual signals of the video content are time-synchronized.

[0069] In step S304, information fusion is performed on the multi-modal data to obtain comprehensive summary features between different modal data.

[0070] In specific implementation, based on the complementarity of multi-modal data, information fusion is performed on the multi-modal data to obtain comprehensive summary features between different modal data, such as joint text and image summaries or audio and video summaries, for example, generating a conference minutes by combining speech audio and PPT slides, or generating an event report by fusing surveillance video and sensor data.

[0071] In step S305, association mining is performed on the multi-modal data to obtain logical relationships between different modal data.

[0072] In specific implementation, association mining is performed on the multi-modal data to obtain logical relationships between different modal data, and then a graph neural network or a knowledge graph is used to explicitly construct logical relationships between multi-modal data, such as spatio-temporal association, causal chain, and theme clustering, for example, associating pictures, subtitles, and background music in a news video to the same event node.

[0073] In this way, the input multi-modal data can be represented and processed, the multi-modal data can be converted into a semantic space similar to the knowledge base, the semantic relationships and logical relationships between the multi-modal data can be established, and therefore accurate data retrieval can be achieved in the subsequent retrieval process.

[0074] In step S202, according to the retrieval order of the plurality of text matching methods, the plurality of text matching methods are used in sequence to perform text retrieval on the multi-modal data features in the pre-constructed knowledge base, and a first associated subgraph is obtained; based on the semantic similarity between the multi-modal data features and the multi-modal data in the pre-constructed knowledge base and a similarity threshold, a second associated subgraph is determined.

[0075] The similarity threshold in the present application can be set according to actual needs, which is not limited in the present application.

[0076] When using the text matching method for data retrieval and recall, it is often difficult to accurately locate relevant information from massive data. Therefore, in the text retrieval part of the system, a chain recall strategy is designed to combine commonly used text matching methods to achieve lightweight and high-performance text accurate recall in a wide range of text data.

[0077] In one possible implementation, the text retrieval is performed using the chain retrieval text recall strategy, so in this application, according to the retrieval order of multiple text matching methods, multiple text matching methods are used in turn to perform text retrieval on the multi-modal data features in the pre-constructed knowledge base, and a first associated subgraph is obtained, which can be executed as shown in the following steps: Figure 4

[0078] In step S401, for multiple text matching methods, the following operations are performed: according to the retrieval order corresponding to the text matching method, the text matching method is used to perform text retrieval on the multi-modal data features from the remaining data in the pre-constructed knowledge base, and the retrieval result corresponding to the text matching method is obtained.

[0079] Among them, the remaining data is the data in the pre-constructed knowledge base except the retrieval result corresponding to the target method set; the target method set includes all text matching methods whose retrieval order is before the text matching method.

[0080] Among them, the retrieval order is determined according to the priority of multiple text matching methods. The priority of multiple text matching methods can be set according to actual needs, which is not limited in this application.

[0081] The commonly used text matching methods in this application include but are not limited to BM25, Jaccard similarity, edit distance similarity, maximum common subsequence, and maximum common substring.

[0082] In specific implementation, in the pre-constructed knowledge base, single-step retrieval is performed by a single text matching method to obtain Topk retrieval results, and then the Topk retrieval results are deleted from the pre-constructed knowledge base, and a second text matching method is used to continue retrieval from the remaining data in the pre-constructed knowledge base, until all multiple text matching methods are executed to obtain the retrieval results of multiple text matching methods.

[0083] In step S402, the retrieval results corresponding to multiple text matching methods are weighted and sorted to obtain the matching results.

[0084] In specific implementation, after obtaining the retrieval results of multiple text matching methods, the retrieval results of multiple text matching methods are weighted and sorted to obtain the matching results.

[0085] ​For example, the retrieval results corresponding to the three text matching methods of Jaccard similarity, edit distance similarity and maximum common subsequence are weighted and sorted. Among them, Jaccard similarity is used to represent the consistency of query and key information of recall data; edit distance similarity is used to represent the consistency of query and expression method of recall data; maximum common subsequence is used to represent the consistency of query and expression method and key information of recall data; therefore, the retrieval results corresponding to the three text matching methods can be weighted according to the weight ratio of 3:3:4 to obtain the rough sorted matching results.

[0086] In the present application, the weights of the retrieval results corresponding to various text matching methods can be set according to actual needs, which are not limited in the present application.

[0087] In step S403, a first connected subgraph associated with the matching result is obtained from the semantic relation graph of the pre-constructed knowledge base; the first connected subgraph is merged to obtain a first associated subgraph.

[0088] In specific implementation, after obtaining the matching result, the connected subgraph related to the data in the matching result is obtained from the multi-modal knowledge base, and the connected subgraph is de-duplicated and merged to obtain the final first associated subgraph. The first associated subgraph is the text recall output result of chain retrieval.

[0089] The text recall strategy of chain retrieval has the advantages of fast speed and light weight, and is more accurate than the retrieval result of a single text matching method, and is suitable for pure text query, and needs to be quickly and lightly recalled in massive text data. As the text starting point of the multi-modal retrieval module in the present application.

[0090] In order to realize cross-modal semantic association, the cross-recall strategy based on multi-modal embedding is used in the present application.

[0091] In a possible implementation, the cross-recall strategy based on multi-modal embedding is used for semantic information retrieval, so that in the present application, the second associated subgraph is determined based on the semantic similarity between the multi-modal data features and the multi-modal data in the pre-constructed knowledge base, and the similarity threshold, which can be executed as shown in the following steps: Figure 5

[0092] In step S501, the multi-modal data features and the multi-modal data in the pre-constructed knowledge base are mapped to the same vector space to obtain the reference features of the multi-modal data in the pre-constructed knowledge base.

[0093] ​In specific implementation, first, the multi-modal data is embedded and aligned, that is, the multi-modal data features and the multi-modal data in the pre-constructed knowledge base are input into the embedding layer of the multi-modal large model, and different modal data such as text, image, audio and video are mapped to a unified semantic vector space.

[0094] In step S502, for the multi-modal data features, the semantic similarity between the modal data features and the reference features of the multi-modal data in the pre-constructed knowledge base is determined, and the reference features of the multi-modal data with a semantic similarity greater than a similarity threshold are taken as candidate retrieval results corresponding to the modal data features.

[0095] For example, for the text features, the semantic similarity between the text features and all modal data in the pre-constructed knowledge base is calculated:

[0096] First, intra-modal retrieval is performed: the cosine similarity between the text features and each text data in the pre-constructed knowledge base is calculated to obtain the semantic similarity;

[0097] Then, cross-modal retrieval is performed: the modal similarity between the text features and each non-text data such as image data, audio data and video data in the pre-constructed knowledge base is calculated to obtain the semantic similarity;

[0098] Finally, each data with a semantic similarity exceeding a similarity threshold is selected as a candidate retrieval result corresponding to the text features. The similarity threshold can be set according to actual needs, for example, it can be set to Top 20% similarity.

[0099] In step S503, the candidate retrieval results corresponding to the multi-modal data features are constructed into a second connected subgraph, and node merging and pruning are performed on the second connected subgraph to obtain a second associated subgraph.

[0100] In specific implementation, after obtaining the candidate retrieval results, the candidate retrieval results are modeled into a graph structure, the nodes are reference features (modal data in the pre-constructed knowledge base), and the edge weights are the semantic similarity between the modal data, to obtain a second connected subgraph. For example, text node A is connected to image node B due to high similarity, and video node C is connected to text node A due to semantic association.

[0101] Then, based on the graph connectivity, the second connected subgraph is extracted to obtain a second associated subgraph:

[0102] First, node merging is performed: repeated nodes (such as images, texts and the like describing the same event) are aggregated;

[0103] Pruning is performed: edges with weights lower than a threshold (weakly associated edges) are removed, and strongly semantically connected subgraphs are retained as a candidate set.

[0104] The cross-recall strategy based on multi-modal embedding can realize true cross-modal retrieval, such as retrieving image, audio, video and other data with text data, retrieving text, audio, video and other data with image, can capture the semantic association between modal data, the result structure is more clear, and is suitable for scenes that need to perform cross-modal retrieval and mine deep semantic association.

[0105] The result of the text recall strategy of the chain retrieval comes from explicit semantic matching, and the result of the cross-recall strategy based on multi-modal embedding is semantic information. To solve the evaluation standard conflict caused by the inconsistency of the two methods, a comprehensive analysis and evaluation module is needed to realize a multi-objective optimization sorting model, and different forms of data are evaluated with the same standard. Therefore, the multi-modal data mixed fine sorting strategy is proposed in the present application.

[0106] In step S203, the multiple candidate results in the first association subgraph and the second association subgraph are evaluated according to the standard, and the multiple candidate results are sorted according to the evaluation results to obtain the retrieval result.

[0107] Among them, the retrieval result includes candidate field knowledge corresponding to the multi-modal data and candidate tool information required for processing the multi-modal data.

[0108] Among them, the first association subgraph includes multiple candidate results of the multi-modal data; the second association subgraph includes multiple candidate results of the multi-modal data; each candidate result includes at least one modal data.

[0109] In one possible implementation, the multi-modal data mixed fine sorting strategy is used for standard evaluation and sorting. Therefore, when the multiple candidate results in the first association subgraph and the second association subgraph are evaluated according to the standard, and the multiple candidate results are sorted according to the evaluation results to obtain the retrieval result in the present application, the steps shown in Figure 6 are respectively performed on the multiple candidate results:

[0110] In step S601, the cosine similarity is used to determine the semantic similarity between the multi-modal data features and the candidate results.

[0111] In specific implementation, the cosine similarity is used to calculate the semantic similarity between the query vector (input modal data features) and the candidate result vector. A fine-grained semantic matching mechanism is introduced, such as based on entity linking, relation extraction and other technologies, to identify the deep semantic association between the query and the candidate result.

[0112] The semantic similarity is determined according to the following formula:

[0113] Sim(q,d)=cosine_similarity(q,d)

[0114] wherein Sim(q, d fuse) represents the semantic similarity between the modal data feature q and the candidate result d, and cosine similarity(q, d) represents the cosine similarity between the modal data feature q and the candidate result d.

[0115] wherein the feature vector of the candidate result is obtained by using multi-modal feature extraction and cross-modal interaction learning.

[0116] In step S602, the modal complementarity score of the multi-modal data feature and the candidate result is determined based on the modal coverage of the multi-modal data feature and the candidate result, and the modal interaction feature of the multi-modal data feature and the candidate result.

[0117] In implementation, the modal coverage index is defined in advance to measure different modal data types contained in the candidate result. For example, the candidate result A contains text and image, and the candidate result B contains only text, so the modal coverage of the candidate result A is higher than that of the candidate result B. The modal interaction feature also needs to be introduced in advance to evaluate the complementarity of multi-modal data in the candidate result. For example, the matching degree of the text description and the image content.

[0118] After determining the modal coverage and the modal interaction feature of the candidate result, the modal complementarity score is determined according to the following formula:

[0119] MC(q, d) = a * M Coverage(d) + β * M Interaction(q, d)

[0120] wherein MC(q, d) represents the modal complementarity score of the modal data feature q and the candidate result d, MCoverage(d) represents the modal coverage of the candidate result d, M Interaction(q, d) represents the modal interaction feature of the modal data feature q and the candidate result d, and a and β are weight parameters.

[0121] In step S603, the information density score of the candidate result is determined based on the information density of each modal data in the candidate result.

[0122] In implementation, the information density of text, the information density of image, and the information density of audio / video are defined in advance; wherein the information density of text is obtained by combining text length, keyword density, named entity quantity and other factors to evaluate the amount of text information. The information density of image is obtained by using image sharpness, object quantity, scene complexity and other indicators to evaluate the amount of image information. The information density of audio / video is obtained by analyzing the time sequence information, content richness and other features of audio / video to evaluate the amount of information.

[0123] The information density score of the candidate result is determined according to the following formula:

[0124] ID(d) = γ * Text Density(d) + δ * Image Density(d) + ε * Audio / Video Density(d)

[0125] wherein ID(d) represents the information density score of the candidate result d, Text Density(d), Image Density(d) and Audio / Video Density(d) represent the information density of the text, the information density of the image and the information density of the audio / video respectively, and γ, δ and ε are weight parameters.

[0126] In step S604, the semantic similarity score, the modality complementarity score and the information density score are weighted and summed to obtain a target score of the candidate result.

[0127] The target score of the candidate result is determined according to the following formula:

[0128] Score(q, d) = λ1 * Sim(q, d) + λ2 * MC(q, d) + λ3 * ID(d)

[0129] wherein Score(q, d) represents the target score between the modality data feature q and the candidate result d; Sim(q, d_fuse) represents the semantic similarity between the modality data feature q and the candidate result d; MC(q, d) represents the modality complementarity score between the modality data feature q and the candidate result d; ID(d) represents the information density score of the candidate result d; λ1, λ2 and λ3 are weight parameters.

[0130] In step S605, the multiple candidate results are sorted according to the target score to obtain the retrieval result.

[0131] The higher the target score is, the higher the ranking of the candidate result is, and the N candidate results with the highest ranking are used as the final retrieval result. N is a positive integer greater than or equal to 1, which can be set according to actual needs.

[0132] When the multi-modal data hybrid fine ranking strategy is subjected to multi-task learning, the semantic relevance, the modality complementarity and the information density are jointly trained as three independent tasks, the bottom layer feature representation is shared, and the generalization ability of the model is improved.

[0133] The input modal data feature q and the candidate result d are first subjected to multi-modal feature extraction and cross-modal interaction learning, feature fusion is performed, and a feature vector d_fuse of the candidate result is obtained; then the semantic similarity, the modal complementarity score and the information density score are determined according to the above manner; the semantic similarity, the modal complementarity score and the information density score are weighted and summed to obtain a target score of the candidate result; all candidate results are ranked according to the target score, and a ranking result is returned.

[0134] The multi-modal data hybrid fine ranking strategy can uniformly evaluate multi-source / multi-modal retrieval results, and the ranking result is more accurate and comprehensive. It is suitable for scenarios that require final fine ranking of mixed results of multiple recall channels (text, embedding, etc.) and have extremely high requirements for result quality.

[0135] The multi-modal retrieval module in the present application takes text retrieval as the basis and extends the cross-retrieval method of multi-modal embedding, realizes comprehensive retrieval of multi-modal data, and realizes accurate evaluation of the retrieval results through the multi-modal hybrid fine ranking method.

[0136] In step S204, the retrieval result and the multi-modal data feature are input into the large model for information analysis to obtain prompt information and a target tool; the prompt information and the target tool are sent to the terminal to enable the terminal to call the target tool to process the modal data corresponding to the prompt information to obtain a data processing result.

[0137] In specific implementation, after the server obtains the retrieval result through the multi-modal retrieval module, the multi-modal data feature and the retrieval result are input into the large model for analysis to generate a large model reply (prompt information) and target tool calling information (which may return multiple tool calls), the terminal device calls and executes the target tool to obtain a calling result of the target tool (when multiple tools are called in parallel, the calling result of all functions needs to be reported before the trigger model is replied), and the terminal device submits the tool execution result, i.e., the data processing result. The server receives and saves the data processing result.

[0138] For ease of understanding, the multi-modal data processing method based on a large model provided by the present application will be described below in conjunction with the interaction schematic diagram between a server and a terminal device as shown in Figure 7

[0139] In step E1, a websocket connection is established between the terminal device and the server.

[0140] In step E2, the server creates basic session information.

[0141] In step E3, the terminal device completes session information management and setting, and sends the session information and setting information to the server. ​

[0142] In step E4, the server updates the session information and notifies the terminal device to perform data transmission;

[0143] In step E5, after receiving the data transmission notification, the terminal device maintains a buffer, inputs the multi-modal data into the buffer, and generates a data processing request;

[0144] In step E6, after receiving the data processing request, the server uses the VAD model to determine the valid speech segment and the video frame corresponding to the valid speech segment;

[0145] In step E7, the terminal device notifies the server that the data transmission has been completed;

[0146] In step E8, the server extracts and correlates the multi-modal data to obtain multi-modal data features and cache them in the current session;

[0147] In step E9, the server obtains a search result by a multi-modal search module based on the multi-modal data features, and registers the search result in the current session;

[0148] The search result includes candidate domain knowledge corresponding to the multi-modal data and candidate tool information required for processing the multi-modal data.

[0149] It should be noted that after the server extracts and correlates the multi-modal data to obtain multi-modal data features and caches them in the current session, the terminal device can also obtain a search result by a multi-modal search module based on the multi-modal data features, and register the search result in the current session.

[0150] In step E10, the server inputs the multi-modal data features and the search result into a large model LLM for analysis to generate a large model reply (prompt information) and target tool invocation information;

[0151] In step E11, the terminal device invokes and executes the target tool to obtain the invocation result of the target tool and submits the tool execution result, i.e., the data processing result;

[0152] In step E12, the server receives the data processing result and caches it in the current session.

[0153] If further execution is required, the large model reply is triggered and the process jumps to step E5 for further execution, and the complete context of the session is maintained. It should be noted that steps E8-E10 can be executed in the terminal device.

[0154] The multi-modal data in the present application refers to data such as text, audio, image, and video. The knowledge base construction module is used to process and store original documents, images, audio and video data, form a knowledge base for the multi-modal retrieval module. The semantic and logical association between multi-modal data is established, so that more comprehensive and efficient relevant information can be obtained during retrieval.

[0155] In a possible implementation, the knowledge base in the present application can be constructed according to the steps as shown in Figure 8

[0156] In step S801, multi-modal sample data is obtained.

[0157] In step S802, the multi-modal sample data is respectively subjected to content analysis and feature extraction, to obtain the modal features and independent abstract features corresponding to the multi-modal sample data respectively.

[0158] In step S803, the multi-modal sample data is subjected to information fusion, to obtain the semantic association features and comprehensive abstract features between different modal sample data.

[0159] In step S804, the multi-modal sample data is subjected to association mining, to obtain the logical relationship between different modal sample data.

[0160] In step S805, based on the modal features and independent abstract features corresponding to the multi-modal sample data respectively, and the semantic association features, comprehensive abstract features and logical relationship between different modal sample data, the knowledge base of the multi-modal sample data is constructed.

[0161] In specific implementation, the functions of the knowledge base construction module cover the following key levels:

[0162] Single-modal information processing: independent feature extraction and content analysis are performed on data of each modality, such as text, image, audio, and video, for example, word segmentation and semantic coding are performed on text, visual feature extraction is performed on image, sound spectrum analysis and speech-to-text are performed on audio, and key frame extraction and dynamic behavior recognition are performed on video, to obtain the modal features of single-modal data.

[0163] Single-modal abstract generation: a compact abstract representation is generated for single-modal data, for example, abstract extraction is performed on text, saliency region description is performed on image, key segment marking is performed on audio, and plot summary generation is performed on video, to obtain the independent abstract features of single-modal data, so as to reduce storage overhead and improve retrieval efficiency.

[0164] ​Mixed modal information fusion: Through cross-modal alignment technology, the modal encoder of the multi-modal large model is extracted, and the data of different modalities are associated at the semantic level for information fusion to obtain the semantic association features between different modal data. For example, matching objects in images with text descriptions, combining dialogues in videos with audio emotion analysis to form a unified representation space.

[0165] In addition, for video and audio data, a time axis alignment needs to be established to realize the synchronization mechanism of video and audio input, and audio and video frames are organized into an interleaved structure to ensure that the audio and visual signals of the video content are time-synchronized.

[0166] Mixed modal summary generation: Based on the complementarity of multi-modal data, information fusion is performed on multi-modal data to obtain comprehensive summary features between different modal data, such as joint text and image summaries or audio and video summaries, for example, generating a conference minutes by combining speech audio and PPT slides, or generating an event report by fusing surveillance video and sensor data.

[0167] Mixed modal association mining: The logical relationship between different modal data is obtained by associating and mining multi-modal data, and then a graph neural network or a knowledge graph is used to explicitly construct the logical relationship between multi-modal data, such as temporal and spatial association, causal chain, and theme clustering, for example, associating pictures, subtitles, and background music in a news video to the same event node.

[0168] In one possible implementation, based on the modal features and independent summary features corresponding to the multi-modal sample data respectively, as well as the semantic association features, comprehensive summary features, and logical relationships between different modal sample data, a knowledge base of multi-modal sample data can be constructed, which can be implemented in the following two ways:

[0169] Implementation one: Based on the modal features and independent summary features corresponding to the multi-modal sample data respectively, as well as the semantic association features, comprehensive summary features, and logical relationships between different modal sample data, a semantic relationship graph of multi-modal sample data is constructed to obtain a completed knowledge base.

[0170] In specific implementation, multi-modal data is processed and represented according to the dimensions of single-modal information processing, single-modal summary generation, mixed modal information fusion, mixed modal summary generation, and mixed modal association mining, a cross-modal semantic relationship graph is established, and a completed multi-modal knowledge base is obtained.

[0171] Implementation two: Based on the modal features and independent summary features corresponding to the multi-modal sample data respectively, as well as the semantic association features, comprehensive summary features, and logical relationships between different modal sample data, a knowledge graph of multi-modal sample data is constructed to obtain a completed knowledge base.

[0172] In specific implementation, the knowledge graph is used to construct data association between different modal data, and meanwhile, data information is extracted to construct a domain knowledge graph, so that the original data information has more extensive semantics and information. Through the combination of the domain multi-modal knowledge graph construction technology, the inference ability of the large model is greatly improved, so that the large model can better complete the domain task and reduce hallucination.

[0173] It should be noted that in addition to the multi-modal data retrieval involved in the present application, the real-time interaction module of the present application can also be applied in a modal-based large model intelligent interaction assistant to realize real-time audio and video stream transmission, understanding and generation. The Websocket can be replaced by WebRTC DataChannel, GRPC, etc. All weight parameters and weights in weighted summation in the embodiments of the present application can be set according to actual needs.

[0174] Therefore, in the present application, when retrieving based on multi-modal data, the multi-modal data is first subjected to feature extraction and association mining to obtain multi-modal data features, then a plurality of text matching methods are used in sequence according to the retrieval order of the plurality of text matching methods to perform text retrieval on the multi-modal data features in the pre-constructed knowledge base, and based on the semantic similarity between the multi-modal data features and the multi-modal data in the pre-constructed knowledge base, semantic retrieval is performed, finally the results of the text retrieval and the semantic retrieval are used for standard evaluation and sorting to obtain the final retrieval result, thereby improving the accuracy of cross-modal retrieval, improving the problem of single modal semantic loss in the traditional RAG scheme, and improving the recall efficiency of multi-modal data and the relevance and coverage of the recall result.

[0175] Based on the same inventive concept, the embodiments of the present application also provide a multi-modal data processing device based on a large model, Figure 9 A multi-modal data processing device based on a large model provided by the embodiments of the present application has a structure diagram, which comprises:

[0176] The feature processing module 901 is configured to receive a data processing request of multi-modal data, perform feature extraction and association mining on the multi-modal data to obtain multi-modal data features, and the multi-modal data features represent semantic relationships and logical relationships between the multi-modal data.

[0177] The multi-modal retrieval module 902 is configured to use a plurality of text matching methods in sequence according to a retrieval order of the plurality of text matching methods to perform text retrieval on the multi-modal data features in a pre-constructed knowledge base to obtain a first associated subgraph, and determine a second associated subgraph based on a semantic similarity between the multi-modal data features and multi-modal data in the pre-constructed knowledge base and a similarity threshold.

[0178] The multi-modal retrieval module 902 is further configured to perform standard evaluation on the plurality of candidate results in the first associated subgraph and the second associated subgraph, and sort the plurality of candidate results according to the evaluation results to obtain a retrieval result; the retrieval result includes candidate domain knowledge corresponding to the multi-modal data and candidate tool information required for processing the multi-modal data;

[0179] The data processing module 903 is configured to input the retrieval result and the multi-modal data feature into a large model to perform information analysis, and obtain prompt information and a target tool; and send the prompt information and the target tool to the terminal, so that the terminal calls the target tool to process the modal data corresponding to the prompt information to obtain a data processing result.

[0180] In a possible implementation, the multi-modal retrieval module 902 is specifically configured to:

[0181] The following operations are performed for each of the plurality of text matching methods: according to a retrieval order corresponding to the text matching method, the text matching method is used to perform text retrieval on the multi-modal data feature from the remaining data in the pre-constructed knowledge base, to obtain a retrieval result corresponding to the text matching method;

[0182] The remaining data is data in the pre-constructed knowledge base other than the retrieval result corresponding to the target method set; the target method set includes all text matching methods whose retrieval orders are in front of the text matching method; the retrieval order is determined according to the priority of the plurality of text matching methods;

[0183] The retrieval results corresponding to the plurality of text matching methods are weighted and sorted to obtain a matching result;

[0184] A first connected subgraph associated with the matching result is obtained from a semantic relationship graph of the pre-constructed knowledge base; and the first connected subgraph is merged to obtain a first associated subgraph.

[0185] In a possible implementation, the multi-modal retrieval module 902 is specifically configured to:

[0186] The multi-modal data feature and the multi-modal data in the pre-constructed knowledge base are mapped to the same vector space to obtain reference features of the multi-modal data in the pre-constructed knowledge base;

[0187] For the multi-modal data feature, the following operations are performed: determining a semantic similarity between the modal data feature and the reference features of the multi-modal data in the pre-constructed knowledge base; and taking the reference features of the multi-modal data whose semantic similarity is greater than a similarity threshold as candidate retrieval results corresponding to the modal data feature;

[0188] The candidate retrieval results corresponding to the multi-modal data feature are constructed into a second connected subgraph; and the second connected subgraph is merged and pruned to obtain a second associated subgraph.

[0189] In a possible implementation, at least one modality data is included in each candidate result;

[0190] The multi-modal retrieval module 902 is specifically configured to:

[0191] The following operations are respectively performed on the plurality of candidate results:

[0192] The semantic similarity between the multi-modal data feature and the candidate result is determined by using the cosine similarity;

[0193] The modality complementarity score between the multi-modal data feature and the candidate result is determined based on the modality coverage of the multi-modal data feature and the candidate result and the modality interaction feature of the multi-modal data feature and the candidate result;

[0194] The information density score of the candidate result is determined based on the information density of each modality data in the candidate result;

[0195] The semantic similarity, the modality complementarity score, and the information density score are weighted and summed to obtain the target score of the candidate result;

[0196] The plurality of candidate results are sorted according to the target score to obtain the retrieval result.

[0197] In a possible implementation, the apparatus further includes a knowledge base construction module 904;

[0198] The knowledge base construction module 904 is configured to construct the knowledge base according to the following manner:

[0199] Obtain multi-modal sample data;

[0200] Perform content analysis and feature extraction on the multi-modal sample data respectively to obtain modality features and independent abstract features corresponding to the multi-modal sample data respectively;

[0201] Perform information fusion on the multi-modal sample data to obtain semantic association features and comprehensive abstract features between different modality sample data;

[0202] Perform association mining on the multi-modal sample data to obtain logical relationships between different modality sample data;

[0203] Construct a knowledge base of the multi-modal sample data based on the modality features and the independent abstract features corresponding to the multi-modal sample data respectively, and the semantic association features, the comprehensive abstract features, and the logical relationships between different modality sample data.

[0204] In a possible implementation, the knowledge base construction module 904 is specifically configured to: based on the modal features and independent abstract features respectively corresponding to the multi-modal sample data, and the semantic association features, the comprehensive abstract features and the logical relationships between different modal sample data, construct a semantic relationship graph of the multi-modal sample data, to obtain the constructed knowledge base.

[0205] Or,

[0206] Based on the modal features and independent abstract features respectively corresponding to the multi-modal sample data, and the semantic association features, the comprehensive abstract features and the logical relationships between different modal sample data, construct a knowledge graph of the multi-modal sample data, to obtain the constructed knowledge base.

[0207] In a possible implementation, the apparatus further includes a real-time detection module 905.

[0208] The real-time detection module 905 is specifically configured to: use a voice activation detection (VAD) model to detect the audio-video data in the multi-modal data, to determine the valid speech segments of the audio-video data and the video frames corresponding to the speech segments.

[0209] Based on the same inventive concept, the embodiments of the present application also provide an electronic device, Figure 10 A structural schematic diagram of an electronic device provided by the embodiments of the present application is shown in Figure 10 As shown, it includes a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002 and the memory 1003 complete mutual communication through the communication bus 1004.

[0210] The memory 1003 stores a computer program, and when the program is executed by the processor 1001, the processor 1001 executes the steps of any one of the multi-modal data processing methods based on large models provided by the embodiments of the present application.

[0211] Since the above-mentioned electronic device solves the problem similar to the multi-modal data processing method based on large models, the implementation of the above-mentioned electronic device can refer to the embodiments of the method, and the repeated parts will not be described here.

[0212] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 1302 is used for communication between the above electronic device and other devices. The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0213] The processor mentioned above can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; can also be a Digital Signal Processing (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0214] On the basis of the above embodiments, the embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program executable by a processor. When the program runs on the processor, the processor executes the steps of any one of the large model-based multi-modal data processing methods provided by the embodiments of the present application.

[0215] Based on the same inventive concept, the embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any one of the large model-based multi-modal data processing methods provided by the embodiments of the present application are implemented.

[0216] Since the above computer readable storage medium solves the problem by the same principle as the large model-based multi-modal data processing method, the implementation of the above computer readable storage medium can refer to the embodiments of the method, and the repeated parts will not be described here.

[0217] Those skilled in the art will appreciate that embodiments of the present application can be devised for a variety of other systems which are currently developed or later developed. Therefore, the present application is intended to cover all such modifications and variations of this application that are within the scope of the appended claims and their equivalents. It is intended that each element of claim 1 and 2 is independent of one another. No element of claim 1 and 2, or any other claim, is implied to depend on any other element or limitation of claim 1 and 2 or any other claim except where expressly recited in that claim.

[0218] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to this application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 Figure 1

[0219] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 Figure 1

[0220] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 Figure 1

[0221] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.​​​​​​

Claims

1. A large model-based multi-modal data processing method, characterized in that, The method comprises: receiving a data processing request of multi-modal data; performing feature extraction and association mining on the multi-modal data to obtain multi-modal data features; the multi-modal data features represent semantic relationships and logical relationships between the multi-modal data; sequentially using a plurality of text matching methods in a retrieval order to perform text retrieval on the multi-modal data features in a pre-constructed knowledge base to obtain a first association subgraph; determining a second association subgraph based on semantic similarity between the multi-modal data features and multi-modal data in the pre-constructed knowledge base and a similarity threshold; performing standard evaluation on a plurality of candidate results in the first association subgraph and the second association subgraph, and sorting the plurality of candidate results according to the evaluation results to obtain a retrieval result; the retrieval result includes candidate domain knowledge corresponding to the multi-modal data and candidate tool information required for processing the multi-modal data; inputting the retrieval result and the multi-modal data features into a large model for information analysis to obtain prompt information and a target tool; and sending the prompt information and the target tool to a terminal to enable the terminal to call the target tool to process modal data corresponding to the prompt information to obtain a data processing result.

2. The method of claim 1, wherein, The method comprises: for each of the plurality of text matching methods, performing the following operations: using the text matching method to perform text retrieval on the multi-modal data features from remaining data in the pre-constructed knowledge base according to a retrieval order corresponding to the text matching method to obtain a retrieval result corresponding to the text matching method; wherein the remaining data is data in the pre-constructed knowledge base excluding retrieval results corresponding to a target method set; the target method set includes all text matching methods whose retrieval orders are before the text matching method; the retrieval order is determined according to priorities of the plurality of text matching methods; performing weighted sorting on the retrieval results corresponding to the plurality of text matching methods to obtain a matching result; obtaining a first connected subgraph associated with the matching result from a semantic relationship graph in the pre-constructed knowledge base; and performing node merging on the first connected subgraph to obtain a first association subgraph.

3. The method of claim 1, wherein, The method comprises: mapping the multi-modal data features and the multi-modal data in the pre-constructed knowledge base to the same vector space to obtain reference features of the multi-modal data in the pre-constructed knowledge base; for each of the multi-modal data features, performing the following operations: determining semantic similarity between the modal data features and the reference features of the multi-modal data in the pre-constructed knowledge base; and taking reference features of multi-modal data with a semantic similarity greater than the similarity threshold as candidate retrieval results corresponding to the modal data features. The candidate search results corresponding to the multi-modal data features are constructed into a second connected subgraph; node merging and pruning are performed on the second connected subgraph to obtain a second associated subgraph.

4. The method of claim 1, wherein, Each candidate result includes at least one modal data; The standard evaluation is performed on the plurality of candidate results in the first associated subgraph and the second associated subgraph, and the plurality of candidate results are sorted according to the evaluation results to obtain a search result, including: The following operations are performed on the plurality of candidate results respectively: The semantic similarity between the multi-modal data features and the candidate results is determined by using cosine similarity; Based on the modal coverage of the multi-modal data features and the candidate results and the modal interaction features of the multi-modal data features and the candidate results, a modal complementarity score of the multi-modal data features and the candidate results is determined; Based on the information density of each modal data in the candidate result, an information density score of the candidate result is determined; The semantic similarity, the modal complementarity score, and the information density score are weighted and summed to obtain a target score of the candidate result; The plurality of candidate results are sorted according to the target score to obtain a search result.

5. The method according to any of claims 1 to 4, characterized in that, The knowledge base is constructed in the following manner: Multi-modal sample data are obtained; Content analysis and feature extraction are performed on the multi-modal sample data respectively to obtain modal features and independent abstract features corresponding to the multi-modal sample data respectively; Information fusion is performed on the multi-modal sample data to obtain semantic association features and comprehensive abstract features between different modal sample data; Association mining is performed on the multi-modal sample data to obtain logical relationships between different modal sample data; Based on the modal features and independent abstract features corresponding to the multi-modal sample data respectively, and the semantic association features, comprehensive abstract features, and logical relationships between different modal sample data, a knowledge base of the multi-modal sample data is constructed.

6. The method of claim 5, wherein, The knowledge base of the multi-modal sample data is constructed based on the modal features and independent abstract features corresponding to the multi-modal sample data respectively, and the semantic association features, comprehensive abstract features, and logical relationships between different modal sample data, including: Based on the modal features and independent abstract features corresponding to the multi-modal sample data respectively, and the semantic association features, comprehensive abstract features, and logical relationships between different modal sample data, a semantic relationship graph of the multi-modal sample data is constructed to obtain a constructed knowledge base; Alternatively, Based on the modal features and independent abstract features corresponding to the multi-modal sample data respectively, and the semantic association features, comprehensive abstract features, and logical relationships between different modal sample data, a knowledge graph of the multi-modal sample data is constructed to obtain a constructed knowledge base.

7. The method of claim 1, wherein, After receiving the data processing request of the multi-modal data, the following steps are further included: A voice activity detection (VAD) model is used to detect audio and video data in the multi-modal data to determine valid voice segments of the audio and video data and video frames corresponding to the valid voice segments.

8. A large model-based multi-modal data processing apparatus, characterized by, The device includes: The feature processing module is configured to receive a data processing request of multi-modal data, perform feature extraction and correlation mining on the multi-modal data to obtain multi-modal data features, and the multi-modal data features represent semantic relationships and logical relationships between the multi-modal data. The multi-modal retrieval module is configured to sequentially use a plurality of text matching methods in a retrieval order of the plurality of text matching methods to perform text retrieval on the multi-modal data features in a pre-constructed knowledge base to obtain a first correlation subgraph, and determine a second correlation subgraph based on semantic similarity between the multi-modal data features and multi-modal data in the pre-constructed knowledge base and a similarity threshold. The multi-modal retrieval module is further configured to perform standard evaluation on a plurality of candidate results in the first correlation subgraph and the second correlation subgraph, sort the plurality of candidate results according to an evaluation result to obtain a retrieval result, and the retrieval result includes candidate domain knowledge corresponding to the multi-modal data and candidate tool information required for processing the multi-modal data. The data processing module is configured to input the retrieval result and the multi-modal data features into a large model to perform information analysis and obtain prompt information and a target tool, and send the prompt information and the target tool to a terminal to enable the terminal to call the target tool to process modal data corresponding to the prompt information to obtain a data processing result.

9. An electronic device, comprising: The electronic device includes a processor configured to implement the steps of the multi-modal data processing method based on a large model according to any one of claims 1-7 when executing a computer program stored in a memory.

10. A computer-readable storage medium, characterized in that, The storage has a computer program, and the computer program is executed by the processor to implement the steps of the multi-modal data processing method based on a large model according to any one of claims 1-7.

Images