Large model-based multi-modal data analysis agent and method

Abstract

The application discloses a large model-based multi-modal data analysis agent and method, which comprises the following steps: obtaining a data analysis prompt; the data analysis prompt comprises data of at least one mode; searching based on the data analysis prompt to obtain a search result; the search result comprises data of at least one mode; based on the mode of the data analysis prompt and the mode of the search result, a target vertical domain data analysis large model is determined from a plurality of candidate vertical domain data analysis large models; when the target data analysis algorithm is determined to be called based on the data analysis prompt by the target vertical domain data analysis large model, data analysis is performed on the data to be analyzed based on the target vertical domain data analysis large model and the target data analysis algorithm to obtain a target data analysis result; the data to be analyzed comprises the data analysis prompt and the search result. The application is used to improve the accuracy of data analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular, to a multi-modal data analysis agent based on a large model and a method. BACKGROUND

[0002] With the acceleration of digital transformation, massive multi-modal data, including images, videos, texts, and time-series sensor data, have been accumulated in the fields of urban emergency safety, industrial production, and energy management. In the face of a large amount of multi-modal data, it is necessary to mine data value and solve the problem of data silos. Therefore, how to efficiently and accurately analyze multi-modal data during data analysis is crucial.

[0003] The existing data analysis method mainly deploys multiple tools independently, relies on manual annotation to train deep learning models, and presets rules to process images or time-series data to achieve analysis of multiple modal data. This method of analyzing data through manual annotation and preset rules results in a huge development workload, and because the processing flow is long, the analysis effect is easily restricted by multiple links in the process, resulting in poor accuracy of the final application effect. SUMMARY

[0004] The present application provides a multi-modal data analysis agent based on a large model and a method to improve the accuracy of data analysis.

[0005] In a first aspect, the present application provides a multi-modal data analysis agent based on a large model, which is used for:

[0006] obtaining a data analysis prompt; the data analysis prompt includes data of at least one modality;

[0007] based on the data analysis prompt, retrieving to obtain a retrieval result; the retrieval result includes data of at least one modality;

[0008] based on the modality of the data analysis prompt and the modality of the retrieval result, determining a target vertical domain data analysis large model from a plurality of candidate vertical domain data analysis large models;

[0009] when determining to call a target data analysis algorithm based on the data analysis prompt through the target vertical domain data analysis large model, performing data analysis on to-be-analyzed data based on the target vertical domain data analysis large model and the target data analysis algorithm to obtain a target data analysis result; the to-be-analyzed data includes the data analysis prompt and the retrieval result.

[0010] In a possible implementation, the data analysis on the to-be-analyzed data based on the target vertical domain data analysis large model and the target data analysis algorithm to obtain a target data analysis result comprises:

[0011] The to-be-analyzed data is format-converted by the target vertical domain data analysis large model to obtain the to-be-analyzed data in a target format; the target format is a data format required by the target data analysis algorithm;

[0012] The to-be-analyzed data in the target format is analyzed based on the target data analysis algorithm to obtain an initial data analysis result;

[0013] The target data analysis result is obtained based on the initial data analysis result by the target vertical domain data analysis large model.

[0014] In a possible implementation, before the searching based on the data analysis prompt to obtain a search result, the multi-modal data analysis intelligent agent further comprises:

[0015] A target database is determined from a plurality of candidate databases based on the data analysis prompt by an intent recognition model;

[0016] The searching based on the data analysis prompt to obtain a search result comprises:

[0017] The search result is obtained by searching the target database based on the data analysis prompt.

[0018] In a possible implementation, the plurality of candidate databases at least comprises a relational database, and the searching the target database based on the data analysis prompt to obtain the search result comprises:

[0019] In response to the target database being the relational database, a database query statement is generated based on the data analysis prompt by a code generation model;

[0020] The relational database is queried based on the database query statement to obtain the search result.

[0021] In a possible implementation, the plurality of candidate vertical domain data analysis large models comprises a vertical domain multi-modal inference model and a vertical domain large language model, and the target vertical domain data analysis large model is determined from the plurality of candidate vertical domain data analysis large models based on the modality of the data analysis prompt and the modality of the search result, comprising:

[0022] In response to the modality of the data analysis prompt being a text modality and the modality of the search result being a text modality, determining that the vertical domain large language model is the target vertical domain data analysis large model;

[0023] Alternatively,

[0024] In response to the modality of the data analysis prompt including a non-text modality and / or the modality of the search result including a non-text modality, determining that the vertical domain multi-modal inference model is the target vertical domain data analysis large model.

[0025] In a possible implementation, the multi-modal data analysis intelligent agent is further configured to:

[0026] obtain a vertical domain multi-modal sample data set and a tool invocation data set, the vertical domain multi-modal sample data set including a vertical domain multi-modal sample prompt and a vertical domain sample response;

[0027] training a preset model based on the vertical domain multi-modal sample data set to obtain an initial vertical domain multi-modal inference model, the preset model including a hidden layer, a text output module, and a multi-modal feature vector output module, the hidden layer being configured to perform semantic feature extraction based on the multi-modal sample prompt to obtain semantic features for representing the multi-modal sample prompt, the text output module being configured to output predicted text for describing the multi-modal sample prompt based on the semantic features, and the multi-modal feature vector output module being configured to output predicted multi-modal features for representing the multi-modal sample prompt based on the semantic features, the predicted text, the predicted multi-modal features, and the vertical domain sample response being used to train the preset model;

[0028] freezing trainable parameters of the multi-modal feature vector output module, and performing reinforcement learning training on parts of the initial vertical domain multi-modal inference model other than the multi-modal feature vector output module based on the tool invocation data set and a vertical domain multi-modal sample preference data set to obtain the vertical domain multi-modal inference model.

[0029] In a possible implementation, before the searching based on the data analysis prompt to obtain a search result, the multi-modal data analysis intelligent agent is further configured to:

[0030] determining whether to perform a search operation based on the data analysis prompt;

[0031] the searching based on the data analysis prompt to obtain a search result includes:

[0032] when it is determined to perform a search operation, searching based on the data analysis prompt to obtain a search result;

[0033] Alternatively,

[0034] In a case where it is determined not to perform the retrieval operation, a target vertical domain data analysis large model is determined from the plurality of candidate vertical domain data analysis large models based on a modality of the data analysis prompt.

[0035] In a second aspect, the present application provides a multi-modal data analysis method based on a large model, the method comprising:

[0036] obtaining a data analysis prompt; the data analysis prompt comprising data of at least one modality;

[0037] performing retrieval based on the data analysis prompt to obtain a retrieval result; the retrieval result comprising data of at least one modality;

[0038] determining a target vertical domain data analysis large model from a plurality of candidate vertical domain data analysis large models based on a modality of the data analysis prompt and a modality of the retrieval result;

[0039] performing data analysis on to-be-analyzed data based on the target vertical domain data analysis large model and a target data analysis algorithm to obtain a target data analysis result in a case where it is determined to call the target data analysis algorithm based on the data analysis prompt by the target vertical domain data analysis large model; the to-be-analyzed data comprising the data analysis prompt and the retrieval result.

[0040] In a third aspect, the present application provides a multi-modal data analysis device, comprising:

[0041] an obtaining module configured to obtain a data analysis prompt; the data analysis prompt comprising data of at least one modality;

[0042] a retrieval module configured to perform retrieval based on the data analysis prompt to obtain a retrieval result; the retrieval result comprising data of at least one modality;

[0043] a determining module configured to determine a target vertical domain data analysis large model from a plurality of candidate vertical domain data analysis large models based on a modality of the data analysis prompt and a modality of the retrieval result;

[0044] an analysis module configured to perform data analysis on to-be-analyzed data based on the target vertical domain data analysis large model and a target data analysis algorithm to obtain a target data analysis result in a case where it is determined to call the target data analysis algorithm based on the data analysis prompt by the target vertical domain data analysis large model; the to-be-analyzed data comprising the data analysis prompt and the retrieval result.

[0045] In a fourth aspect, an electronic device is provided, comprising: a processor, and a memory connected to the processor in communication;

[0046] The memory stores computer-executable instructions.

[0047] The memory stores computer-executable instructions.

[0048] The processor executes the computer-executable instructions stored in the memory to implement the method of any one of the second aspect.

[0049] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, including, the computer-readable storage medium stores computer-executable instructions, the computer-executable instructions are executed by a processor to implement the method of any one of the second aspect.

[0050] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program, the computer program is executed by a processor to implement the method of any one of the second aspect.

[0051] The present application provides a multi-modal data analysis agent based on a large model. By obtaining a data analysis prompt, at least one modality of data can be obtained, which lays a foundation for subsequent data retrieval and data analysis. Based on the data analysis prompt, retrieval can be performed to obtain a retrieval result, improving the reliability of the retrieval result. Based on the modality of the data analysis prompt and the modality of the retrieval result, a target vertical domain data analysis large model can be determined from a plurality of candidate vertical domain data analysis large models, realizing multi-modal data analysis. When the target vertical domain data analysis large model is determined to call a target data analysis algorithm based on the data analysis prompt, data analysis can be performed on the data to be analyzed based on the target vertical domain data analysis large model and the target data analysis algorithm, and a target data analysis result can be obtained, realizing the professionalism of data analysis, thereby improving the data analysis capability and the accuracy of data analysis. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0053] Figure 1 A flowchart of a multi-modal data analysis method provided by an embodiment of the present application;

[0054] Figure 2 A flowchart of another multi-modal data analysis method provided by the present application;

[0055] Figure 3 A flowchart of an automatic processing of massive vertical domain data method provided by an embodiment of the present application;

[0056] Figure 4 A model architecture schematic diagram of a multi-modal large model provided for an embodiment of the present application;

[0057] Figure 5 A process schematic diagram of multi-modal large model training provided for an embodiment of the present application;

[0058] Figure 6 A process schematic diagram of vertical large language model training provided for an embodiment of the present application;

[0059] Figure 7 A process schematic diagram of a method for constructing a candidate database provided for an embodiment of the present application;

[0060] Figure 8 A process schematic diagram of a method for constructing a knowledge graph database provided for an embodiment of the present application;

[0061] Figure 9 A process schematic diagram of a multi-modal retrieval method provided for an embodiment of the present application;

[0062] Figure 10 A structure schematic diagram of a multi-modal data analysis device provided for the present application;

[0063] Figure 11 A hardware structure schematic diagram of an electronic device provided for the present application.

[0064] Through the above-mentioned drawings, the specific embodiments of the present application have been shown, and will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0065] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals refer to like elements, unless otherwise indicated. The following exemplary embodiments described in the following detailed description are not meant to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0066] First, the terms involved in the present application are explained: prompt: an input instruction used to guide the large model to perform a specific task.

[0067] In the existing technical system, data analysis is mainly performed by humans for data cleaning, feature engineering, model construction, and result interpretation. Among them, when analyzing structured data, artificial feature engineering is relied on to regularize the data into a table for relational operation and analysis. When analyzing unstructured data, multiple tool chains need to be independently deployed, relying on manual annotation and preset rules to process images.

[0068] However, in the face of massive multi-modal data such as images, videos, texts, and time-series sensor data, the existing data analysis method is difficult to integrate data and mine data value, can only analyze single-modal data, resulting in the problems of data silos and insufficient data analysis capabilities.

[0069] Considering the reasons for the above problems of the existing data analysis method, the present application proposes a method for analyzing single-modal data without relying on human intervention, but for analyzing multi-modal data to reduce the impact of data silos on analysis results and improve data analysis capabilities.

[0070] The execution subject of the multi-modal data analysis method may be, for example, a multi-modal data analysis agent or a multi-modal data analysis method agent. Taking the execution subject of the multi-modal data analysis method as a multi-modal data analysis method agent for example, the multi-modal data analysis method agent may be deployed in a server cluster or a cloud environment. The deployment environment of the multi-modal data analysis method agent is not limited by the present application.

[0071] The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0072] The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0073] Figure 1 A flowchart of a multi-modal data analysis method provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the multi-modal data analysis method includes: Figure 1

[0074] S101, obtaining a data analysis prompt; the data analysis prompt includes data of at least one modality.

[0075] For example, the data of at least one modality may be any one of text, image, and video data, or a combination of multiple types of data.

[0076] ​Optionally, the multi-modal data analysis method agent may, for example, obtain a data analysis prompt through user input of text, pictures, or files, etc. It should be understood that the present application does not limit the implementation method of obtaining the data analysis prompt, which can be any existing implementation method of obtaining the data analysis prompt, and the present application will not be repeated here.

[0077] S102, retrieving based on the data analysis prompt to obtain a retrieval result; the retrieval result includes data of at least one modality.

[0078] The data of at least one modality included in the retrieval result may, for example, be any one of data including text, image, and video, or a combination of multiple data.

[0079] Optionally, the multi-modal data analysis method agent may, for example, input the data analysis prompt into a retrieval model to obtain the retrieval result. For example, the retrieval model may, for example, be any existing model with retrieval capability such as a large language model, and the present application will not be repeated here.

[0080] S103, based on the modality of the data analysis prompt and the modality of the retrieval result, determining a target vertical domain data analysis large model from a plurality of candidate vertical domain data analysis large models.

[0081] The plurality of candidate vertical domain data analysis large models may, for example, be a vertical domain multi-modal reasoning model and a vertical domain large language model.

[0082] Optionally, the multi-modal data analysis method agent may, for example, determine the target vertical domain data analysis large model as a relational database when the modality of the data analysis prompt and the modality of the retrieval result are both pure text modalities.

[0083] The multi-modal data analysis method agent may, for example, determine the target vertical domain data analysis large model as a vertical domain multi-modal reasoning model when the modality of the data analysis prompt and the modality of the retrieval result include non-text modalities (such as image modalities, etc.).

[0084] The multi-modal data analysis method agent may, for example, determine the target vertical domain data analysis large model as a vertical domain multi-modal reasoning model when the modality of the data analysis prompt or the modality of the retrieval result includes a non-text modality (such as an image modality, etc.).

[0085] S104, when determining to call a target data analysis algorithm based on the data analysis prompt through the target vertical domain data analysis large model, performing data analysis on the data to be analyzed based on the target vertical domain data analysis large model and the target data analysis algorithm to obtain a target data analysis result; the data to be analyzed includes the data analysis prompt and the retrieval result.

[0086] The multi-modal data analysis method agent may, for example, first determine whether to call the target data analysis algorithm based on the data analysis prompt by the target vertical domain data analysis large model. For example, the multi-modal data analysis method agent may, based on the data analysis prompt, construct a prompt word, input the prompt word into the target vertical domain data analysis large model, and obtain a judgment result of whether to call the target data analysis algorithm. The prompt word is used to indicate whether to call the target data analysis algorithm based on the data analysis prompt.

[0087] In some embodiments, the multi-modal data analysis method agent may, for example, first convert the format of the to-be-analyzed data to obtain the to-be-analyzed data in a target format by the target vertical domain data analysis large model. Then, based on the target data analysis algorithm, the to-be-analyzed data in the target format is analyzed to obtain an initial data analysis result. Then, based on the initial data analysis result, the target data analysis result is obtained by the target vertical domain data analysis large model.

[0088] For example, the to-be-analyzed data may be the data analysis prompt, and the retrieval result.

[0089] For example, the target format may be a data format required by the target data analysis algorithm. For example, the target format may be a format that meets the input specification required by the target data analysis algorithm, including but not limited to data structure, dimension, data type, and encoding method.

[0090] For example, the target data analysis algorithm may be a computer vision algorithm, a machine learning algorithm, or a professional mechanism algorithm. It should be understood that the specific implementation type of the target data analysis algorithm is not limited in the present application, and it can be any existing algorithm with data analysis function, which will not be described herein.

[0091] Optionally, the multi-modal data analysis method agent may, for example, perform information extraction and format conversion on the to-be-analyzed data by the target vertical domain data analysis large model to obtain the to-be-analyzed data in a target format. The multi-modal data analysis method agent may also call the target data analysis algorithm based on the to-be-analyzed data in the target format. For example, based on the to-be-analyzed data in the target format, the target data analysis algorithm may analyze data rules, perform image recognition, or perform simulation deduction to obtain an initial data analysis result. Then, the initial data analysis result is input into the target vertical domain data analysis large model, and the target vertical domain data analysis large model is analyzed to obtain the target data analysis result.

[0092] By the above method, the target format of the to-be-analyzed data can be obtained by the target vertical domain data analysis large model through format conversion on the to-be-analyzed data, which lays a foundation for subsequent data analysis based on the to-be-analyzed data. Based on the target data analysis algorithm, data analysis can be performed on the to-be-analyzed data in the target format, and an initial data analysis result can be obtained. Based on the initial data analysis result, the target data analysis result can be obtained through the target vertical domain data analysis large model. The accuracy and reliability of the data analysis result are improved.

[0093] In this embodiment, at least one modality of data can be obtained by obtaining a data analysis prompt, which lays a foundation for subsequent data retrieval and data analysis. Based on the data analysis prompt, a retrieval result can be obtained, improving the reliability of the retrieval result. Based on the modality of the data analysis prompt and the modality of the retrieval result, a target vertical domain data analysis large model can be determined from a plurality of candidate vertical domain data analysis large models, realizing multi-modal data analysis. When the target vertical domain data analysis large model is determined based on the data analysis prompt by the target vertical domain data analysis large model, the target data analysis algorithm is called, and the to-be-analyzed data is analyzed based on the target vertical domain data analysis large model and the target data analysis algorithm, a target data analysis result is obtained, realizing the professionalism of data analysis, thereby improving the data analysis ability and the accuracy of data analysis.

[0094] The following describes how the multi-modal data analysis method agent retrieves based on the data analysis prompt to obtain a retrieval result:

[0095] In some embodiments, the multi-modal data analysis method agent can determine a target database from a plurality of candidate databases based on the data analysis prompt through an intent recognition model, and then retrieve from the target database based on the data analysis prompt to obtain the retrieval result.

[0096] For example, the intent recognition model can be a deep learning model or a machine learning model obtained by training the model in advance based on a user intent sample data set. The execution subject of the model training method, for example, can be any electronic device with processing function. The implementation of the intent recognition model based on the data analysis prompt to identify user intent can refer to any existing deep learning model training method, which will not be described herein.

[0097] Optionally, the intent recognition model may, for example, also analyze the data analysis prompt, rewrite the data analysis prompt based on the analysis, and obtain a second data analysis prompt. For example, the multi-modal data analysis method agent may, for example, construct a prompt word based on the data analysis prompt, input the prompt word into the intent recognition model, and obtain the user intent or the second data analysis prompt. The prompt word may be used to indicate that the user intent is identified based on the data analysis prompt or the data analysis prompt is rewritten.

[0098] For example, the plurality of candidate databases may be a relational database, a multi-modal database, or a domain knowledge graph database.

[0099] Optionally, the multi-modal database may be constructed based on a vertical system. For example, the multi-modal data analysis method agent may, for example, determine a key image sequence in an abnormal event video segment based on the abnormal event video segment of the vertical system by frame extraction at a certain rate. Then, the key image sequence is input into the multi-modal large model to obtain a text description and a representation vector of the key image sequence through the multi-modal large model. Then, the text description and the representation vector are stored in association to obtain the multi-modal database.

[0100] Optionally, the multi-modal large model may obtain a text description and a representation vector based on picture, table, and video processing data. For example, the multi-modal large model may use a language model output head to generate a text description of the processing data, and may also use a visual encoder to extract features of the processing data to obtain a representation vector of the processing data.

[0101] For example, the domain knowledge graph knowledge base may be constructed based on multi-modal large model analysis of rich text documents and pure text documents. For example, the multi-modal large model may first process the rich text document through a rich text document analysis module to convert it into a unified format, and then use a layout analysis model to identify and cut out independent elements such as text and charts. Then, for different types of elements, the multi-modal large model is guided to analyze the elements using system prompt words to obtain the analysis results of all elements in the page. Then, the analysis results of all elements in the page are reorganized into a Markdown (a type of markup language) format pure text document in reading order, and then processed by a text encoder to generate semantic vectors. Then, the document metadata, text slices, and their corresponding vectors are associated and integrated to construct the domain knowledge graph knowledge base.

[0102] Optionally, the multi-modal data analysis method agent may, for example, construct a prompt based on the above-mentioned to-be-processed data, the rich text document, and the plain text document, and input the prompt into the multi-modal large model to obtain a text description and a representation vector, or a parsing result of all elements in the page. The prompt may be used to instruct the construction of a multi-modal database or a domain knowledge graph database based on the to-be-processed data, the rich text document, and the plain text document.

[0103] In this embodiment, the multi-modal data analysis method agent may, through an intent recognition model, determine a user intent based on the data analysis prompt or rewrite the data analysis prompt to obtain a second data analysis prompt. According to the user intent or the second data analysis, a target database to be searched can be determined. Then, the multi-modal data analysis method agent searches the target database based on the data analysis prompt or the second data analysis prompt to obtain the search result, thereby improving the accuracy of the search result.

[0104] As a possible implementation manner, the multi-modal data analysis method agent may, in response to the target database being the relational database, generate a database query statement based on the data analysis prompt through a code generation model. Then, the relational database is queried based on the database query statement to obtain the search result.

[0105] For example, the code generation model may be a model that converts a natural language description of a data analysis prompt into an executable structured query language (SQL) that conforms to the query syntax specification of the relational database.

[0106] For example, the multi-modal data analysis method agent may, based on the data analysis prompt, construct a prompt and input the prompt and the data analysis prompt into the code generation model to obtain SQL query code. The prompt may be used to instruct the construction of SQL query code based on the table name, purpose, all column names, and data meaning of each column contained in the relational database.

[0107] Optionally, the multi-modal data analysis method agent may, for example, determine the relational database to be the target database through an intent recognition model. Then, the multi-modal data analysis method agent may input the data analysis prompt into the above-mentioned code generation model to generate a database query statement that conforms to the query syntax specification of the relational database through the code generation model. The multi-modal data analysis method agent may call a SQL database query tool to query the relational database based on the database query statement to obtain the search result.

[0108] Through the method, based on the target database being a relational database, based on the data analysis prompt, a database query statement can be obtained through a code generation model. Based on the database query statement, the relational database is queried to obtain the search result, improving the efficiency of data retrieval during data analysis.

[0109] The following describes how the multi-modal data analysis method agent determines the target vertical domain data analysis large model from multiple candidate vertical domain data analysis large models based on the modality of the data analysis prompt and the modality of the search result:

[0110] In some embodiments, the multi-modal data analysis method agent can also determine the vertical domain large language model as the target vertical domain data analysis large model in response to the modality of the data analysis prompt being a text modality and the modality of the search result being a text modality, taking multiple candidate vertical domain data analysis large models including a vertical domain multi-modal reasoning model and a vertical domain large language model as an example.

[0111] Alternatively, the multi-modal data analysis method agent can determine the vertical domain multi-modal reasoning model as the target vertical domain data analysis large model in response to the modality of the data analysis prompt including a non-text modality or the modality of the search result including a non-text modality.

[0112] Alternatively, the multi-modal data analysis method agent can determine the vertical domain multi-modal reasoning model as the target vertical domain data analysis large model in response to the modality of the data analysis prompt including a non-text modality and the modality of the search result including a non-text modality.

[0113] In this embodiment, the target vertical domain data analysis large model can be determined from multiple candidate vertical domain data analysis large models based on the modality of the data analysis prompt and the modality of the search result, improving the data analysis capability.

[0114] The following describes the training method of the vertical domain multi-modal reasoning model in detail:

[0115] It should be understood that the execution subject of the training method can be any electronic device with processing capability (such as a terminal or a server, etc.). Alternatively, the execution subject of the training method can also be the aforementioned multi-modal data analysis method agent. The execution subject of the training method is not limited in the present application, and it should be understood that the following training method is only exemplarily described taking the execution subject of the training method as the multi-modal data analysis method agent.

[0116] As a possible implementation manner, the multi-modal data analysis method agent may, for example, first acquire a vertical multi-modal sample dataset composed of vertical multi-modal sample prompts and vertical sample responses, and a tool invocation dataset. Then, the multi-modal data analysis method agent may train a preset model based on the vertical multi-modal sample dataset to obtain an initial model.

[0117] The preset model has a hidden layer for extracting semantic features of an input prompt, a text output module for generating a predicted text based on the features, and a multi-modal feature vector output module for generating a predicted multi-modal feature. Then, the multi-modal data analysis method agent may train the preset model based on the predicted text, the predicted multi-modal feature, and the vertical sample response. Then, the multi-modal data analysis method agent may freeze the trainable parameters of the multi-modal feature vector output module, and perform reinforcement learning training on parts of the initial vertical multi-modal inference model other than the multi-modal feature vector output module based on the tool invocation dataset and a vertical multi-modal sample preference dataset to obtain the vertical multi-modal inference model.

[0118] For example, the vertical multi-modal sample dataset may be composed of multiple groups of vertical multi-modal sample prompts and corresponding vertical sample responses. For example, the vertical multi-modal sample prompt may be input data containing at least one modality such as text and picture. The vertical sample response may be an expected output result of the vertical multi-modal inference model.

[0119] For example, the tool invocation dataset may be composed of multiple data groups containing the data analysis prompt and corresponding structured tool invocation instructions. For example, the tool invocation dataset may be a set of pairs of multiple user questions and correct structured tool invocation instructions.

[0120] For example, the preset model may be any deep learning model or machine learning model having a hidden layer, a text output module, and a multi-modal feature vector output module.

[0121] For example, the hidden layer in the preset model may extract semantic features based on the vertical multi-modal sample prompt to obtain semantic features for representing the vertical multi-modal sample prompt.

[0122] Optionally, the multi-modal data analysis method agent may, for example, construct a prompt word based on the vertical multi-modal sample prompt and input the prompt word into the hidden layer of the vertical multi-modal inference model to obtain semantic features for representing the vertical multi-modal sample prompt. The prompt word may be used to indicate semantic feature extraction based on the vertical multi-modal sample prompt.

[0123] Exemplarily, the preset model can be configured to output a predicted text for describing the vertical multi-modal sample prompt based on the semantic feature. For example, the text output module of the preset model can be a language model output head configured to receive the semantic feature of the hidden layer and generate a corresponding text sequence based on the semantic feature.

[0124] Exemplarily, the preset model can be configured to output a predicted multi-modal feature for representing the vertical multi-modal sample prompt based on the semantic feature. For example, the multi-modal feature vector output module of the preset model can be an embedding vector head configured to analyze the semantic feature and generate a predicted multi-modal feature for representing the vertical multi-modal sample prompt.

[0125] Exemplarily, the vertical multi-modal sample preference dataset can include a plurality of artificially annotated preference records. For example, the vertical multi-modal sample preference dataset can include a plurality of vertical multi-modal sample prompts, a plurality of candidate responses generated based on the vertical multi-modal sample prompts, and a preference ranking determined by a domain expert according to a standard.

[0126] Optionally, the multi-modal data analysis method agent can obtain the vertical multi-modal sample dataset, and the tool invocation dataset can be obtained in a user input manner. For example, any existing method for obtaining a vertical multi-modal sample dataset and invoking a dataset can be used, and details are not described herein.

[0127] Optionally, the multi-modal data analysis method agent can obtain the semantic feature, the predicted text, and the predicted multi-modal feature of the vertical multi-modal sample prompt based on the obtained vertical multi-modal sample dataset by using the hidden layer, the text output module, and the multi-modal feature vector output module of the preset model. The multi-modal data analysis method agent can further train the preset model through a plurality of rounds of dialogue based on the semantic feature, the predicted text, and the predicted multi-modal feature.

[0128] The multi-modal data analysis method agent may also freeze the trainable parameters of the multi-modal feature vector output module, based on the tool invocation dataset and the vertical multi-modal sample preference dataset, and perform reinforcement learning training on parts other than the multi-modal feature vector output module to obtain the vertical multi-modal reasoning model. The implementation of performing reinforcement learning training on parts other than the multi-modal feature vector output module may, for example, adjust the data proportions of the tool invocation dataset and the vertical multi-modal sample preference dataset using cosine dynamics. For example, the data proportions of the tool invocation dataset and the vertical multi-modal sample preference dataset are 1:4 in the initial stage, and the proportion is adjusted to 1:1 when training to half the batch and remains unchanged until the end of training.

[0129] It should be understood that the multi-modal data analysis method agent may also refer to any existing deep learning model training process to obtain the implementation of the vertical multi-modal reasoning model, which will not be described here.

[0130] Through the above method, the vertical multi-modal sample dataset and the tool invocation dataset can be obtained, and the vertical multi-modal sample prompt, the vertical sample response, and the tool invocation dataset can be obtained. Based on the multi-modal sample dataset, the predicted text, the predicted multi-modal feature, and the vertical sample can be obtained through the preset model. Based on the predicted text, the predicted multi-modal feature, and the vertical sample, the preset model can be trained. By freezing the trainable parameters of the multi-modal feature vector output module, based on the tool invocation dataset and the vertical multi-modal sample preference dataset, reinforcement learning training is performed on parts other than the multi-modal feature vector output module to obtain the vertical multi-modal reasoning model. The reliability of the vertical multi-modal reasoning model is enhanced, thereby improving the data analysis capability.

[0131] For example, the vertical large language model may be a deep learning model or a machine learning model based on vertical pure text with multi-round conversation function. For example, the multi-modal data analysis method agent may first fine-tune the vertical large language model through a vertical supervised fine-tuning (SFT) dataset.

[0132] For example, the SFT dataset may be a plurality of instruction sets and a plurality of matched sets of answers conforming to the vertical specification for the instructions.

[0133] Optionally, the multi-modal data analysis method agent may fine-tune the vertical large language model through the SFT dataset, the vertical pure text preference dataset, and the tool invocation dataset to obtain the vertical large language model.

[0134] Optionally, the multi-modal data analysis method agent may, for example, adopt cosine dynamic adjustment to adjust the tool call data set and the data proportion of the vertical pure text preference data set. For example, the data proportion of the tool call data set and the vertical pure text preference data set is 1:4 in the initial stage, and the proportion is adjusted to 1:1 when training to half of the batch, and remains to the end of training.

[0135] It should be understood that the multi-modal data analysis method agent may, for example, also refer to any existing deep learning model in the training process to obtain the implementation mode of the vertical large language model, which will not be described herein. In some embodiments, the multi-modal data analysis method agent may, for example, also determine whether to perform a retrieval operation based on the data analysis prompt before retrieving based on the data analysis prompt to obtain a retrieval result, and then perform retrieval based on the data analysis prompt to obtain a retrieval result.

[0136] Optionally, the multi-modal data analysis method agent may, for example, perform retrieval based on the data analysis prompt to obtain a retrieval result when it is determined to perform a retrieval operation. Alternatively, the multi-modal data analysis method agent may, for example, determine the target vertical data analysis large model from the plurality of candidate vertical data analysis large models based on the modality of the data analysis prompt when it is determined not to perform a retrieval operation.

[0137] Optionally, the multi-modal data analysis method agent may, for example, determine whether a retrieval operation is needed based on the data analysis prompt through an intent recognition model. When the intent recognition model determines that a retrieval operation is not needed, the multi-modal data analysis method agent may, for example, determine the target vertical data analysis large model from the plurality of candidate vertical data analysis large models based on the modality of the data analysis prompt. For example, when the modality of the data analysis prompt includes a non-text modality, the target vertical data analysis large model is determined to be the vertical multi-modal reasoning model. For example, when the modality of the data analysis prompt is a text modality, the target vertical data analysis large model is determined to be the vertical large language model.

[0138] Optionally, when the intent recognition model determines that a retrieval operation is needed, the multi-modal data analysis method agent may, for example, perform retrieval based on the data analysis prompt to obtain a retrieval result.

[0139] Through the above method, it can be determined whether a retrieval operation is needed based on the data analysis prompt. When it is determined that a retrieval operation is needed, retrieval based on the data analysis prompt can obtain a retrieval result. If it is determined that a retrieval operation is not needed, the target vertical data analysis large model can be determined based on the modality of the data analysis prompt, thereby improving the efficiency of data analysis.

[0140] Optionally, the multi-modal data analysis method agent may, for example, determine the target vertical domain data analysis large model from the vertical domain multi-modal reasoning model and the vertical domain large language model according to the modality of the data analysis prompt and the modality of the search result.

[0141] Optionally, the multi-modal data analysis method agent may, for example, determine the target vertical domain data analysis large model to be the vertical domain large language model based on the modality of the data analysis prompt being a text modality and the modality of the search result being a text modality.

[0142] The multi-modal data analysis method agent may, for example, also determine the target vertical domain data analysis large model to be the vertical domain multi-modal reasoning model based on the modality of the data analysis prompt including a non-text modality and the modality of the search result including a non-text modality.

[0143] The multi-modal data analysis method agent may, for example, also determine the target vertical domain data analysis large model to be the vertical domain multi-modal reasoning model based on the modality of the data analysis prompt including a non-text modality or the modality of the search result including a non-text modality.

[0144] Figure 2 Another flowchart of a multi-modal data analysis method is provided for the present application. As shown in Figure 2 The multi-modal data analysis method may, for example, be as follows: a multi-modal data analysis method agent performs data analysis through selectable invocation of professional algorithms based on multi-modal data such as images, videos, and text, thereby improving data analysis capabilities.

[0145] An exemplary Figure 3 A flowchart of an automatic processing of massive vertical domain data method is provided for an embodiment of the present application. As shown in Figure 3 The method includes:

[0146] First, existing multi-modal large models are used to process all pictures, tables, and videos to be processed to obtain corresponding detailed text descriptions, i.e., image-text pairs.

[0147] Second, existing multi-modal large models are used to score the image-text pairs generated in the first step, and image-text pairs with text description compliance higher than a threshold are retained. For example, image-text pairs with text description compliance higher than 8 points are retained.

[0148] Third, existing large language models are used to analyze the text descriptions in all image-text pairs, and image-text pairs that do not conform to the target field are filtered out.

[0149] The fourth step is to use the visual encoder in the existing multimodal large model as the feature extractor for the image data, extract the representation vectors only for the image data in all image-text pairs, and store the representation vectors of these images in the vector database.

[0150] The fifth step is to filter the vectors in the vector database based on similarity. For example, only one image is kept for the representation vector with a similarity greater than 0.95, and 50% of the images with a similarity between 0.85 and 0.95 are randomly removed, resulting in high-quality, non-repeating image-text pairs for the target domain.

[0151] Figure 4 This is a schematic diagram of a multimodal large model provided in an embodiment of this application. For example... Figure 4 As shown, this multimodal large model may include image preprocessing, a vision encoder (Vision Transformer), pixel unshuffle, a projector, prompts, a text tokenizer, a large language model (LLM), a language model head (LM Head), an embedding projector, output text, and multimodal feature vector outputs. For example, the multimodal large model can have both an LM Head and an embedding head set in the last layer, allowing it to simultaneously output the image's embedding vectors.

[0152] Figure 5 This is a schematic diagram illustrating a multimodal large model training process provided in an embodiment of this application. Figure 5 As shown, the training process of this multimodal large model is as follows:

[0153] (1) Training process of multimodal embedding model:

[0154] Step 1: Freeze the Embedding task and train the LM Head only on the vertical multimodal dataset.

[0155] Step 2: Freeze the text generation task and train the Embedding task only on the vertical multimodal dataset.

[0156] Step three, for the same batch of training data, randomly start one of the generation or Embedding tasks with a 50% chance, and perform both tasks at the same time with a 50% chance. The training data can include a vertical multi-modal dataset, and an Optical Character Recognition (OCR) dataset. The vertical multi-modal dataset can account for 70%, and the OCR dataset can account for 30%.

[0157] Step four, all batches of training data are performed simultaneously for both tasks, and only one batch (epoch) is executed in this stage. The training data is consistent with the training data in step three.

[0158] For example, the vertical multi-modal dataset used in the training process of the multi-modal embedding model is in the format of a single round of dialogue.

[0159] (2) Multi-modal inference question and answer model training process:

[0160] Step one, freeze the Embedding task, use the vertical multi-modal dataset in the format of a single round of dialogue, and only train the LM Head on the vertical multi-modal dataset. Through this step, the multi-modal inference question and answer model can master the professional terms and expression methods in the vertical field, and establish a field knowledge base.

[0161] Step two, freeze the Embedding task, use the vertical multi-modal dataset in the format of a multi-round dialogue, and only train the LM Head on the vertical multi-modal dataset. Through this step, the multi-modal inference question and answer model can have multi-round dialogue capability and improve the model inference ability.

[0162] Step three, freeze the Embedding task, train the LM Head based on the vertical multi-modal dataset and the vertical multi-modal preference dataset through Reinforcement Learning from Human Feedback (RLHF). Through this step, the preference alignment of the multi-modal inference question and answer model output can be optimized, the answer quality and user experience can be improved, and the multi-modal inference question and answer model can better meet the expectations of field experts.

[0163] Step four, based on the vertical multi-modal dataset, the vertical multi-modal preference dataset, and the function calling data, train the LM Head through Reinforcement Learning from Human Feedback (RLHF) based on the function calling data.

[0164] For example, the ratio of function calling data to vertical multimodal data can be dynamically varied using cosine, starting at 1:4 at the beginning of training, increasing to 1:1 when the training batch reaches halfway through, and then remaining constant thereafter.

[0165] Figure 6 This is a schematic diagram illustrating the training process of a large-scale language model in a vertical domain, as provided in an embodiment of this application. Figure 6 As shown, the training process of this vertical domain large language model is as follows:

[0166] Step one involves fine-tuning instructions on the vertical domain's plain text SFT dataset. This step enables the large language model in this vertical domain to master the specialized terminology and expressions of the vertical field, thus establishing a foundation of domain knowledge.

[0167] Step two involves fine-tuning the first segment of the model based on a vertical domain plain text preference dataset, using reinforcement learning based on human feedback. This step aligns the model's output with human preferences.

[0168] Step three, building upon step two, adds a function calling dataset and performs fine-tuning using reinforcement learning based on human feedback in the second stage. This step enables the large-scale language model for this vertical domain to acquire function calling capabilities.

[0169] Optionally, during the training process of step three of this vertical domain large language model, the ratio of function calling data to vertical domain plain text data in each batch of training data can be dynamically adjusted using the cosine annealing algorithm. The ratio is 1:4 at the beginning of training, and increases to 1:1 when the training batch reaches halfway, and remains unchanged thereafter.

[0170] Figure 7 This is a flowchart illustrating a method for constructing a candidate database, as provided in an embodiment of this application. Figure 7 As shown, images and videos can be used to obtain a multimodal database based on a multimodal large-scale model and system prompts. This multimodal database can include multi-vectors (multimodal feature vectors) and text descriptions. Image-text documents and text documents can be used to construct a domain knowledge graph based on a rich text document parsing module and a multimodal large-scale model. This candidate database can include both a multimodal database and a domain knowledge graph.

[0171] Exemplarily, the multi-modal data analysis method agent can also obtain a multi-modal database based on the obtained vertical field image or vertical field video through the multi-modal embedding model. Exemplarily, the multi-modal embedding model may, for example, obtain multiple pictures of an abnormal event to be detected based on a vertical field video of an abnormal event segment with a time length of 5 seconds by frame extraction at 1-3 frames per second. The visual encoder based on the multi-modal embedding model processes the image and the video to obtain detailed description text and semantic representation multi-vectors through the LM Head and the Embedding Head respectively, and the multi-vectors are saved into a vector database to obtain the multi-modal database.

[0172] Figure 8 A flowchart of a method for constructing a knowledge graph database provided by an embodiment of the present application is shown in FIG. 1. Figure 8 As shown in FIG. 1, the method for constructing a knowledge graph database can include:

[0173] The domain knowledge document can include a Portable Document Format (PDF), an office (a document format), and a pure text document.

[0174] Exemplarily, the multi-modal embedding model may, for example, convert all the above domain knowledge documents into a PDF format and then into an image in a Joint Photographic Experts Group (JPG) format. The layout element extraction model can perform layout element recognition on the above image, extract layout element numbers, and perform region segmentation based on pictures, tables, texts, and titles.

[0175] Optionally, the multi-modal embedding model can extract element text and knowledge based on system prompts (System prompt) by batch processing of text elements and batch processing of picture and table elements. The multi-modal embedding model can also extract embedding vectors from sliced text, including recombining element recognition results into a Markdown (a markup language) pure text document and extracting vectors based on document slicing.

[0176] The layout element numbers, the document slicing extraction vectors, and the document metadata constitute the domain knowledge graph.

[0177] In an embodiment, the domain knowledge graph can include six entity types and six relationship types, and the entity types are as follows:

[0178] Domain (emergency safety subfield): gas, water, petrochemical, transportation, etc.

[0179] Document: papers, management regulations, etc.

[0180] Document Category: Enterprise production documents, academic literature, etc.

[0181] Page: page1, page2, page3, etc.

[0182] Content Element: Image elements, table elements, heading elements, etc.

[0183] Knowledge Chunk: Document Chunk 1, Document Chunk 2, Document Chunk 3, etc.

[0184] The page entity contains four attributes: original document identifier (ID), number of images, number of tables, and page overview. The element entity contains five attributes: original document ID, original page ID, save path, text information, and representation vector. The knowledge fragment entity corresponds to each document slice, and each document slice corresponds to one or more element entities. The knowledge fragment entity contains three attributes: representation vector, text information, and number of elements.

[0185] The relation types are as follows:

[0186] BELONGS_TO (belongs to): describes which category a document belongs to. For example, "Gas Management Regulations" BELONGS_TO "Management Regulations" indicates that "Gas Management Regulations" belongs to the Management Regulations category.

[0187] IN_DOMAIN (Domain): Describes which sub-domain of emergency safety a document type belongs to. For example, "Management Regulations" IN_DOMAIN "Gas" indicates that the Management Regulations belong to the gas field of emergency safety.

[0188] HAS_PAGE (Containing Page): Describes which pages a document includes. For example, "Report" HAS_PAGE Page 1 indicates that the "Report" is located on page 1.

[0189] LOCATED_IN (located on): Describes which page an element is located on. For example, Table 3 LOCATED_IN Page 5 indicates that Table 3 is located on page 5.

[0190] COMPRISED_OF (Composition): Describes the elements contained in a knowledge fragment. For example, knowledge fragment ACOMPRISED_OF Table 3 + Text Paragraph 5 indicates that the knowledge fragment contains Table 3 and Text Paragraph 5.

[0191] CITES (citation): describes which document the current document refers to, e.g., an academic paper cites another paper, indicating that the academic paper refers to another paper.

[0192] Figure 9 A flowchart of a multi-modal retrieval method provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the multi-modal retrieval method includes the following steps. Figure 9

[0193] For example, the problem to be retrieved first passes through an intent recognition model. The large model has two functions. One function is to determine, according to the user's intention, whether the multi-modal database is to be retrieved next (for example, to search whether the current monitoring system has captured personnel abnormal behavior), or the vertical knowledge graph (for example, to query the safety production regulations of gas), or the relational database (for example, to query which devices are offline today). Or, without retrieval, directly engage in question and answer dialogue with the vertical large language model or the multi-modal large model.

[0194] The other function is to rewrite the user's input problem to improve the subsequent retrieval hit rate and accuracy. Only when it is determined that the user wants to query the multi-modal database, the problem will be rewritten.

[0195] When the multi-modal database or the domain knowledge graph is to be queried, the user's problem passes through the text encoder of the multi-modal embedding model to obtain the corresponding representation vector of each problem. The input picture or video also passes through the visual encoder of the multi-modal embedding model to obtain the corresponding representation vector. Then, the problem representation vector, the picture representation vector, or the video representation vector is respectively retrieved in the multi-modal database or the domain knowledge graph to obtain the corresponding multi-modal data or domain knowledge.

[0196] For example, the cross-modal retrieval service can retrieve in the multi-modal database or the domain knowledge graph based on the above representation vectors. For example, the cross-modal retrieval service includes basic semantic retrieval, cross-modal retrieval, keyword retrieval, and hybrid retrieval, etc.

[0197] For example, if the intent recognition model determines that the multi-modal database needs to be retrieved, the intent recognition model will give the next multi-modal database to be retrieved, and also rewrite the user's input problem. For example, the user inputs "Help me check if there is any personnel abnormal behavior in the park today", and the rewritten output is "Personnel behavior 1; Personnel behavior 2; Behavior 3..."

[0198] ​For example, the multimodal fusion retrieval module retrieves each rewritten question separately from the multimodal database, and inputs all the search results, along with the user's input question, into the vertical multimodal big data model. Then, the vertical multimodal big data model analyzes all dialogue history and retrieved images to answer the user's question.

[0199] Optionally, after analyzing the user's question, the multimodal big data model in the vertical domain may call upon specialized algorithms for analysis. In this case, the multimodal big data model will send the input or retrieved images to the computer vision (CV) algorithm within the specialized algorithm, according to the relevant format. For example, if the user's question is to help them count the number of people and their distribution in an image, the model will wait for the results of the specialized algorithm's analysis, and then, based on the context, answer the user's question.

[0200] Figure 10 This is a schematic diagram of the structure of a multimodal data analysis device provided in this application. Figure 10 As shown, the multimodal data analysis device 1000 may include: an acquisition module 1001, a retrieval module 1002, a determination module 1003, and an analysis module 1004. Among them,

[0201] The acquisition module 1001 is used to acquire data analysis prompts; the data analysis prompts include data of at least one modality.

[0202] The retrieval module 1002 is used to perform a retrieval based on the data analysis prompts and obtain retrieval results; the retrieval results include data of at least one modality.

[0203] The determination module 1003 is used to determine the target vertical domain data analysis model from multiple candidate vertical domain data analysis models based on the modality suggested by the data analysis and the modality of the retrieval results.

[0204] The analysis module 1004 is used to perform data analysis on the data to be analyzed based on the target vertical domain data analysis model and the target data analysis algorithm when the target data analysis algorithm is determined to be invoked, and to obtain the target data analysis results; the data to be analyzed includes: the data analysis prompt and the search results.

[0205] Optionally, the retrieval module 1002 is specifically used to determine the target database from multiple candidate databases based on the data analysis prompts through an intent recognition model; the retrieval based on the data analysis prompts to obtain retrieval results includes: retrieving from the target database based on the data analysis prompts to obtain the retrieval results.

[0206] Optionally, the retrieval module 1002 is further configured to determine whether to perform a retrieval operation based on the data analysis prompt; and perform retrieval based on the data analysis prompt to obtain a retrieval result, including: when it is determined to perform the retrieval operation, performing retrieval based on the data analysis prompt to obtain the retrieval result; or when it is determined not to perform the retrieval operation, determining the target vertical domain data analysis large model from the plurality of candidate vertical domain data analysis large models based on the modality of the data analysis prompt.

[0207] Optionally, the retrieval module 1002 is further configured to determine whether to perform a retrieval operation based on the data analysis prompt; and perform retrieval based on the data analysis prompt to obtain a retrieval result, including: when it is determined to perform the retrieval operation, performing retrieval based on the data analysis prompt to obtain the retrieval result; or when it is determined not to perform the retrieval operation, determining the target vertical domain data analysis large model from the plurality of candidate vertical domain data analysis large models based on the modality of the data analysis prompt.

[0208] Optionally, the determination module 1003 is specifically configured to determine that the vertical domain large language model is the target vertical domain data analysis large model in response to the modality of the data analysis prompt being a text modality and the modality of the retrieval result being a text modality; or determine that the vertical domain multimodal inference model is the target vertical domain data analysis large model in response to the modality of the data analysis prompt including a non-text modality and / or the modality of the retrieval result including a non-text modality.

[0209] Optionally, the determination module 1003 is further configured to obtain a vertical domain multimodal sample data set and a tool invocation data set before performing retrieval based on the data analysis prompt to obtain a retrieval result, and the vertical domain multimodal sample data set includes a vertical domain multimodal sample prompt and a vertical domain sample response.

[0210] The preset model includes a hidden layer, a text output module, and a multimodal feature vector output module; the hidden layer is configured to perform semantic feature extraction based on the multimodal sample prompt to obtain semantic features for representing the multimodal sample prompt; the text output module is configured to output predicted text for describing the multimodal sample prompt based on the semantic features; the multimodal feature vector output module is configured to output predicted multimodal features for representing the multimodal sample prompt based on the semantic features; and the predicted text, the predicted multimodal features, and the vertical domain sample response are used to train the preset model.

[0211] Optionally, the determination module 1003 is further configured to obtain a vertical domain multimodal sample data set and a tool invocation data set before performing retrieval based on the data analysis prompt to obtain a retrieval result, and the vertical domain multimodal sample data set includes a vertical domain multimodal sample prompt and a vertical domain sample response.

[0212] Optionally, the analysis module 1004 is specifically configured to perform format conversion on the to-be-analyzed data by the target vertical domain data analysis large model to obtain the to-be-analyzed data in a target format; the target format is a data format required by the target data analysis algorithm; perform data analysis on the to-be-analyzed data in the target format based on the target data analysis algorithm to obtain an initial data analysis result; and obtain the target data analysis result based on the initial data analysis result by the target vertical domain data analysis large model.

[0213] The multi-modal data analysis apparatus 1000 provided in the application is used to execute the data analysis method embodiments described above, and has similar implementation principles and technical effects, which will not be described herein again.

[0214] Figure 11 A hardware structure schematic diagram of an electronic device provided in the application is provided. Figure 11 The electronic device 1100 shown includes a memory 1101, a processor 1102, and a communication interface 1103. The memory 1101, the processor 1102, and the communication interface 1103 are in communication connection with each other. For example, the memory 1101, the processor 1102, and the communication interface 1103 can be in communication connection in a network connection manner. Alternatively, the electronic device 1100 described above can further include a bus 1104. The memory 1101, the processor 1102, and the communication interface 1103 are in communication connection with each other through the bus 1104.

[0215] The memory 1101 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1101 can store a program, and when the program stored in the memory 1101 is executed by the processor 1102, the processor 1102 and the communication interface 1103 are configured to execute the data analysis method of any one of the embodiments described above. The memory can also store data required for at least one method in the data analysis.

[0216] The processor 1102 can be a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits.

[0217] The processor 1102 can also be an integrated circuit chip having a processing capability for signals. In the implementation process, the data analysis method of the present application can be completed by the integrated logic circuit of hardware or the instruction in the form of software in the processor 1102. The processor 1102 described above can also be a general processor, a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, which can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory 1101, and the processor 1102 reads the information in the memory 1101 and combines the hardware to complete the data analysis method of the present application.

[0218] The communication interface 1103 uses a transceiving module such as but not limited to a transceiver to realize the communication between the electronic device 1100 and other devices or communication networks. For example, the data set can be obtained through the communication interface 1103.

[0219] When the above electronic device 1100 includes a bus 1104, the bus 1104 can include a path for transmitting information between various components (for example, the memory 1101, the processor 1102, the communication interface 1103) of the electronic device 1100.

[0220] The present application also provides a computer readable storage medium, which can include: a U disk, a mobile hard disk, a read only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk and various storage program codes. Specifically, the computer readable storage medium stores program instructions, which are used for the method in the above embodiments.

[0221] The present application also provides a program product, which includes execution instructions stored in a readable storage medium. At least one processor of an electronic device can read the execution instructions from the readable storage medium, and the at least one processor executes the execution instructions to make the electronic device implement the method provided by the various embodiments.

[0222] It should be noted that, for the foregoing method embodiments, the method is described as a series of acts for simplicity, but the application is not limited to the order of the acts for performing the application, as some steps can be performed in other order or simultaneously. Moreover, those skilled in the art will understand that the embodiments described in the specification are all optional embodiments of the application, and the acts and modules involved are not necessarily essential to the application.

[0223] Further, it should be noted that, although the steps in the flowchart are shown in order according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in order, and the steps can be executed in other order. Moreover, at least part of the steps in the flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or sub-steps or stages of other steps.

[0224] It should be understood that the above-described device embodiments are only schematic, and the device of the application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical function division, and actual implementation can have another division manner. For example, multiple units / modules or components can be combined, or can be integrated into another system, or some features can be omitted or not executed.

[0225] In addition, unless otherwise specified, each functional unit / module in each embodiment of the application can be integrated in one unit / module, or each unit / module can exist physically, or two or more units / modules can be integrated together. The integrated unit / module can be realized in the form of hardware or in the form of a software program module.

[0226] If the integrated units / modules are implemented in the form of hardware, the hardware can be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit can be any appropriate magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0227] If the integrated units / modules are implemented in the form of software program modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the essential part or all or part of the technical solutions that make contributions to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0228] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0229] It should be understood that many of the materials and devices exemplified in this disclosure are articles of manufacture (i.e., articles of manufacture) according to this disclosure. The articles of manufacture can be employed as such to provide advantageous results. The articles of manufacture can also be employed in the practice or testing of the present disclosure. The articles of manufacture can also be employed in the practice or testing of other examples of the present disclosure. The articles of manufacture employed as such for practicing the present disclosure: include those devices and those manufacturing techniques that are known to those of skill in the art in this field. The articles of manufacture employed as such for practicing the present disclosure: include those devices and those manufacturing techniques that are analogous to those devices and those manufacturing techniques that are known to those of skill in the art in this field. The articles of manufacture employed as such for practicing the present disclosure: include those devices and those manufacturing techniques that are later developed by those of skill in the art in this field and are equivalent to those devices and those manufacturing techniques that are known to those of skill in the art in this field.

Claims

1. A multimodal data analysis intelligent agent based on a large model, characterized in that, The multimodal data analysis agent is used for: Obtain data analysis prompts; the data analysis prompts include data from at least one modality; Based on the data analysis prompts, a search is performed to obtain search results; the search results include data in at least one modality. Based on the modality suggested by the data analysis and the modality of the search results, the target vertical domain data analysis model is determined from multiple candidate vertical domain data analysis models. The multiple candidate vertical domain data analysis models include: a vertical domain multimodal reasoning model, and a vertical domain large language model; When determining the target data analysis algorithm to be invoked based on the data analysis prompts using the target vertical domain data analysis model, data analysis is performed on the data to be analyzed based on the target vertical domain data analysis model and the target data analysis algorithm to obtain the target data analysis results; the data to be analyzed includes: the data analysis prompts, and the search results; The process of determining the target vertical domain data analysis model from multiple candidate vertical domain data analysis models, based on the modalities suggested by the data analysis and the modalities of the search results, includes: In response to the data analysis prompt being in text mode and the search result being in text mode, the vertical domain large language model is determined to be the target vertical domain data analysis large model; or, In response to the modalities of the data analysis prompts, including non-textual modalities, and / or, the modalities of the search results, including non-textual modalities, the vertical domain multimodal reasoning model is determined to be the target vertical domain data analysis big model; The method, based on the target vertical domain data analysis model and the target data analysis algorithm, performs data analysis on the data to be analyzed to obtain target data analysis results, including: The target vertical domain data analysis model is used to convert the data to be analyzed into a format to obtain the data to be analyzed in the target format; the target format is the data format required by the target data analysis algorithm. Based on the target data analysis algorithm, the data to be analyzed in the target format is analyzed to obtain initial data analysis results; Based on the initial data analysis results, the target data analysis results are obtained through the target vertical domain data analysis model.

2. The multimodal data analysis intelligent agent according to claim 1, characterized in that, Before performing the retrieval based on the data analysis prompts and obtaining the retrieval results, the multimodal data analysis agent is also used to: Based on the data analysis prompts, the target database is determined from multiple candidate databases using an intent recognition model. The search based on the data analysis prompts yields the following search results: Based on the data analysis suggestions, a search is performed from the target database to obtain the search results.

3. The multimodal data analysis intelligent agent according to claim 2, characterized in that, The plurality of candidate databases includes at least: relational databases, and the search results obtained by retrieving from the target database based on the data analysis suggestions include: In response to the fact that the target database is the relational database, a database query statement is generated based on the data analysis prompts using a code generation model; Based on the database query statement, the relational database is queried to obtain the retrieval results.

4. The multimodal data analysis agent according to any one of claims 1-3, characterized in that, The multimodal data analysis agent is also used for: Obtain a vertical multimodal sample dataset and a tool call dataset, wherein the vertical multimodal sample dataset includes: vertical multimodal sample prompts and vertical sample responses; An initial vertical multimodal inference model is obtained by training a preset model using the aforementioned vertical multimodal sample dataset. The preset model includes a hidden layer, a text output module, and a multimodal feature vector output module. The hidden layer is used to extract semantic features based on the multimodal sample prompts to obtain semantic features that characterize the multimodal sample prompts. The text output module is used to output predicted text describing the multimodal sample prompts based on the semantic features. The multimodal feature vector output module is used to output predicted multimodal features characterizing the multimodal sample prompts based on the semantic features. The predicted text, the predicted multimodal features, and the vertical sample responses are used to train the preset model. The trainable parameters of the multimodal feature vector output module are frozen, and the dataset and the vertical multimodal sample preference dataset are called through the tool. Reinforcement learning training is performed on the part of the initial vertical multimodal inference model other than the multimodal feature vector output module to obtain the vertical multimodal inference model.

5. The multimodal data analysis agent according to any one of claims 1-3, characterized in that, Before the multimodal data analysis agent performs a search based on the data analysis prompts and obtains the search results, it is also used to: Based on the data analysis suggestions, determine whether to perform a search operation; The search based on the data analysis prompts yields the following search results: When a search operation is determined, the search is performed based on the data analysis prompts to obtain the search results; or, When it is determined that no search operation will be performed, the target vertical domain data analysis model is determined from the multiple candidate vertical domain data analysis models based on the modality suggested by the data analysis.

6. A multimodal data analysis method based on a large model, characterized in that, The method includes: Obtain data analysis prompts; the data analysis prompts include data from at least one modality; Based on the data analysis prompts, a search is performed to obtain search results; the search results include data in at least one modality. Based on the modality suggested by the data analysis and the modality of the retrieval results, a target vertical domain data analysis model is determined from multiple candidate vertical domain data analysis models; the multiple candidate vertical domain data analysis models include: a vertical domain multimodal reasoning model and a vertical domain large language model; When determining the target data analysis algorithm to be invoked based on the data analysis prompts using the target vertical domain data analysis model, data analysis is performed on the data to be analyzed based on the target vertical domain data analysis model and the target data analysis algorithm to obtain the target data analysis results; the data to be analyzed includes: the data analysis prompts, and the search results; The process of determining the target vertical domain data analysis model from multiple candidate vertical domain data analysis models, based on the modalities suggested by the data analysis and the modalities of the search results, includes: In response to the data analysis prompt being in text mode and the search result being in text mode, the vertical domain large language model is determined to be the target vertical domain data analysis large model; or, In response to the modalities of the data analysis prompts, including non-textual modalities, and / or, the modalities of the search results, including non-textual modalities, the vertical domain multimodal reasoning model is determined to be the target vertical domain data analysis big model; The method, based on the target vertical domain data analysis model and the target data analysis algorithm, performs data analysis on the data to be analyzed to obtain target data analysis results, including: The target vertical domain data analysis model is used to convert the data to be analyzed into a format to obtain the data to be analyzed in the target format; the target format is the data format required by the target data analysis algorithm. Based on the target data analysis algorithm, the data to be analyzed in the target format is analyzed to obtain initial data analysis results; Based on the initial data analysis results, the target data analysis results are obtained through the target vertical domain data analysis model.

7. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in claim 6.

8. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of claim 6.

Images