Multi-modal data exploration method and device

Abstract

The invention discloses a multi-modal data exploration method and device. The method comprises the steps that a first instruction text input by a target object is obtained, and the first instruction text comprises task information of at least one probing task of the target object; analyzing the first instruction text by using an intention recognition model to obtain a task intention of the target object; under the condition that the task intention is probing multi-modal data, task information of all probing tasks in the first instruction text is analyzed through a pre-trained task planning decision model, and a target task execution strategy with the lowest execution cost is obtained; and according to the target task execution strategy, calling the corresponding model context protocol service agents in sequence to execute the corresponding probing tasks, and obtaining probing results fed back by the model context protocol service agents. According to the method and the device, the technical problems of high difficulty, low efficiency and poor accuracy of probing the multi-modal data by adopting manual programming and manual configuration in related technologies are solved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and more specifically, to a method and apparatus for multimodal data exploration. Background Technology

[0002] In the field of big data and artificial intelligence applications, data exploration is a crucial step in achieving effective data utilization. Its purpose is to comprehensively understand the structure, content, distribution, and relationships within the data. Current data exploration techniques primarily rely on manual programming, and the specific process is as follows:

[0003] For different types of data (such as database data, text files, audio files, video files, image files, etc.), corresponding probing code must be written. Users must also manually configure probing rules, such as setting data filtering conditions, thresholds for extracting key information, and dimensions for data distribution statistics. Furthermore, users must manually configure the task execution method, including selecting the execution order of probing tasks, allocating computing resources, and setting task trigger conditions.

[0004] Furthermore, due to the significant differences in the structure and characteristics of different types of data, manual programming is difficult to effectively identify the intrinsic relationships between multimodal data. Users often need to rely on subjective judgment based on experience, resulting in low accuracy and efficiency in the detection of correlations.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides a method and apparatus for probing multimodal data, which at least solves the technical problems of high difficulty, low efficiency and poor accuracy in probing multimodal data by manually programming and configuring related technologies.

[0007] According to one aspect of the embodiments of this application, a multimodal data exploration method is provided, comprising: acquiring a first instruction text input by a target object, wherein the first instruction text includes task information of at least one exploration task of the target object, and the task information includes at least: task quantity and exploration object; analyzing the first instruction text using an intent recognition model to obtain the task intent of the target object; when the task intent is to explore multimodal data, analyzing the task information of each exploration task in the first instruction text using a pre-trained task planning and decision model to obtain a target task execution strategy with the lowest execution cost, wherein the task planning and decision model is trained using a reinforcement learning algorithm; sequentially calling the corresponding model context protocol service agent to execute the corresponding exploration task according to the target task execution strategy, and obtaining the exploration results fed back by each model context protocol service agent.

[0008] Optionally, the training process of the intent recognition model includes: embedding multiple pre-collected second instruction texts using word embedding technology to obtain sentence vectors corresponding to each second instruction text, wherein the sentence vectors include word vectors of each word in the corresponding second instruction text; for each sentence vector, performing semantic analysis on the sentence vector using a first semantic encoding model to obtain the corresponding intent vector, and performing semantic analysis on each word vector in the sentence vector using a second semantic encoding model to obtain character-level semantic vectors corresponding to each word, and combining the annotation vectors corresponding to each word according to the order of words in the second instruction text to obtain the corresponding sequence annotation vectors; determining the training dataset based on the intent vectors and sequence annotation vectors of each sentence vector; and iteratively training the preset neural network model using the training dataset to obtain the intent recognition model.

[0009] Optionally, the training dataset is determined based on the intent vector and sequence label vector of each sentence vector, including: for each sentence vector, determining the first probability distribution corresponding to the intent vector based on the first semantic category of the second instruction text corresponding to the sentence vector, and determining the second probability distribution of each label vector in the sequence label vector based on the second semantic category of each word in the second instruction text corresponding to the sentence vector; and taking the intent vector and the first probability distribution corresponding to each second instruction text, and the second probability distribution of each label vector in the sequence label vector corresponding to the second instruction text as a set of training sample data to form the training dataset.

[0010] Optionally, the neural network model includes a first sub-network model and a second sub-network model. The intent recognition model is obtained by iteratively training the pre-built neural network model using a training dataset. This includes: for each group of training sample data in the training dataset, analyzing the intent vector in the training sample data using the first sub-network model to obtain a corresponding first predicted probability distribution, and analyzing the sequence label vector in the training sample data using the second sub-network model to obtain a corresponding second predicted probability distribution; constructing a first loss function based on the first probability distribution and the corresponding first predicted probability distribution in each group of training sample data, and constructing a second loss function based on the second probability distribution and the corresponding second predicted probability distribution in each group of training sample data; weighting the first loss function and the second loss function to obtain a target loss function, and adjusting the model parameters of the neural network model using gradient descent until the value of the target loss function is less than a preset termination threshold, thus obtaining a trained intent recognition model.

[0011] Optionally, the training process of the task planning decision model includes: constructing a value network and a target network for solving the task execution strategy using neural network technology, and initializing the network parameters of the value network and the target network; initializing the agent state in each iteration cycle and determining a preset number of computation cycles, wherein the agent state includes at least: task information for each exploration task and available computational resource information for the target agent; inputting the current agent state into the value network in each computation cycle and calculating the Q-values ​​corresponding to different task execution strategies; selecting the first task execution strategy with the largest Q-value based on a greedy strategy, executing the first task execution strategy and calculating the corresponding reward, and obtaining... The new agent state is defined as follows: the current agent state, the target task execution strategy, the reward, and the new agent state are taken as a sample and stored in a pre-defined experience pool. Multiple randomly sampled samples from the experience pool are input into a neural network containing a bidirectional long short-term memory network. The target Q value corresponding to each sample is calculated, and a target loss function is constructed based on the reward and the corresponding target Q value of each sample. Stochastic gradient descent is used to update the network parameters of the value network to minimize the value of the target loss function. After a pre-defined number of calculation cycles, the network weight parameters of the target network are updated according to the network weight parameters of the value network. After the iteration is completed, the obtained target network is used as the task planning decision model.

[0012] Optionally, the current agent state is input into the value network to calculate the Q-values ​​corresponding to different task execution strategies, including: determining multiple task execution strategies; for each task execution strategy, calculating the execution cost corresponding to the task execution strategy using the current agent state and the task execution strategy, wherein the execution cost includes: the execution time cost and execution resource cost of all exploration tasks; determining the Q-value corresponding to each task execution strategy based on the execution cost corresponding to each task execution strategy, wherein the higher the execution cost of the task execution strategy, the smaller the corresponding Q-value.

[0013] Optionally, after obtaining the exploration results fed back by each model context protocol service agent, the method further includes: converting the exploration results fed back by each model context protocol service agent into the embedding representation of the corresponding modality, and mapping the embedding representation of each modality to the same semantic space to obtain the target embedding representation of each modality; calculating the similarity between the target embedding vectors of each pair of modalities, and if the similarity is higher than a preset threshold, determining the association rule between the exploration results corresponding to the two target embedding vectors based on the similarity and the frequency of occurrence of the exploration results; and storing the exploration results fed back by each model context protocol service agent and the association rule between each exploration result in a preset association rule base.

[0014] According to another aspect of the embodiments of this application, a multimodal data exploration device is also provided, comprising: an acquisition module, configured to acquire a first instruction text input by a target object, wherein the first instruction text includes task information of at least one exploration task of the target object, and the task information includes at least: task quantity and exploration object; an analysis module, configured to analyze the first instruction text using an intent recognition model to obtain the task intent of the target object; a planning module, configured to, when the task intent is to explore multimodal data, analyze the task information of each exploration task in the first instruction text using a pre-trained task planning decision model to obtain the target task execution strategy with the lowest execution cost, wherein the task planning decision model is trained using a reinforcement learning algorithm; and an execution feedback module, configured to sequentially call the corresponding model context protocol service agent to execute the corresponding exploration task according to the target task execution strategy, and obtain the exploration results fed back by each model context protocol service agent.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, the computer program product comprising: a computer program, wherein the computer program, when executed by a processor, implements the above-described multimodal data probing method.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described multimodal data probing method through the computer program.

[0017] In this embodiment, an intent recognition model is used to accurately understand the task requirements corresponding to the first instruction text input by the target object, avoiding the limitations of traditional methods where users need to precisely format instructions or the system needs to understand instructions. When the task intent is to explore multimodal data, a task planning and decision model is used to analyze the task information of each exploration task in the first instruction text to obtain the target task execution strategy with the lowest execution cost. Based on the target task execution strategy, the corresponding model context protocol service agent is called sequentially to execute the corresponding exploration task to automatically obtain the exploration results. This reduces the need for manual task configuration and intervention, significantly improves the automation level of multimodal data exploration, and solves the technical problems of high difficulty, low efficiency, and poor accuracy in exploring multimodal data by using manual programming and configuration. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1This is a flowchart illustrating an optional multimodal data exploration method according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of an optional multimodal data exploration system according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the structure of an optional multimodal data exploration device according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the hardware structure of a computer terminal for implementing a multimodal data exploration method according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] Example 1

[0026] According to an embodiment of this application, a multimodal data exploration method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] Figure 1 This is a flowchart illustrating a multimodal data exploration method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:

[0028] Step S102: Obtain the first instruction text input by the target object, wherein the first instruction text includes task information of at least one exploration task of the target object, and the task information includes at least: task quantity and exploration object.

[0029] Step S104: Analyze the first instruction text using the intent recognition model to obtain the task intent of the target object.

[0030] Step S106: When the task intent is to explore multimodal data, the task information of each exploration task in the first instruction text is analyzed using a pre-trained task planning and decision model to obtain the target task execution strategy with the lowest execution cost. The task planning and decision model is trained using a reinforcement learning algorithm.

[0031] Step S108: Based on the target task execution strategy, the corresponding model context protocol service agents are sequentially invoked to execute the corresponding exploration tasks, and the exploration results fed back by each model context protocol service agent are obtained.

[0032] Based on the scheme defined in steps S102 to S108 above, it can be seen that in this embodiment of the application, the intent recognition model is used to accurately understand the task requirements corresponding to the first instruction text input by the target object, avoiding the limitations of traditional methods where users need to precisely format instructions or the system needs to understand instructions. When the task intent is to explore multimodal data, the task planning and decision model is used to analyze the task information of each exploration task in the first instruction text to obtain the target task execution strategy with the lowest execution cost. Based on the target task execution strategy, the corresponding model context protocol service agent is called in sequence to execute the corresponding exploration task to automatically obtain the exploration results. This reduces the need for manual task configuration and intervention, and significantly improves the automation level of multimodal data exploration.

[0033] Specifically, Figure 2 This is a schematic diagram of an optional multimodal data system provided according to an embodiment of this application, such as... Figure 2 As shown, the system includes at least: an agent service layer for executing multimodal data exploration methods and an MCP service layer for interacting with it. The multimodal data exploration methods executed by the agent service layer (also known as target agents, which include task scheduling agents, chat agents, etc.) will be described below through the following embodiments.

[0034] In the technical solution provided in step S102 above, the target agent can first obtain the first instruction input by the target object in the form of natural language on the user interface; then parse the first instruction to obtain the corresponding first quality text, wherein the first instruction text describes the task information of each exploration task that the target object wants to perform, and the task information includes, but is not limited to: task quantity, storage location of the exploration object (such as database data, text file, audio file, video file, image file), exploration complexity, etc.

[0035] In the technical solution provided in step S104 above, the target intelligent agent can further analyze the first instruction text using the intent recognition model to obtain the task intent of the target object.

[0036] As an optional implementation, the training process of the above-mentioned intent recognition model includes:

[0037] First, word embedding technology is used to embed multiple pre-collected second instruction texts to obtain sentence vectors corresponding to each second instruction text. The sentence vectors include word vectors of each word in the corresponding second instruction text.

[0038] In other words, for each second instruction text, word embedding techniques (such as Word2Vec, GloVe, or more advanced BERT, RoBERTa techniques, etc.) are used to generate context-sensitive word vectors for each word in the second instruction text. Then, these word vectors are merged into a sentence vector that can represent the entire second instruction text through aggregation operations (such as average pooling, max pooling, etc.).

[0039] Next, for each sentence vector, the first semantic coding model is used to perform semantic analysis on the sentence vector to obtain the corresponding intent vector, and the second semantic coding model is used to perform semantic analysis on each word vector within the sentence vector to obtain the character-level semantic vector corresponding to each word. Then, the annotation vectors corresponding to each word are combined according to the order of words in the second instruction text to obtain the corresponding sequence annotation vector.

[0040] In other words, the first semantic encoding model can be a gated recurrent unit (GRU) or a long short-term memory (LSTM) network, used to process sentence vectors and extract their deep semantic features to obtain an intent vector, which is a high-dimensional abstract representation of the intent of the instruction text. The second semantic encoding model can be a character-level convolutional neural network or a recurrent neural network, used to generate a character-level semantic vector for each word. This vector contains semantic information about the internal structure and details of the word, helping the model understand the meaning of words in a specific context. Furthermore, the character-level semantic vectors of each word are arranged according to the natural order of the words in the second instruction text to form a sequence label vector. This sequence label vector provides semantic annotations of the word sequence in the second instruction text, helping the model capture the relationships between words and the importance of word order.

[0041] Then, the training dataset is determined based on the intent vector and sequence label vector of each sentence vector.

[0042] Finally, the pre-set neural network model is iteratively trained using the training dataset to obtain the intent recognition model.

[0043] During the training process of the aforementioned intent recognition model, deep learning techniques, including word embedding transformation and semantic encoding models, are used to meticulously analyze the semantic structure of user commands. This not only captures the intent of the entire sentence but also focuses on the character-level semantics of each word, thereby constructing a high-precision intent recognition model.

[0044] Optionally, the specific process of determining the training dataset may include: for each sentence vector, determining the first probability distribution corresponding to the intent vector based on the first semantic category of the second instruction text corresponding to the sentence vector, and determining the second probability distribution of each annotation vector in the sequence label vector based on the second semantic category of each word in the second instruction text corresponding to the sentence vector; and taking the intent vector and the first probability distribution corresponding to each second instruction text, and the second probability distribution of each annotation vector in the sequence label vector corresponding to the second instruction text as a set of training sample data to form a training dataset.

[0045] The above technical solution can be understood as follows: The semantic category of the second instruction text corresponding to the sentence vector is labeled manually or using existing semantic analysis tools. Based on the semantic category of the sentence vector, a first probability distribution of the corresponding intent vector is determined. This first probability distribution reflects the confidence level of the semantic category to which the second instruction text belongs and can be considered a soft label for a classification task. Similarly, the second semantic category of each word in the second instruction text corresponding to the sentence vector is labeled manually or using existing semantic analysis tools. Based on the second semantic category of each word, a second probability distribution of the label vector corresponding to each word is determined. This second probability distribution reflects the confidence level of the semantic category to which the corresponding word belongs. Furthermore, the intent vector and its first probability distribution of each second instruction text, and the second probability distributions of all label vectors within the sequence label vectors corresponding to the second instruction text, are combined to form a set of training sample data. Thus, each set of training sample data not only contains the global semantic information of the second instruction text (represented by the intent vector and its probability distribution) but also retains the local semantic details of the second instruction text (represented by the label vectors within the sequence label vectors and their probability distributions).

[0046] By repeating the above steps, corresponding training sample data is constructed for all second instruction texts, forming a complete training dataset. This training dataset will be used to iteratively train the neural network model to learn how to predict the correct intent vector from the instruction text.

[0047] Furthermore, the neural network model includes: a first sub-network model and a second sub-network model. The first sub-network model focuses on processing global intent, which is used to capture the global semantic features of the sentence vector of the instruction text and output the predicted probability distribution of various intents. The second sub-network model focuses on local semantic understanding, which is used for word-by-word analysis and outputs the predicted probability distribution of the semantic category corresponding to each word.

[0048] Optionally, the above iterative training process may include: for each group of training sample data in the training dataset, analyzing the intent vector in the training sample data using a first sub-network model to obtain the corresponding first predicted probability distribution, and analyzing the sequence label vector in the training sample data using a second sub-network model to obtain the corresponding second predicted probability distribution; constructing a first loss function based on the first probability distribution and the corresponding first predicted probability distribution in each group of training sample data, and constructing a second loss function based on the second probability distribution and the corresponding second predicted probability distribution in each group of training sample data; weighting the first loss function and the second loss function to obtain the target loss function, and adjusting the model parameters of the neural network model using gradient descent until the value of the target loss function is less than a preset termination threshold, thereby obtaining a trained intent recognition model.

[0049] The first loss function is obtained by comparing the error between the true first probability distribution and the first predicted probability distribution predicted by the first sub-network model, and is used for global intent prediction. The second loss function is obtained by comparing the error between the true second probability distribution and the second predicted probability distribution predicted by the second sub-network model, and is used for lexical-level semantic classification. Both the first and second loss functions can be cross-entropy loss functions. Furthermore, when weighting and summing the first and second loss functions, the weight ratio can be adjusted according to task requirements; this embodiment does not impose specific limitations on this.

[0050] Using the above method, the embodiments of this application can train a deep learning model that can understand the global intent of the instruction text and capture the semantic details of local words, which greatly improves the accuracy and efficiency of user interaction in multimodal data exploration scenarios.

[0051] In one implementation, the intent recognition model can be loaded into memory. For example, the raw data of the intent recognition model can be loaded from non-volatile memory into volatile memory so that the processor can run the intent recognition model. The raw data of the intent recognition model refers to unprocessed data, which typically includes the parameters and structural data of the intent recognition model. The structural data can be parameter-based computational relationships, such as forward propagation computational relationships between intermediate layers or between neurons. Specifically, the structural data can include structure-related code of the intent recognition model, such as code used to perform related computations between intermediate layers or between neurons.

[0052] Additionally, a region can be partitioned in memory for loading the intent recognition model, which may include a structure data storage area and a parameter storage area. The structure data storage area stores structure-related code, and the parameters referenced by it can be accessed via pointers to the addresses of specific parameters in the parameter storage area. During the training of the intent recognition model, frequent parameter updates may be required; in this case, updating the parameter values ​​in the parameter storage area is sufficient.

[0053] Therefore, the target agent can use the trained intent recognition model to analyze the first instruction text and obtain the target object's task intent. The types of task intent include, but are not limited to: exploring multimodal data, exploring data permissions, exploring data security, and exploring data timeliness.

[0054] For example, if the first instruction text is "Help me explore the company's customer data, including customer information tables in the database, relevant customer feedback texts, audio recordings of customer inquiries, product promotional videos, and customer photos, and analyze the relationships between them", then by analyzing the first instruction text through the intent recognition model, it can be determined that the task intent is to explore multimodal data.

[0055] In the technical solution provided in step S106 above, when the task intent is to explore multimodal data, a pre-trained task planning and decision model is used to analyze the task information of each exploration task in the first instruction text to obtain the target task execution strategy with the lowest execution cost.

[0056] As an optional implementation, the training process of the above task planning decision model includes:

[0057] (1) Network construction and parameter initialization:

[0058] We construct a value network and a target network using neural network technology to solve the task execution strategy, and initialize the network parameters of the value network and the target network.

[0059] Among them, the value network is used to predict the instantaneous Q-values ​​of different task execution strategies online, while the target network is used to calculate the expected Q-values ​​of the future agent state. Its network structure is similar to that of the value network but its parameters are independent.

[0060] (2) Decision execution and learning within the iterative cycle:

[0061] First, initialize the agent state and determine a preset number of computation cycles. The agent state includes at least: task information for each exploration task and corresponding available computing resource information. The task information includes, but is not limited to: task type, data volume, exploration complexity, etc., and the available computing resource information includes, but is not limited to: CPU utilization, memory usage, network bandwidth, etc.

[0062] Within each calculation cycle, perform the following steps:

[0063] Step 1: Input the current agent state into the value network and calculate the Q value corresponding to different task execution strategies, where the Q value represents the expected reward that can be obtained by executing the task execution strategy in the current agent state;

[0064] Step 2: Select the first task execution strategy with the largest Q value based on the greedy strategy, execute the first task execution strategy and calculate the corresponding reward, and obtain the new agent state;

[0065] Step 3: Take the current agent state, target task execution strategy, reward and new agent state as a sample and store the sample in the preset experience pool;

[0066] Step 4: Input multiple randomly sampled samples from the experience pool into a neural network containing a bidirectional long short-term memory network, calculate the target Q value for each sample, construct a target loss function based on the reward of each sample and the corresponding target Q value, and use stochastic gradient descent to update the network parameters of the value network to minimize the value of the target loss function.

[0067] After a preset number of calculation cycles, the network weight parameters of the target network are updated based on the network weight parameters of the value network.

[0068] After iteration, the resulting target network is used as a task planning and decision model for agents to select task execution strategies in real-world environments.

[0069] Specifically, the Q-value corresponding to the above task execution strategies can be determined as follows: Multiple task execution strategies are determined, whereby the task execution strategy represents the execution order and resource allocation plan of multiple exploration tasks; for each task execution strategy, the execution cost corresponding to the task execution strategy is calculated using the current agent state and the task execution strategy, where the execution cost includes: the execution time cost and execution resource cost of all exploration tasks; the Q-value corresponding to each task execution strategy is determined based on the execution cost of each task execution strategy, where the higher the execution cost of the task execution strategy, the smaller the corresponding Q-value.

[0070] The aforementioned execution time cost is calculated by simulating the task execution strategy to assess the total time required to complete all exploration tasks. The execution resource cost, on the other hand, is calculated by simulating the task execution strategy to assess the total computing resources (such as CPU, GPU, memory, network bandwidth, etc.) required to complete all exploration tasks. Then, the execution time cost and execution resource cost are weighted and summed to obtain the execution cost corresponding to the task execution strategy. Based on the execution cost corresponding to the task execution strategy, the Q-value corresponding to the task execution strategy is calculated in reverse (which represents the expected reward obtained by executing a certain strategy in a given state in reinforcement learning).

[0071] In one implementation, the task planning and decision model can be loaded into memory. For example, the raw data of the task planning and decision model can be loaded from non-volatile memory into volatile memory so that the processor can run the task planning and decision model. The raw data of the intent recognition model refers to unprocessed data, which typically includes the parameters and structural data of the task planning and decision model. The structural data can be parameter-based computational relationships, such as forward propagation computational relationships between intermediate layers or neurons. Specifically, the structural data can include structure-related code of the task planning and decision model, such as code used to perform related computations between intermediate layers or neurons.

[0072] Additionally, a region can be partitioned in memory for loading the task planning and decision model, which may include a structure data storage area and a parameter storage area. The structure data storage area stores structure-related code, and the parameters referenced by it can be accessed via pointers to the addresses of specific parameters in the parameter storage area. During the training of the task planning and decision model, frequent parameter updates may be required; in this case, updating the parameter values ​​in the parameter storage area is sufficient.

[0073] Therefore, the target agent can use the task planning and decision model to analyze the task information of each exploration task in the first instruction text, obtain the target task execution strategy with the lowest execution cost, realize the technical effect of automatically optimizing the execution strategy of multimodal data exploration tasks, and achieve the goal of improving exploration efficiency and maximizing resource utilization.

[0074] In the technical solution provided in step S108 above, the target agent can sequentially call the corresponding Model Context Protocol (MCP) service agents to execute the corresponding exploration tasks according to the target task execution strategy, and obtain the exploration results fed back by each Model Context Protocol service agent.

[0075] For example, consider the first instruction text, whose intent is identified as exploring multimodal data: "Help me explore the company's customer data, including customer information tables in the database, relevant customer feedback text, audio recordings of customer inquiries, product promotional videos, and customer photos, and analyze the relationships between them." Then, by analyzing the task information of each exploration task within this first instruction text using a task planning and decision model, the resulting target task execution strategy could be: first explore the database data and text files, then explore the audio, video, and image files.

[0076] Specifically, the MCP service agents that perform the aforementioned exploration tasks include:

[0077] (1) The database MCP service agent adopts a client-server architecture. The server deploys a database connection pool, supporting connections with various mainstream databases (such as MySQL, PostgreSQL, Oracle, etc.). It provides services to the database MCP service agent through a unified interface, thereby realizing the automatic generation of SQL statements. Specifically, the database MCP service agent automatically generates corresponding SQL query statements based on the task information of the exploration task (such as database name, table name, and exploration type), avoiding the tediousness of manually writing SQL. At the same time, the database MCP service agent can also convert the exploration results returned by the database into the JSON format specified by the MCP protocol, ensuring that each data exploration agent can parse it correctly.

[0078] (2) The text MCP service agent supports reading various text formats (such as txt, doc, docx, etc.) and parses the command text file by calling third-party libraries. At the same time, the text MCP service agent has a built-in text content extraction engine that can remove redundant information in the text (such as headers, footers, blank lines, etc.) and accurately extract the core content.

[0079] (3) The audio MCP service agent can convert audio files of different formats (such as mp3, wav, flac, etc.) into a unified processing format, which is convenient for subsequent analysis.

[0080] (4) The video MCP service agent can use a timed sampling mechanism to extract video frames. The sampling interval can be determined by the video duration and frame rate. As for video subtitles, optical character recognition technology and image recognition technology can be used to extract text information such as subtitles and text labels from video frames and identify objects, scenes and other content in the picture.

[0081] (5) The image MCP service agent supports decoding and reading of various image formats (such as jpg, png, gif, etc.) to obtain image pixel data and metadata. Simultaneously, it employs optical character recognition technology to extract non-handwritten text from images and a handwritten character recognition model to extract handwritten text, thereby improving image recognition accuracy. Additionally, a pre-trained image captioning model can be used to generate natural language descriptions of the image content.

[0082] Therefore, the target agent can sequentially assign database exploration tasks to the database MCP service agent to explore the database, table, and field structure of the customer information table, query data content, statistical data distribution, and other information, and store it in a vector database or knowledge base; assign text exploration tasks to the text MCP service agent to explore basic information such as the name, type, and size of customer feedback text, extract text content and key information (such as customer satisfaction, opinions, and suggestions), and store it in a vector database or knowledge base; assign audio exploration tasks to the audio MCP service agent to explore basic information such as the name, type, size, and frequency of customer consultation audio records, and convert the audio... For text content, key information (such as consultation questions and needs) is extracted and stored in a vector database or knowledge base. Video exploration tasks are assigned to video MCP service agents to explore basic information such as the name, type, size, frequency, and resolution of product promotional videos. The videos are then converted into text content, key information (such as product features and promotional focus) is extracted, and stored in a vector database or knowledge base. Image exploration tasks are assigned to image MCP service agents to explore basic information such as the name, type, size, and resolution of customer photos. The images are converted into text descriptions, key information (such as customer appearance features and the surrounding environment) is extracted, and stored in a vector database or knowledge base. Additionally, the target agent can preprocess the exploration results from each MCP service agent (such as data cleaning and formatting), converting the processed structured data (such as database table content) into vector form for storage in the vector database, while storing unstructured data (such as text content) in the knowledge base.

[0083] Furthermore, each Model Context Protocol (MCP) service agent can leverage the rule engine and large model inference capabilities to generate standardized exploration step sequences based on data type and exploration target. It can also automatically generate invocation parameters based on these sequences, achieving seamless integration with the corresponding MCP service. For example, the database MCP service agent can generate an exploration step sequence of "connecting to the database → querying the table structure → extracting data content → statistically analyzing data distribution" for exploring a customer information table.

[0084] Furthermore, in addition to storing the exploration results fed back by each Model Context Protocol Service Agent to the vector database and knowledge base respectively, the target agent can also associate the multimodal data fed back by these Model Context Protocol Service Agents. The specific implementation is as follows: the exploration results fed back by each Model Context Protocol Service Agent are converted into the corresponding modality's embedding representation using an embedding model, and the embedding representations of each modality are mapped to the same semantic space to obtain the target embedding representations of multiple modalities; the similarity between the target embedding vectors of each pair of modalities is calculated, and if the similarity is higher than a preset threshold, the association rule between the exploration results corresponding to the two target embedding vectors is determined based on the similarity and the frequency of the exploration results; the exploration results fed back by each Model Context Protocol Service Agent and the association rules between the exploration results are stored in a preset association rule base.

[0085] Specifically, the target agent can use the Schema2vec / table embedding model to convert the exploration results returned by the database MCP service agent into structured data embedding vectors; and use the text embedding model to convert the exploration results returned by the text MCP service agent into text embedding vectors v. text =f LM (d text The image MCP service agent's exploration results are converted into image vectors v using a visual coding model. image =f vision (d image For the exploration results fed back by the audio MCP service agent, the audio exploration results can first be converted into text, and then the converted text content can be converted into corresponding embedding vectors using a text embedding model. Finally, a joint embedding vector v can be generated by combining the audio features. audio =α·f ASR (d audio )+(1-α)·f MFCC (d audio For the exploration results fed back by the video MCP service agent, the video frames in the video exploration results can be extracted first, then the image feature extraction network can be used to identify the image frames of the video frames, and optical character recognition technology can be used to identify the subtitles in the video frames to construct the corresponding embedding vectors v. video =[f image (frame),f OCR[subtitle] These modal embeddings capture the deep semantic features of the data. A cross-modal alignment model is then used to map the embeddings of each modality to the same semantic space to find consistent semantic representations across multiple modalities, resulting in target embeddings for each modality. Then, within the same semantic space, the similarity between the target embedding vectors of each pair of modalities is calculated using metrics such as cosine similarity or Euclidean distance. When the similarity exceeds a preset threshold, the frequency and context of the exploration results are further analyzed. Statistical analysis and pattern recognition are used to determine the association rules between the exploration results of two modalities. That is, based on similarity, association rules are formed by combining domain knowledge or explicit identifiers (such as customer ID, timestamps, etc.). Where A and B represent the exploration results of different modalities, conf represents the confidence level determined based on similarity, and supp represents the support level determined based on the frequency of occurrence of the exploration results. Finally, the exploration results fed back by each model context protocol service agent and the association rules between each exploration result are stored in a preset association rule library.

[0086] Furthermore, the target agent can also associate the association rule base with the vector database and knowledge base to obtain association rule-vector mapping tables and association rule-knowledge mapping tables. This allows the target agent to directly retrieve the corresponding multimodal exploration results from the vector database and / or knowledge base based on the association rule base and provide them to the user when they subsequently query for information.

[0087] It should be noted that, in the embodiments of this application, the association rule base can also store the association strength determined by the similarity and frequency of occurrence between the exploration results corresponding to the two target embedding vectors. The higher the similarity and the more frequent the occurrence, the higher the corresponding association strength.

[0088] Through the above steps, the target agent can utilize the task planning and decision model to analyze the task information of each exploration task within the first instruction text, automatically obtaining the target task execution strategy with the lowest execution cost. This avoids manually configuring the execution order and allocation of exploration tasks, which leads to low task execution efficiency and a high risk of errors. Simultaneously, using the model context protocol, the service agent can automatically plan task execution steps, achieving automation and standardization of data exploration, reducing user operation difficulty, and improving exploration efficiency and data management standardization. Furthermore, by automatically associating and saving the exploration results of various modalities, the accuracy of the relationships between exploration results of different modalities is improved, providing strong support for the comprehensive analysis of multimodal data.

[0089] Example 2

[0090] According to embodiments of this application, a multimodal data probing device for implementing the multimodal data probing method in Embodiment 1 is also provided, such as... Figure 3 As shown, the multimodal data exploration device includes at least: an acquisition module 32, an analysis module 34, a planning module 36, and an execution feedback module 38, wherein:

[0091] The acquisition module 32 is used to acquire the first instruction text input by the target object, wherein the first instruction text includes task information of at least one exploration task of the target object, and the task information includes at least: task quantity and exploration object.

[0092] Analysis module 34 is used to analyze the first instruction text using an intent recognition model to obtain the task intent of the target object.

[0093] As an optional implementation, the training process of the above-mentioned intent recognition model includes:

[0094] First, word embedding technology is used to embed multiple pre-collected second instruction texts to obtain sentence vectors corresponding to each second instruction text. The sentence vectors include word vectors of each word in the corresponding second instruction text.

[0095] Next, for each sentence vector, the first semantic coding model is used to perform semantic analysis on the sentence vector to obtain the corresponding intent vector, and the second semantic coding model is used to perform semantic analysis on each word vector within the sentence vector to obtain the character-level semantic vector corresponding to each word. Then, the annotation vectors corresponding to each word are combined according to the order of words in the second instruction text to obtain the corresponding sequence annotation vector.

[0096] Then, the training dataset is determined based on the intent vector and sequence label vector of each sentence vector.

[0097] Finally, the pre-set neural network model is iteratively trained using the training dataset to obtain the intent recognition model.

[0098] During the training process of the aforementioned intent recognition model, deep learning techniques, including word embedding transformation and semantic encoding models, are used to meticulously analyze the semantic structure of user commands. This not only captures the intent of the entire sentence but also focuses on the character-level semantics of each word, thereby constructing a high-precision intent recognition model.

[0099] Optionally, the specific process of determining the training dataset may include: for each sentence vector, determining the first probability distribution corresponding to the intent vector based on the first semantic category of the second instruction text corresponding to the sentence vector, and determining the second probability distribution of each annotation vector in the sequence label vector based on the second semantic category of each word in the second instruction text corresponding to the sentence vector; and taking the intent vector and the first probability distribution corresponding to each second instruction text, and the second probability distribution of each annotation vector in the sequence label vector corresponding to the second instruction text as a set of training sample data to form a training dataset.

[0100] By repeating the above steps, corresponding training sample data is constructed for all second instruction texts, forming a complete training dataset. This training dataset will be used to iteratively train the neural network model to learn how to predict the correct intent vector from the instruction text.

[0101] Furthermore, the neural network model includes: a first sub-network model and a second sub-network model. The first sub-network model focuses on processing global intent, which is used to capture the global semantic features of the sentence vector of the instruction text and output the predicted probability distribution of various intents. The second sub-network model focuses on local semantic understanding, which is used for word-by-word analysis and outputs the predicted probability distribution of the semantic category corresponding to each word.

[0102] Optionally, the above iterative training process may include: for each group of training sample data in the training dataset, analyzing the intent vector in the training sample data using a first sub-network model to obtain the corresponding first predicted probability distribution, and analyzing the sequence label vector in the training sample data using a second sub-network model to obtain the corresponding second predicted probability distribution; constructing a first loss function based on the first probability distribution and the corresponding first predicted probability distribution in each group of training sample data, and constructing a second loss function based on the second probability distribution and the corresponding second predicted probability distribution in each group of training sample data; weighting the first loss function and the second loss function to obtain the target loss function, and adjusting the model parameters of the neural network model using gradient descent until the value of the target loss function is less than a preset termination threshold, thereby obtaining a trained intent recognition model.

[0103] Planning module 36 is used to analyze the task information of each exploration task in the first instruction text when the task intent is to explore multimodal data, and obtain the target task execution strategy with the lowest execution cost. The task planning decision model is trained using a reinforcement learning algorithm.

[0104] As an optional implementation, the training process of the above task planning decision model includes:

[0105] (1) Network construction and parameter initialization:

[0106] We construct a value network and a target network using neural network technology to solve the task execution strategy, and initialize the network parameters of the value network and the target network.

[0107] Among them, the value network is used to predict the instantaneous Q-values ​​of different task execution strategies online, while the target network is used to calculate the expected Q-values ​​of the future agent state. Its network structure is similar to that of the value network but its parameters are independent.

[0108] (2) Decision execution and learning within the iterative cycle:

[0109] First, initialize the agent state and determine a preset number of computation cycles. The agent state includes at least: task information for each exploration task and corresponding available computing resource information. The task information includes, but is not limited to: task type, data volume, exploration complexity, etc., and the available computing resource information includes, but is not limited to: CPU utilization, memory usage, network bandwidth, etc.

[0110] Within each calculation cycle, perform the following steps:

[0111] Step 1: Input the current agent state into the value network and calculate the Q value corresponding to different task execution strategies, where the Q value represents the expected reward that can be obtained by executing the task execution strategy in the current agent state;

[0112] Step 2: Select the first task execution strategy with the largest Q value based on the greedy strategy, execute the first task execution strategy and calculate the corresponding reward, and obtain the new agent state;

[0113] Step 3: Take the current agent state, target task execution strategy, reward and new agent state as a sample and store the sample in the preset experience pool;

[0114] Step 4: Input multiple randomly sampled samples from the experience pool into a neural network containing a bidirectional long short-term memory network, calculate the target Q value for each sample, construct a target loss function based on the reward of each sample and the corresponding target Q value, and use stochastic gradient descent to update the network parameters of the value network to minimize the value of the target loss function.

[0115] After a preset number of calculation cycles, the network weight parameters of the target network are updated based on the network weight parameters of the value network.

[0116] After iteration, the resulting target network is used as a task planning and decision model for agents to select task execution strategies in real-world environments.

[0117] Specifically, the Q-value corresponding to the above task execution strategies can be determined as follows: Multiple task execution strategies are determined, whereby the task execution strategy represents the execution order and resource allocation plan of multiple exploration tasks; for each task execution strategy, the execution cost corresponding to the task execution strategy is calculated using the current agent state and the task execution strategy, where the execution cost includes: the execution time cost and execution resource cost of all exploration tasks; the Q-value corresponding to each task execution strategy is determined based on the execution cost of each task execution strategy, where the higher the execution cost of the task execution strategy, the smaller the corresponding Q-value.

[0118] The execution feedback module 38 is used to sequentially call the corresponding model context protocol service agents to execute the corresponding exploration tasks according to the target task execution strategy, and obtain the exploration results fed back by each model context protocol service agent.

[0119] Specifically, the execution feedback module 38 can not only store the exploration results fed back by each model context protocol service agent to a vector database or knowledge base, but also associate the multimodal data fed back by these model context protocol service agents. The specific implementation method is as follows: the exploration results fed back by each model context protocol service agent are converted into the embedding representation of the corresponding modality using an embedding model, and the embedding representation of each modality is mapped to the same semantic space to obtain the target embedding representation of each modality; the similarity between the target embedding vectors of each two modalities is calculated, and if the similarity is higher than a preset threshold, the association rule between the exploration results corresponding to the two target embedding vectors is determined based on the similarity and the frequency of the exploration results; the exploration results fed back by each model context protocol service agent and the association rules between the exploration results are stored in a preset association rule base.

[0120] It should be noted that each module in the multimodal data probing device in this application embodiment corresponds one-to-one with each implementation step of the multimodal data probing method in Embodiment 1. Since Embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to Embodiment 1, and will not be elaborated further here.

[0121] Example 3

[0122] According to an embodiment of this application, a computer program product is also provided, which includes a computer program, wherein when the computer program is executed by a processor, it implements the multimodal data probing method in embodiment 1.

[0123] According to an embodiment of this application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device containing the non-volatile storage medium executes the multimodal data probing method in Embodiment 1 by running the computer program.

[0124] According to an embodiment of this application, a processor is also provided for running a computer program, wherein the computer program executes the multimodal data probing method in embodiment 1 during runtime.

[0125] According to an embodiment of this application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the multimodal data probing method of embodiment 1 through the computer program.

[0126] Specifically, the computer program executes the following steps during runtime: It acquires a first instruction text input by the target object, wherein the first instruction text includes task information for at least one exploration task of the target object, and the task information includes at least: task quantity and exploration object; it analyzes the first instruction text using an intent recognition model to obtain the target object's task intent; when the task intent is to explore multimodal data, it analyzes the task information of each exploration task within the first instruction text using a pre-trained task planning and decision model to obtain the target task execution strategy with the lowest execution cost, wherein the task planning and decision model is trained using a reinforcement learning algorithm; and it sequentially calls the corresponding model context protocol service agent to execute the corresponding exploration task according to the target task execution strategy, and obtains the exploration results fed back by each model context protocol service agent.

[0127] As an alternative implementation, the above-mentioned electronic device may exist in the form of a mobile terminal, a computer terminal, or a similar computing device. Figure 4 A hardware block diagram of a computer terminal for implementing a multimodal data exploration method is shown. Figure 4 As shown, the computer terminal 40 may include one or more processors 402 (shown as 402a, 402b, ..., 402n in the figure) 402 (processor 402 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 404 for storing data, and a transmission device 406 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 4 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, computer terminal 40 may also include... Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown.

[0128] It should be noted that the aforementioned one or more processors 402 and / or other data processing circuitry are generally referred to herein as "data processing circuitry". This data processing circuitry may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 40. As involved in the embodiments of this application, this data processing circuitry serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0129] The memory 404 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the multimodal data probing method in this embodiment. The processor 402 executes various functional applications and data processing by running the software programs and modules stored in the memory 404, thereby implementing the above-mentioned application vulnerability detection method. The memory 404 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 404 may further include memory remotely located relative to the processor 402, and these remote memories can be connected to the computer terminal 40 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0130] The transmission device 406 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 40. In one example, the transmission device 406 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 406 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0131] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 40.

[0132] The sequence numbers of the above embodiments are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0133] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0135] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0136] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0137] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0138] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for multimodal data exploration, characterized in that, include: Obtain the first instruction text input by the target object, wherein the first instruction text includes task information of at least one exploration task of the target object, and the task information includes at least: task quantity and exploration object; The first instruction text is analyzed using an intent recognition model to obtain the task intent of the target object; When the task intent is to explore multimodal data, the task information of each exploration task in the first instruction text is analyzed using a pre-trained task planning and decision model to obtain the target task execution strategy with the lowest execution cost. The task planning and decision model is trained using a reinforcement learning algorithm. According to the target task execution strategy, the corresponding model context protocol service agents are sequentially invoked to execute the corresponding exploration tasks, and the exploration results fed back by each model context protocol service agent are obtained.

2. The method according to claim 1, characterized in that, The training process of the intent recognition model includes: Word embedding technology is used to embed multiple pre-collected second instruction texts to obtain a sentence vector corresponding to each second instruction text, wherein the sentence vector includes word vectors of each word in the corresponding second instruction text; For each sentence vector, the sentence vector is semantically analyzed using a first semantic coding model to obtain the corresponding intent vector, and the word vectors within the sentence vector are semantically analyzed using a second semantic coding model to obtain the character-level semantic vectors corresponding to each word. The annotation vectors corresponding to each word are combined according to the order of the words in the second instruction text to obtain the corresponding sequence annotation vector. The training dataset is determined based on the intent vector and sequence label vector of each sentence vector; The intent recognition model is obtained by iteratively training a preset neural network model using the training dataset.

3. The method according to claim 2, characterized in that, The training dataset is determined based on the intent vector and sequence label vector of each sentence vector, including: For each sentence vector, a first probability distribution corresponding to the intent vector is determined based on the first semantic category of the second instruction text corresponding to the sentence vector, and a second probability distribution of each annotation vector in the sequence annotation vector is determined based on the second semantic category of each word in the second instruction text corresponding to the sentence vector. The training dataset is formed by taking the intent vector and first probability distribution corresponding to each second instruction text, and the second probability distribution of each annotation vector within the sequence annotation vector corresponding to the second instruction text as a set of training sample data.

4. The method according to claim 3, characterized in that, The neural network model includes: a first sub-network model and a second sub-network model, wherein the intent recognition model is obtained by iteratively training the pre-built neural network model using the training dataset, including: For each set of training sample data in the training dataset, the intent vector in the training sample data is analyzed using the first sub-network model to obtain the corresponding first prediction probability distribution, and the sequence label vector in the training sample data is analyzed using the second sub-network model to obtain the corresponding second prediction probability distribution. A first loss function is constructed based on the first probability distribution and the corresponding first predicted probability distribution in the training sample data of each group, and a second loss function is constructed based on the second probability distribution and the corresponding second predicted probability distribution in the training sample data of each group. The first loss function and the second loss function are weighted to obtain the target loss function. The model parameters of the neural network model are then adjusted using gradient descent until the value of the target loss function is less than a preset termination threshold, thus obtaining the trained intent recognition model.

5. The method according to claim 1, characterized in that, The training process of the task planning decision model includes: A value network and a target network for solving task execution strategies are constructed using neural network technology, and the network parameters of the value network and the target network are initialized. Within each iteration cycle, the agent state is initialized, and a preset number of computation cycles are determined. The agent state includes at least: task information for each exploration task and available computational resource information for the target agent. Within each computation cycle, the current agent state is input into the value network to calculate the Q-values ​​corresponding to different task execution strategies. A first task execution strategy with the maximum Q-value is selected based on a greedy strategy, executed, and the corresponding reward is calculated to obtain a new agent state. The current agent state, the target task execution strategy, the reward, and the new agent state are treated as a sample and stored in a preset experience pool. Multiple randomly sampled samples from the experience pool are input into a neural network containing a bidirectional long short-term memory network. The target Q-value corresponding to each sample is calculated, and a target loss function is constructed based on the reward and corresponding target Q-value of each sample. Stochastic gradient descent is used to update the network parameters of the value network to minimize the value of the target loss function. After a preset number of calculation cycles, the network weight parameters of the target network are updated based on the network weight parameters of the value network. After the iteration is completed, the resulting target network is used as the task planning decision model.

6. The method according to claim 5, characterized in that, The current agent state is input into the value network to calculate the Q-values ​​corresponding to different task execution strategies, including: Determine multiple task execution strategies; For each of the task execution strategies, the execution cost corresponding to the task execution strategy is calculated using the current agent state and the task execution strategy, wherein the execution cost includes: the execution time cost and execution resource cost of all the exploration tasks; The Q value corresponding to each task execution strategy is determined based on the execution cost of each task execution strategy, wherein the higher the execution cost of the task execution strategy, the smaller the corresponding Q value.

7. The method according to claim 1, characterized in that, After obtaining the exploration results fed back by each of the model context protocol service agents, the method further includes: The exploration results fed back by each of the model context protocol service agents are converted into the embedding representations of the corresponding modalities, and the embedding representations of each modality are mapped to the same semantic space to obtain the target embedding representations of multiple modalities respectively. Calculate the similarity between the target embedding vectors of each two modalities. If the similarity is higher than a preset threshold, determine the association rule between the exploration results corresponding to the two target embedding vectors based on the similarity and the frequency of exploration results. The exploration results fed back by each of the model context protocol service agents and the association rules between the exploration results are stored in a preset association rule library.

8. A multimodal data exploration device, characterized in that, include: The acquisition module is used to acquire a first instruction text input by the target object, wherein the first instruction text includes task information of at least one exploration task of the target object, and the task information includes at least: task quantity and exploration object; The analysis module is used to analyze the first instruction text using an intent recognition model to obtain the task intent of the target object; The planning module is used to analyze the task information of each exploration task in the first instruction text using a pre-trained task planning decision model when the task intent is to explore multimodal data, and to obtain the target task execution strategy with the lowest execution cost. The task planning decision model is trained using a reinforcement learning algorithm. The execution feedback module is used to sequentially call the corresponding model context protocol service agents to execute the corresponding exploration tasks according to the target task execution strategy, and obtain the exploration results fed back by each of the model context protocol service agents.

9. A computer program product, characterized in that, include: A computer program, wherein when executed by a processor, the computer program implements the multimodal data exploration method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the multimodal data exploration method of any one of claims 1 to 7 through the computer program.

Images