An AI large model agent construction method and system

Abstract

The application relates to the technical field of computers, in particular to an AI large model intelligent agent construction method and system, which comprises the following steps: calling a target API interface by using an initial intelligent agent constructed, obtaining a plurality of environment multi-modal data corresponding to a plurality of interactive behaviors of a user, and the plurality of interactive behaviors occurring in a same session created between the user and the initial intelligent agent; calculating a multi-modal perception coefficient of the initial intelligent agent in environment perception for different modal data types under each interactive behavior according to the plurality of environment multi-modal data corresponding to the plurality of interactive behaviors; calculating a weight coefficient of different modal data types based on the multi-modal perception coefficient; constructing a preloaded training data set based on the weight coefficient, correcting a calling sequence of the target API interface by the initial intelligent agent in environment perception by using the preloaded training data set, and obtaining a final intelligent agent meeting the user demand. The application can improve the accuracy of intelligent agent response.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for constructing an AI large-model intelligent agent. Background Art

[0002] Artificial Intelligence (AI) agents are essentially AI-powered software tools that can perform multi-step tasks with minimal supervision. In addition to natural language processing, AI agents can also make decisions, solve problems, and interact with the environment while performing tasks. The agent itself is a small program that accepts natural language commands, interacts with the scene, and has a preliminary thought process. It can split tasks and has the ability to memorize, plan, call tools, and execute actions. Furthermore, through closed-loop thinking during actions, the agent transforms the knowledge of the larger model into long-term memory and even perception, allowing it to perform specific tasks independently of the larger model.

[0003] In actual use, large-scale AI agents rely on multimodal data for environmental perception. For example, by simultaneously analyzing video images and audio conversations, agents can more accurately understand human behavioral intentions and achieve scenario recognition and reasoning. Environmental perception information is typically real-time or near real-time, requiring agents to respond quickly to environmental changes. However, excessive multimodal information can reduce environmental perception capabilities and make it difficult to ensure the accuracy of the agent's responses. Summary of the Invention

[0004] To address the technical problem that excessive multimodal information can lead to a decrease in environmental perception and an inability to ensure the accuracy of intelligent agent responses, the present invention aims to provide a method and system for constructing an AI large-model intelligent agent. The technical solutions adopted are as follows:

[0005] In a first aspect, the present invention provides a method for constructing an AI large model intelligent agent, comprising:

[0006] Using the constructed initial agent to call a target API interface, obtain multiple environmental multimodal data corresponding to multiple interactive behaviors of the user, wherein the multiple interactive behaviors occur in the same session created between the user and the initial agent;

[0007] Calculating, based on the multiple environmental multimodal data corresponding to the multiple interactive behaviors, the multimodal perception coefficient of the initial agent when perceiving the environment for different modal data types under each interactive behavior;

[0008] Calculating weight coefficients of the different modal data types based on the multimodal perception coefficients, wherein the weight coefficients are used to represent the environmental perception contribution of the initial intelligent agent to the different modal data types;

[0009] A preloaded training data set is constructed based on the weight coefficients, and the preloaded training data set is used to correct and train the calling order of the initial intelligent agent for the target API interface during environment perception, so as to obtain a final intelligent agent that meets user needs.

[0010] Optionally, the constructed initial agent can be used to call the target API interface to obtain multiple environmental multimodal data corresponding to the user's multiple interactive behaviors, including:

[0011] Use the constructed initial intelligent agent to call the neural network to analyze whether each interactive behavior input by the user is effective;

[0012] If the interactive behavior is determined to be valid, the target API interface is called using the initial intelligent agent to obtain multiple environmental multimodal data corresponding to the user's multiple interactive behaviors.

[0013] Optionally, based on the multiple environmental multimodal data corresponding to the multiple interactive behaviors, calculating the multimodal perception coefficient of the initial agent when perceiving the environment for different modal data types under each interactive behavior includes:

[0014] Calculating the user demand bias coefficient for each interactive behavior based on the multiple environmental multimodal data corresponding to the multiple interactive behaviors;

[0015] Based on the environmental multimodal data and the user demand bias coefficient, the multimodal perception coefficient of the initial intelligent agent when perceiving the environment for different modal data types under each interactive behavior is calculated.

[0016] Optionally, the calculating of the user demand bias coefficient for each interactive behavior based on the multiple environmental multimodal data corresponding to the multiple interactive behaviors includes:

[0017] Determine each interactive behavior in the multiple interactive behaviors in sequence as a current interactive behavior, and determine at least one historical interactive behavior corresponding to the current interactive behavior in the multiple interactive behaviors;

[0018] Based on the environmental multimodal data corresponding to the at least one historical interactive behavior, the calculation process of the user demand bias coefficient for the current interactive behavior is repeatedly executed until the user demand bias coefficient for each interactive behavior in the multiple interactive behaviors is obtained.

[0019] Optionally, the process of calculating the user demand bias coefficient for the current interactive behavior includes:

[0020] Performing input demand analysis for each of the historical interactive behaviors to determine the required quantity of modal data types corresponding to each of the historical interactive behaviors;

[0021] determine, based on the environment multi-modal data corresponding to the at least one historical interaction behavior, a number of perceptions of the initial agent for the modal data type in each of the historical interaction behaviors;

[0022] calculate, based on the demand number of the modal data type corresponding to each of the historical interaction behaviors and the number of perceptions of the initial agent for the modal data type in each of the historical interaction behaviors, a user demand bias coefficient in the current interaction behavior.

[0023] Optionally, based on the environment multi-modal data and the user demand bias coefficient, a multi-modal perception coefficient of the initial agent for environment perception of different modal data types in each interaction behavior is calculated, including:

[0024] quantify the environment multi-modal data into an environment perception vector of the initial agent for different modal data types according to an acquisition sequence of different modal data types in the environment multi-modal data corresponding to each interaction behavior;

[0025] based on the environment perception vector and the user demand bias coefficient, a multi-modal perception coefficient of the initial agent for environment perception of different modal data types in each interaction behavior is calculated.

[0026] Optionally, based on the environment perception vector and the user demand bias coefficient, a multi-modal perception coefficient of the initial agent for environment perception of different modal data types in each interaction behavior is calculated, including:

[0027] each of the multiple interaction behaviors is sequentially determined as a current interaction behavior, and a previous interaction behavior corresponding to the current interaction behavior is determined, and the calculation process of the multi-modal perception coefficient for the current interaction behavior is repeatedly executed until the multi-modal perception coefficient in each of the multiple interaction behaviors is obtained;

[0028] wherein the calculation process of the multi-modal perception coefficient includes:

[0029] determine a first environment perception vector and a first user demand bias coefficient corresponding to the current interaction behavior, and a second environment perception vector and a second user demand bias coefficient corresponding to the previous interaction behavior;

[0030] based on the first environment perception vector, the first user demand bias coefficient, the second environment perception vector and the second user demand bias coefficient, a multi-modal perception coefficient of the initial agent for environment perception of different modal data types in the current interaction behavior is calculated.

[0031] Optionally, calculating the weight coefficients of the different modal data types based on the multimodal perception coefficients includes:

[0032] Acquire multiple multimodal perception coefficients when the initial agent perceives the environment for each modal data type under the multiple interactive behaviors;

[0033] A weight coefficient corresponding to the modal data type is calculated based on the multiple multimodal perception coefficients.

[0034] Optionally, the method further includes:

[0035] Integrate API interfaces from different content providers and service providers, the API interfaces are used to support calling external service resources to obtain environmental multimodal data under the different modal data types;

[0036] Create an interface calling relationship between the initial agent and each of the API interfaces.

[0037] In the second aspect, an embodiment of the present invention also provides an AI large-model intelligent agent construction system, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the computer program implements the steps of any one of the above-mentioned methods when executed by the processor.

[0038] The present invention has the following beneficial effects: Through the technical solution provided by the present invention, an initial intelligent agent is first used to obtain multiple environmental multimodal data corresponding to multiple user interactions; then, based on the multiple environmental multimodal data corresponding to the multiple interactions, the multimodal perception coefficient of the initial intelligent agent when performing environmental perception for different modal data types under each interaction is calculated; further, based on the multimodal perception coefficient, weight coefficients of different modal data types are calculated; finally, a preloaded training data set is constructed based on the weight coefficients, and the preloaded training data set is used to correct the call order of the target API interface when training the initial intelligent agent during environmental perception, thereby obtaining a final intelligent agent that meets user needs. Through the calculation of the multimodal perception coefficient, the present invention can accurately quantify the intelligent agent's perception ability of different environmental multimodal data and clarify the importance and sensitivity of different environmental multimodal data in meeting user needs. Further, based on the multimodal perception coefficient, the weight coefficients of different modal data types are calculated. The weight coefficients are used to optimize the calling order of the target API interface when training the intelligent agent during environmental perception. This can enable the final intelligent agent to improve its environmental perception ability, more efficiently obtain multimodal information that is valuable to user needs, reduce unnecessary information acquisition, and thus improve the speed and accuracy of the intelligent agent's response.

[0039] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application, as claimed. Other features and advantages of the present application will become apparent in the course of the following DETAILED DESCRIPTION, which describes the embodiments of the application in greater detail. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings required to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0041] Figure 1 A flowchart of an AI large model agent construction method provided by an embodiment of the present application;

[0042] Figure 2 A flowchart of an AI large model agent construction method provided by another embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined purpose, the specific implementation, structure, features and effects of the AI large model agent construction method and system according to the present application are described in detail below in combination with the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0045] The specific scheme of the AI large model agent construction method and system provided by the present application is specifically described below in combination with the drawings.

[0046] Please refer to Figure 1 which shows a method flowchart of an AI large model agent construction method provided by an embodiment of the present application. The method comprises the following steps:

[0047] Step 110, calling the target API interface by using the constructed initial agent to obtain a plurality of environment multi-modal data corresponding to a plurality of interactive behaviors of a user, the plurality of interactive behaviors occurring in the same session created between the user and the initial agent.

[0048] Among them, environmental multimodal data includes multiple modal data in the interactive environment, such as text data generated when users enter questions in text form, voice data when asking voice questions, image data when sending pictures, etc.

[0049] During the construction of a large-scale AI model agent, an initial agent is first established. This initial agent has the ability to call relevant application programming interfaces (APIs), which act as a channel connecting the agent to various external information resources. A session is established between the user and this initial agent. A session can be understood as an interactive process, during which the user and the initial agent continuously communicate and interact. During a single session, the user may engage in multiple interactions, such as initially asking a question through text input, then sending a voice command, or even uploading an image. Each interaction generates different types of data, encompassing multimodal information from the environment, such as semantic information from text, audio information from speech, and visual information from images. The initial agent can collect the multimodal environmental data corresponding to these multiple interactions generated within the same session by calling the API interfaces associated with these interactions (i.e., target API interfaces). The API interfaces supported by the initial agent can be those of content providers (CPs) or service providers (SPs). These API interfaces are key external resources for building large-scale AI model agents. They enable agents to access and utilize various online services and information, including hardware devices in the actual environment to obtain video, audio and other information, so that they can understand and respond to user input.

[0050] In a specific application scenario, before using the initial intelligent agent to call the target API interface, it is also necessary to create an interface calling relationship between the initial intelligent agent and each API interface. Accordingly, the implementation steps may also include: integrating API interfaces from different content providers and service providers, and the API interface is used to support calling external service resources to obtain environmental multimodal data under different modal data types; creating an interface calling relationship between the initial intelligent agent and each API interface.

[0051] Step 120: Calculate the multimodal perception coefficient of the initial agent when perceiving the environment for different modal data types under each interactive behavior based on the multiple environmental multimodal data corresponding to the multiple interactive behaviors.

[0052] During the initial agent's operation, users interact with it multiple times. Each interaction generates multimodal environmental data, encompassing text, voice, images, and other types. This is known as multimodal environmental data. Based on this rich multimodal environmental data, the initial agent's environmental perception capabilities for each interaction, across different modal data types (e.g., text, voice, and image), are quantitatively evaluated. The result of this quantitative evaluation is the multimodal perception coefficient. Environmental perception refers to the process by which the initial agent understands and analyzes diverse multimodal environmental data to gain information about user needs and the surrounding environment. For example, it can understand the user's intended question by analyzing text and voice input; it can also identify relevant objects or scenes from images to assist in determining environmental conditions.

[0053] When calculating the multimodal perception coefficient, multiple factors are considered. These include the richness of the modal data types included in the user's input behavior and the number of modal data types perceived by the initial agent after each interaction. Through a specific calculation method, this complex information is converted into a specific numerical value, which measures the initial agent's comprehensive ability to perceive the environment across different modal data during each interaction. This coefficient is crucial for optimizing the initial agent's environmental perception capabilities. It can subsequently be used to adjust the initial agent's processing strategy for different modal data, improving the agent's performance and the accuracy of its response to user needs.

[0054] Step 130: Calculate weight coefficients of different modal data types based on the multimodal perception coefficients.

[0055] In specific application scenarios, the multimodal perception coefficient reflects the initial agent's environmental perception of different modal data types during each interaction. By performing specific calculations on the multimodal perception coefficient, we can derive the weight coefficients corresponding to different modal data types. This calculation takes into account various factors, such as the changing trend of the perception coefficient of a particular modal data type over multiple interactions and the varying importance of different modal data in meeting user needs.

[0056] The weight coefficient is used to characterize the initial agent's contribution to environmental perception for different modal data types. In practical applications, different modal data contributes differently to the initial agent's understanding of user needs and accurate response. Modal data types with higher weight coefficients mean that the initial agent will prioritize and pay attention to such data when performing environmental perception, placing it at a higher priority in the perception order; while modal data types with lower weight coefficients will have a relatively lower priority when processed by the initial agent. In this way, the initial agent can reasonably allocate resources and attention according to the weight coefficient, process environmental multimodal data more efficiently, improve the accuracy and efficiency of environmental perception, and better meet user needs.

[0057] Step 140: construct a preloaded training data set based on the weight coefficient, and use the preloaded training data set to correct the calling order of the target API interface of the initial intelligent agent during environmental perception, so as to obtain the final intelligent agent that meets the user's needs.

[0058] In the disclosed embodiments, various types of multimodal data can be organized and combined according to corresponding proportions and rules based on weight coefficients to construct a preloaded training dataset. For example, if the weight coefficient of text modal data is higher, the proportion of text-related training data in the dataset may increase accordingly. The dataset constructed in this way contains various data samples that match the weights of different modal data, providing targeted material for subsequent training of the initial intelligent agent. The initial intelligent agent's calls to the target API interface may not fully consider the differences in the importance of multimodal data for different user needs. When training the initial intelligent agent using the preloaded training dataset, the initial intelligent agent can learn the relationship between different modal data and user needs. Through continuous training, the initial intelligent agent adjusts the order of its calls to the target API interfaces during environmental perception based on the weight information reflected in the dataset. For example, if the dataset shows that voice modal data is more critical to a certain type of user need, after training, the initial intelligent agent will prioritize calling API interfaces that can obtain voice-related information, rather than randomly or indiscriminately calling interfaces as before.

[0059] By refining and training the initial agent using a preloaded training dataset, it becomes more accurate and efficient in environmental perception and responding to user requests. Because the initial agent learns to appropriately call target APIs based on weighted coefficients and obtain the most valuable multimodal information for user needs, it can more accurately understand user intent when processing user input and provide responses that better meet user expectations. This optimized training ultimately becomes the final agent that meets user needs, improving both the user experience and the agent's practicality.

[0060] In summary, according to the AI large model agent construction method provided by the present application, the initial agent is used to obtain a plurality of environment multi-modal data corresponding to a plurality of interactive behaviors of a user; then, according to the plurality of environment multi-modal data corresponding to the plurality of interactive behaviors, a multi-modal perception coefficient of the initial agent in environment perception for different modal data types under each interactive behavior is calculated; further, a weight coefficient of different modal data types is calculated based on the multi-modal perception coefficient; finally, a preloaded training data set is constructed based on the weight coefficient, and the preloaded training data set is used to correct the calling sequence of the initial agent for the target API interface in environment perception, so as to obtain a final agent meeting the user demand. Through the calculation of the multi-modal perception coefficient, the present application can accurately quantify the perception ability of the initial agent for different modal data, and clearly determine the importance and sensitivity of different modal data in meeting the user demand. Further, the weight coefficient of different modal data types is calculated based on the multi-modal perception coefficient, and the weight coefficient is used to optimize the calling sequence of the initial agent for the target API interface in environment perception, so that the final agent can improve the environment perception ability, more efficiently obtain multi-modal information valuable for the user demand, reduce unnecessary information acquisition, and further improve the speed and accuracy of the agent response.

[0061] Based on Figure 1 As a refinement and extension of the above-mentioned embodiment, in order to fully describe the specific implementation process of the present embodiment method, the present embodiment provides a specific method as shown in Figure 2 As shown in Figure 2 Based on Figure 1 As shown in Figure 2 The method comprises the following steps:

[0062] Step 210, using the constructed initial agent to call the target API interface to obtain a plurality of environment multi-modal data corresponding to a plurality of interactive behaviors of a user, the plurality of interactive behaviors occurring in the same session created between the user and the initial agent.

[0063] For the present embodiment, the step 210 of using the constructed initial agent to call the target API interface to obtain a plurality of environment multi-modal data corresponding to a plurality of interactive behaviors of a user can comprise the following steps:

[0064] Step 210-1, using the constructed initial agent to call a neural network to analyze whether each interactive behavior input by the user is valid.

[0065] In the disclosed embodiments, when the initial agent invokes the neural network, it can input each interactive behavior entered by the user as input data into the neural network. The neural network analyzes this input data to determine whether the interactive behavior is valid. For example, if the user inputs meaningless garbled characters, repetitive and worthless content, or information completely irrelevant to the interactive scenario set by the initial agent, the neural network may determine that the interactive behavior is invalid. However, if the user inputs content that meets the functional positioning of the agent and can guide the initial agent to perform effective information processing, such as clear questions or reasonable instructions, it will be determined to be a valid interactive behavior.

[0066] Step 210 - 2 : If the interactive behavior is determined to be valid, the target API interface is called by the initial agent to obtain multiple environmental multimodal data corresponding to the user's multiple interactive behaviors.

[0067] Once the neural network determines that the user's interaction is valid, the initial agent executes the next step, calling the target API interface. The API interface serves as a bridge for the agent to interact with the external environment, connecting to a variety of data sources and services. By calling these API interfaces, the initial agent can obtain multiple environmental multimodal data corresponding to multiple user interactions. Environmental multimodal data covers a variety of forms, such as text data generated when users enter questions in text form, voice data when asking questions, and image data when sending pictures. This data contains rich environmental information.

[0068] Step 220: Calculate the user demand bias coefficient for each interactive behavior based on the multiple environmental multimodal data corresponding to the multiple interactive behaviors.

[0069] In the embodiment of the present disclosure, calculating the user demand bias coefficient for each interactive behavior based on the multiple environmental multimodal data corresponding to the multiple interactive behaviors in step 220 may include the following steps:

[0070] Step 220 - 1 : sequentially determine each interactive behavior among the multiple interactive behaviors as a current interactive behavior, and determine at least one historical interactive behavior corresponding to the current interactive behavior among the multiple interactive behaviors.

[0071] Step 220-2: Based on the environmental multimodal data corresponding to at least one historical interactive behavior, repeatedly perform the calculation process of the user demand bias coefficient for the current interactive behavior until the user demand bias coefficient for each interactive behavior in multiple interactive behaviors is obtained.

[0072] The calculation process of the user demand bias coefficient for the current interactive behavior includes the following steps:

[0073] Step 220 - 2 - 1 : Perform input demand analysis for each historical interactive behavior to determine the required quantity of modal data types corresponding to each historical interactive behavior.

[0074] Each time a user interacts with the initial agent, such as entering text, issuing voice commands, or uploading images, the initial agent conducts a thorough analysis of these inputs. This analysis aims to determine the user's intended purpose through this input, whether it's a request for information, a request to perform a task, or emotional communication. Based on this analysis of user input needs, the initial agent determines the required number of modal data types involved in each historical interaction. Multimodal data includes text, voice, images, and videos. For example, when a user asks, "What movies are good at the nearest theater today?" this historical interaction may require not only text data (to understand the question), but also image data (to display movie posters) and voice data (to play sound clips of movie trailers) to more fully meet the user's needs. Here, text, image, and voice represent three different modal data types, totaling three.

[0075] Step 220-2-2: Based on the environmental multimodal data corresponding to at least one historical interactive behavior, determine the number of perceptions of the initial intelligent agent for the modal data type in each historical interactive behavior.

[0076] In specific application scenarios, the initial agent needs to analyze and process the multimodal environmental data received during each historical interaction. The number of perceptions refers to the number of modal data types that the agent can recognize and understand. For example, in a historical interaction, a user sent a text message, an image, and shared location information (a sensor data modality). If the initial agent can recognize and process these three different types of data, then the number of perceptions of the initial agent for each modal data type during this historical interaction is 3. This number reflects the initial agent's ability to capture and process different types of environmental information during a specific interaction.

[0077] Step 220-2-3: Calculate the first user demand bias coefficient under the current interactive behavior based on the number of demands for the modal data type corresponding to each historical interactive behavior and the number of perceptions of the initial agent for the modal data type in each historical interactive behavior.

[0078] Among them, the user demand bias coefficient is used to measure the user's demand bias during the interaction process. The first user demand bias coefficient is the user demand bias coefficient, which is used to distinguish the user demand bias coefficient of the current interactive behavior from the previous interactive behavior. It comprehensively considers the number of demands and the number of perceptions, and can reflect the relationship between the user's input demand and the actual perception ability of the initial intelligent agent. This embodiment does not analyze the case where the number of demands and the number of perceptions are zero, so under normal circumstances, the number of demands and the number of perceptions will not be zero. The closer the match between the number of demands and the number of perceptions, and the more orderly the correspondence between the overall feedback and the type of environmental perception information, the higher the degree of understanding and satisfaction of the user's needs by the initial intelligent agent, and the larger the user demand bias coefficient; conversely, if the difference between the two is large, the correspondence is chaotic, and the coefficient is smaller.

[0079] For the embodiment of the present disclosure, the number of demands for the modal data type corresponding to each historical interactive behavior and the number of perceptions of the initial intelligent agent for the modal data type in each historical interactive behavior can be substituted into the user demand bias coefficient calculation formula to obtain the user demand bias coefficient under each historical interactive behavior. Through this formula, the number of demands and the number of perceptions in multiple historical interactive behaviors are quantified and calculated, and the obtained user demand bias coefficient can accurately reflect the user demand bias characteristics, providing a basis for subsequent adjustment of the intelligent agent perception method, so as to achieve efficient and accurate construction of the initial intelligent agent. Among them, the formula characteristics of the user demand bias coefficient calculation formula are described as follows:

[0080]

[0081] Where, Indicates the user's The user demand bias coefficient under the interaction behavior (i.e. the current interaction behavior); Indicates the first Interaction behavior; j represents the The jth historical interaction behavior before the jth interaction behavior; Indicates the The number of historical interactive behaviors included in each interactive behavior; [ ] is a natural constant An exponential function with base ; Indicates the user's The number of requests for modal data types corresponding to the j-th historical interaction behavior before the j-th interaction behavior; Indicates that the initial agent is in The number of perceptions of modal data types in the jth historical interaction behavior before the jth interaction behavior; Indicates the In the jth historical interaction behavior before the jth interaction behavior, the number of input and output categories is consistent; Indicates the The maximum value between the number of demands and the number of perceptions of the modal data type in the jth historical interaction behavior before the jth interaction behavior.

[0082] Step 230: Based on the environmental multimodal data and the user demand bias coefficient, calculate the multimodal perception coefficient of the initial intelligent agent when perceiving the environment for different modal data types under each interactive behavior.

[0083] The multimodal perception coefficient measures the initial agent's comprehensive ability to perceive the environment across different modal data types during each interaction. By calculating this coefficient, the initial agent can understand its own perception of different modal data, determine which modal data is most critical to meeting user needs in the current interaction, and which modal data perception capabilities need improvement. This helps the initial agent optimize its perception strategy and improve the accuracy and efficiency of its response to user needs.

[0084] In the embodiment of the present disclosure, the calculation of the multimodal perception coefficient of the initial agent for each interaction behavior based on the multimodal environmental data and the user demand bias coefficient in step 230 may include the following steps:

[0085] Step 230-1: quantify the environmental multimodal data into environmental perception vectors of the initial intelligent agent for different modal data types according to the order of obtaining different modal data types in the environmental multimodal data corresponding to each interactive behavior.

[0086] Environmental multimodal data includes various modal data types, including text, voice, images, and sensor data (such as temperature and humidity). When a user interacts with the initial agent, data from different modalities is acquired by the initial agent in a specific order. For example, if a user sends a voice command and then uploads an image, the voice data is acquired first, followed by the image data. This acquisition order contains a wealth of information, potentially reflecting the focus of user needs or the logical relationships within information transmission. Quantization is the process of converting non-numerical information into numerical form for computer processing and analysis. For environmental multimodal data, quantization involves converting the data into numerical representations based on the order in which the different modal data types were acquired. This helps the agent understand and process this data more accurately.

[0087] The environmental perception vector is formed based on the quantified environmental multimodal data. It describes the initial agent's perception of the environment under different modal data types in the form of a vector. Similarly, the first environmental perception vector and the second environmental perception vector mentioned below are environmental perception vectors, which are also used to distinguish the environmental perception vectors of the current interactive behavior and the previous interactive behavior. After quantization according to the acquisition order of different modal data types, each modal data type corresponds to an environmental perception vector, which means that for each different type of modal data, the initial agent will create a special vector to describe the perception of that type of data. Through the environmental perception vector, the initial agent can analyze and compare environmental information more intuitively, understand the differences and characteristics of environmental information under different interactive behaviors, and provide strong support for subsequent decision-making and response.

[0088] Step 230-2: Based on the environmental perception vector and the user demand bias coefficient, calculate the multimodal perception coefficient of the initial agent when perceiving the environment for different modal data types under each interactive behavior.

[0089] For the disclosed embodiment, the environmental perception vector and the user demand bias coefficient can be substituted into the multimodal perception coefficient calculation formula to calculate the multimodal perception coefficient of the initial intelligent agent when performing environmental perception for different modal data types. The larger this coefficient is, the more sensitive the user demand bias is in the intelligent agent's environmental perception implementation process in the current interaction, and the more important the corresponding modal data is in meeting user needs; vice versa. Through such a calculation method, the environmental perception vector and the user demand bias coefficient are organically combined, providing a quantitative basis for the initial intelligent agent to accurately evaluate its own perception ability of different modal data, thereby laying the foundation for optimizing the initial intelligent agent's environmental perception method and improving performance.

[0090] Among them, the formula feature description of the multimodal perception coefficient calculation formula is:

[0091]

[0092] Where, Indicates the user's Under the interaction behavior, the initial agent targets the modal data type Multimodal perception coefficients when performing environmental perception; represents the maximum and minimum normalization function; Indicates the first Interaction behavior; Indicates the user's The first user demand bias coefficient under the interaction behavior; Indicates the user's The second user demand bias coefficient under the previous interactive behavior corresponding to the interactive behavior; Indicates the user's The first environmental perception vector of the initial agent for modal data type a under the interaction behavior; Indicates the user's The second environmental perception vector of the initial agent for modal data type a under the previous interaction behavior corresponding to the second interaction behavior; It represents the influence of the initial agent on the multimodal type order in the environmental perception condition. The greater the influence, the higher the sensitivity of the user demand preference in the environmental perception implementation process of the initial agent. If an extreme case occurs and makes it zero, then add a non-zero constant after its position, such as 0.01, that is, .

[0093] Accordingly, for the embodiments of the present disclosure, the embodiment steps may include: determining each interactive behavior in multiple interactive behaviors as the current interactive behavior in turn, and determining the previous interactive behavior corresponding to the current interactive behavior, and repeatedly executing the following calculation process of the multimodal perception coefficient for the current interactive behavior until the multimodal perception coefficient for each interactive behavior in the multiple interactive behaviors is obtained; wherein, the calculation process of the multimodal perception coefficient includes: determining the first environmental perception vector and the first user demand bias coefficient corresponding to the current interactive behavior, and the second environmental perception vector and the second user demand bias coefficient corresponding to the previous interactive behavior; based on the first environmental perception vector, the first user demand bias coefficient, the second environmental perception vector and the second user demand bias coefficient, calculating the multimodal perception coefficient of the initial intelligent agent when performing environmental perception for different modal data types under the current interactive behavior.

[0094] Step 240: Calculate weight coefficients of different modal data types based on the multimodal perception coefficients.

[0095] In the embodiment of the present disclosure, calculating weight coefficients of different modal data types based on the multimodal perception coefficients in step 240 may include the following steps:

[0096] Step 240 - 1 : Obtain multiple multimodal perception coefficients when the initial agent perceives the environment for each modal data type under multiple interactive behaviors.

[0097] During multiple interactions, the initial agent calculates the corresponding multimodal perception coefficient for each modal data type. For example, in one interaction, the user first enters text and then sends a voice message. The initial agent will calculate the multimodal perception coefficients for the text modality and voice modality in this interaction respectively. In the next interaction, if image data is involved, the initial agent will also calculate the multimodal perception coefficient for the image modality. By obtaining multiple multimodal perception coefficients for each modal data type under multiple interactions, the initial agent can more comprehensively understand the impact of different modal data on its own environmental perception capabilities in different scenarios, thereby optimizing its own processing strategy for multimodal data and improving the quality and efficiency of interaction with users.

[0098] Step 240 - 2 : Calculate weight coefficients corresponding to modal data types based on multiple multimodal perception coefficients.

[0099] Among them, the weight coefficient is used to characterize the contribution of the initial intelligent agent to environmental perception for different modal data types, which can be used to determine the priority of the initial intelligent agent when performing environmental perception for different modal data types. Different modal data play different roles in meeting user needs and realizing intelligent agent functions. By calculating the weight coefficient, the relative importance of each modal data in the intelligent agent's decision-making and information processing process can be clarified. For example, if the weight coefficient of a certain modal data type is high, it means that the initial intelligent agent should give priority to this modal data when processing information and responding to user needs; conversely, modal data with a low weight coefficient has a relatively low priority in processing. This can help the initial intelligent agent reasonably allocate computing resources and attention, improve overall operating efficiency and the accuracy of responding to user needs.

[0100] In the embodiment of the present disclosure, the weight coefficient of each modal data type can be calculated by substituting the multiple multimodal perception coefficients of the initial intelligent agent when perceiving the environment into the weight coefficient calculation formula. The weight coefficient calculation formula is characterized as follows:

[0101]

[0102] Where, Represents a modal data type The weight coefficient of Indicates the number of times the user sends interactive actions to the initial agent in the same session; Indicates the first Interaction behavior; represents the linear normalization function; Indicates the user's Under the interaction behavior, the initial agent targets the modal data type Multimodal perception coefficients for environmental perception.

[0103] Step 250: construct a preloaded training data set based on the weight coefficient, and use the preloaded training data set to correct the calling order of the target API interface of the initial intelligent agent during environmental perception to obtain the final intelligent agent that meets user needs.

[0104] For the specific implementation process of the embodiment of the present disclosure, please refer to the relevant description in step 140 of the embodiment, which will not be repeated here.

[0105] In summary, the technical solution in this application calculates the multimodal perception coefficient of the initial intelligent agent when perceiving the environment for different modal data types under each interactive behavior, based on the multiple environmental multimodal data corresponding to the multiple interactive behaviors obtained. This coefficient comprehensively considers the user demand bias coefficient and the environmental perception vector. The user demand bias coefficient is obtained by analyzing factors such as the number of modal data types included in the user's multiple input demands, the number of modal data types perceived by the intelligent agent, and so on, reflecting the bias of the user's demand. The environmental perception vector quantifies the environmental multimodal perception results according to the order in which the data types are obtained, and the size of the difference reflects the strength of the environmental association. By calculating the multimodal perception coefficient, the initial intelligent agent's perception ability of different modal data can be accurately quantified, and the importance and sensitivity of different modal data types in the process of satisfying user needs can be clarified. By calculating the multimodal perception coefficient, the initial intelligent agent's perception ability of different environmental multimodal data can be accurately quantified, and the importance and sensitivity of different environmental multimodal data in the process of satisfying user needs can be clarified. Further, based on the multimodal perception coefficient, the weight coefficients of different modal data types are calculated. The weight coefficients are used to optimize the calling order of the target API interface when training the intelligent agent during environmental perception. This can enable the final intelligent agent to improve its environmental perception ability, more efficiently obtain multimodal information that is valuable to user needs, reduce unnecessary information acquisition, and thus improve the speed and accuracy of the intelligent agent's response.

[0106] Based on the same inventive concept as the above method, an embodiment of the present invention also provides an AI large model intelligent agent construction system, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-mentioned AI large model intelligent agent construction methods.

[0107] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. Furthermore, the foregoing descriptions of specific embodiments of this specification are provided. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0108] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0109] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for constructing an AI large model intelligent agent, characterized in that: The method comprises: Using the constructed initial agent to call a target API interface, obtain multiple environmental multimodal data corresponding to multiple interactive behaviors of the user, wherein the multiple interactive behaviors occur in the same session created between the user and the initial agent; Determine each interactive behavior in the multiple interactive behaviors in sequence as a current interactive behavior, and determine at least one historical interactive behavior corresponding to the current interactive behavior in the multiple interactive behaviors; Based on the environmental multimodal data corresponding to at least one historical interactive behavior, the calculation process of the user demand bias coefficient for the current interactive behavior is repeatedly performed until the user demand bias coefficient for each interactive behavior in multiple interactive behaviors is obtained; wherein the user demand bias coefficient for the current interactive behavior is calculated based on the number of demands for the modal data type corresponding to each historical interactive behavior and the number of perceptions of the initial intelligent agent for the modal data type in each historical interactive behavior, the number of demands is determined by performing input demand analysis on each historical interactive behavior, and the number of perceptions is determined based on the environmental multimodal data corresponding to at least one historical interactive behavior; Based on the multimodal environmental data and the user demand bias coefficient, the multimodal perception coefficient of the initial agent when perceiving the environment for different modal data types is calculated under each interaction behavior. Calculating weight coefficients of the different modal data types based on the multimodal perception coefficients, wherein the weight coefficients are used to represent the environmental perception contribution of the initial intelligent agent to the different modal data types; A preloaded training data set is constructed based on the weight coefficients, and the preloaded training data set is used to correct and train the calling order of the initial intelligent agent for the target API interface during environment perception, so as to obtain a final intelligent agent that meets user needs.

2. The AI ​​large model agent construction method according to claim 1 is characterized in that: The constructed initial agent calls the target API interface to obtain multiple environmental multimodal data corresponding to the user's multiple interactive behaviors, including: Use the constructed initial intelligent agent to call the neural network to analyze whether each interactive behavior input by the user is effective; If the interactive behavior is determined to be valid, the target API interface is called using the initial intelligent agent to obtain multiple environmental multimodal data corresponding to the user's multiple interactive behaviors.

3. The AI ​​large model agent construction method according to claim 1 is characterized in that: Based on the environmental multimodal data and the user demand bias coefficient, the multimodal perception coefficient of the initial agent when perceiving the environment for different modal data types under each interactive behavior is calculated, including: quantizing the environmental multimodal data into environmental perception vectors of the initial intelligent agent for different modal data types according to the order of obtaining different modal data types in the environmental multimodal data corresponding to each interactive behavior; Based on the environmental perception vector and the user demand bias coefficient, the multimodal perception coefficient of the initial agent when performing environmental perception for different modal data types under each interactive behavior is calculated.

4. The AI ​​large model agent construction method according to claim 3 is characterized in that: Based on the environmental perception vector and the user demand bias coefficient, a multimodal perception coefficient of the initial agent when performing environmental perception for different modal data types under each interactive behavior is calculated, including: Determining each interactive behavior in the multiple interactive behaviors as a current interactive behavior in sequence, and determining a previous interactive behavior corresponding to the current interactive behavior, and repeatedly performing the following calculation process for the multimodal perception coefficient of the current interactive behavior until the multimodal perception coefficient of each interactive behavior in the multiple interactive behaviors is obtained; The calculation process of the multimodal perception coefficient includes: Determining a first environment perception vector and a first user demand bias coefficient corresponding to the current interactive behavior, and a second environment perception vector and a second user demand bias coefficient corresponding to the previous interactive behavior; Based on the first environmental perception vector, the first user demand bias coefficient, the second environmental perception vector and the second user demand bias coefficient, the multimodal perception coefficient of the initial intelligent agent when performing environmental perception for different modal data types under the current interactive behavior is calculated.

5. The AI ​​large model agent construction method according to claim 1 is characterized in that: Calculating the weight coefficients of the different modal data types based on the multimodal perception coefficients includes: Acquire multiple multimodal perception coefficients when the initial agent perceives the environment for each modal data type under the multiple interactive behaviors; A weight coefficient corresponding to the modal data type is calculated based on the multiple multimodal perception coefficients.

6. The AI ​​large model agent construction method according to claim 1, characterized in that: The method further comprises: Integrate API interfaces from different content providers and service providers, the API interfaces are used to support calling external service resources to obtain environmental multimodal data under the different modal data types; Create an interface calling relationship between the initial agent and each of the API interfaces.

7. An AI large model intelligent agent construction system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of the method for constructing an AI large model intelligent agent are implemented as described in any one of claims 1 to 6.

Images