Intelligent agent adaptive decision-making method and device based on multimodal semantic alignment

Abstract

The present application discloses an intelligent agent adaptive decision-making method and device based on multimodal semantic alignment, the method comprising: collecting voice information, visual information and tactile information through a multimodal sensor array for timestamp synchronization to obtain time-aligned multimodal data; performing fine-grained semantic alignment on the multimodal data according to a preset hierarchical attention mechanism to obtain a unified semantic representation, the preset hierarchical attention mechanism includes an intra-modal attention layer and a cross-modal attention layer, the intra-modal attention layer is used to capture the dependency relationship of the multimodal data and extract the key features within each modal data based on the dependency relationship, and the cross-modal attention layer is used to perform semantic alignment and semantic fusion on each modal data based on the key features; generating a decision result based on the unified semantic representation and a pre-trained generative adversarial network. Therefore, by adopting the embodiments of the present application, the intelligent agent will not have misunderstandings or erroneous operations when performing tasks, thereby improving the accuracy of the intelligent agent's decision-making.

Description

Technical Field

[0001] The present application relates to the field of embodied intelligence technology, and in particular to an intelligent agent adaptive decision-making method and device based on multimodal semantic alignment. Background Art

[0002] In the field of modern intelligent services, embodied intelligent agents (such as service robots and intelligent assistants) are widely used in a variety of complex environments, including homes, commercial spaces, and healthcare. These agents need to interact with humans naturally and efficiently while making accurate decisions based on real-time environmental information.

[0003] Currently, embodied agents typically employ multimodal interaction technologies to interact with their environments and users. These technologies primarily involve the acquisition and processing of data from modalities such as speech recognition, computer vision, and tactile perception. After acquisition, these modal data are typically integrated through simple fusion methods (such as weighted averaging or feature concatenation) to support the agent's decision-making. Some also employ large multimodal models such as LMMs.

[0004] However, due to the differences in feature space and semantic expression of different modal data, especially the frequent semantic ambiguity and modal conflicts in human-machine collaboration scenarios, intelligent agents may make misunderstandings or incorrect operations when performing tasks, and there is no more appropriate and accurate intelligent agent decision-making solution. Summary of the Invention

[0005] The embodiments of this application provide a method and apparatus for intelligent agent adaptive decision-making based on multimodal semantic alignment. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is provided below. This summary is not intended to be a comprehensive review, identify key or important elements, or delineate the scope of protection for these embodiments. Its sole purpose is to present some concepts in a simplified form, serving as a prelude to the detailed description that follows.

[0006] In a first aspect, an embodiment of the present application provides an agent adaptive decision-making method based on multimodal semantic alignment, the method comprising:

[0007] The multimodal sensor array pre-deployed on the intelligent body collects voice information, visual information and tactile information and synchronizes the timestamps to obtain time-aligned multimodal data;

[0008] According to the preset hierarchical attention mechanism, the multimodal data is semantically aligned in a fine-grained manner to obtain a unified semantic representation. The preset hierarchical attention mechanism includes an intra-modal attention layer and a cross-modal attention layer. The intra-modal attention layer is used to capture the dependencies of multimodal data and extract the key features within each modal data based on the dependencies. The cross-modal attention layer is used to semantically align and fuse the data of each modality based on the key features.

[0009] Generate decision results based on unified semantic representation and pre-trained generative adversarial network.

[0010] In a second aspect, an embodiment of the present application provides an intelligent agent adaptive decision-making device based on multimodal semantic alignment, the device comprising:

[0011] A multimodal data acquisition module is used to collect voice information, visual information, and tactile information through a multimodal sensor array pre-deployed on the intelligent agent, synchronize the timestamps, and obtain time-aligned multimodal data;

[0012] A fine-grained semantic alignment module is used to perform fine-grained semantic alignment on multimodal data based on a preset hierarchical attention mechanism to obtain a unified semantic representation. The preset hierarchical attention mechanism includes an intra-modal attention layer and a cross-modal attention layer. The intra-modal attention layer is used to capture the dependencies of multimodal data and extract key features within each modal data based on the dependencies. The cross-modal attention layer is used to perform semantic alignment and semantic fusion of each modal data based on key features.

[0013] The decision result generation module is used to generate decision results based on the unified semantic representation and the pre-trained generative adversarial network.

[0014] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0015] In an embodiment of the present application, according to a preset hierarchical attention mechanism, multimodal data is subjected to fine-grained semantic alignment. The intra-modal attention layer in the hierarchical attention mechanism can extract key features within each modal data. These key features are the most representative and informative parts of the modal data, providing an accurate basis for subsequent semantic alignment. At the same time, the cross-modal attention layer further performs semantic alignment and fusion on each modal data to generate a unified semantic representation, which solves the problem of differences in semantic expression between different modal data, greatly reduces semantic ambiguity and modal conflict problems in human-computer collaboration scenarios, and enables the intelligent agent to quickly and accurately grasp the semantic meaning of environmental information as a whole, and then accurately understand environmental information and user instructions, avoiding misunderstandings or erroneous operations caused by semantic conflicts or time inconsistencies between modalities, thereby improving the accuracy of the intelligent agent's decision-making.

[0016] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] Figure 1 This is a flowchart of a method for an intelligent agent adaptive decision-making method based on multimodal semantic alignment provided by an embodiment of the present application;

[0019] Figure 2 This is a schematic diagram of sensors deployed on an intelligent body provided in an embodiment of the present application;

[0020] Figure 3 This is an architectural diagram of a preset hierarchical attention mechanism provided in an embodiment of the present application;

[0021] Figure 4 This is a schematic diagram of a process of an intelligent agent adaptive decision-making process based on multimodal semantic alignment provided by an embodiment of the present application;

[0022] Figure 5 This is a schematic block diagram of an optimization process under a user feedback mechanism provided in an embodiment of the present application;

[0023] Figure 6 This is an architecture diagram of a pre-trained generative adversarial network provided in an embodiment of the present application;

[0024] Figure 7 This is a flow chart of a training method for a generative adversarial network provided in an embodiment of the present application;

[0025] Figure 8 This is a schematic block diagram of an adversarial training process provided by an embodiment of the present application;

[0026] Figure 9 This is a schematic diagram of the structure of an intelligent agent adaptive decision-making device based on multimodal semantic alignment provided by an embodiment of the present application;

[0027] Figure 10 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following description and the drawings sufficiently illustrate specific embodiments of the application to enable those skilled in the art to practice them.

[0029] It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0030] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0031] In the description of this application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances. In addition, in the description of this application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0032] Currently, embodied agents typically employ multimodal interaction technologies to interact with their environments and users. These technologies primarily involve the acquisition and processing of data from modalities such as speech recognition, computer vision, and tactile perception. After acquisition, these modal data are typically integrated using simple fusion methods (such as weighted averaging or feature concatenation) to support the agent's decision-making.

[0033] The applicant of this application realized that due to the differences in feature space and semantic expression of data of different modalities, simple fusion methods are difficult to achieve fine-grained semantic alignment, which may lead to misunderstandings or incorrect operations of the intelligent agent when performing tasks, thereby reducing the accuracy of the intelligent agent's decision-making.

[0034] In order to solve the above problems, the present application provides an intelligent agent adaptive decision-making method and device based on multimodal semantic alignment to solve the problems existing in the above-mentioned related technical problems. In an embodiment of the present application, according to a preset hierarchical attention mechanism, the multimodal data is fine-grained semantically aligned. The intra-modal attention layer in the hierarchical attention mechanism can extract the key features within each modal data. These key features are the most representative and informative parts of the modal data, providing an accurate basis for subsequent semantic alignment. At the same time, the cross-modal attention layer further semantically aligns and fuses each modal data to generate a unified semantic representation, solving the problem of differences in semantic expression between different modal data, greatly reducing the semantic ambiguity and modal conflict problems in human-computer collaboration scenarios, enabling the intelligent agent to quickly and accurately grasp the semantic meaning of environmental information as a whole, and then quickly and accurately understand environmental information and user instructions, avoiding misunderstandings or erroneous operations caused by semantic conflicts or time inconsistencies between modalities, thereby improving the accuracy of the intelligent agent's decision-making. The following exemplary embodiments are used for detailed explanation.

[0035] The following will be combined with the Figure 1 -Attached Figure 8 This article details the agent-based adaptive decision-making method for multimodal semantic alignment provided by an embodiment of the present application. This method can be implemented as a computer program and run on a von Neumann-based agent-based adaptive decision-making device for multimodal semantic alignment. This computer program can be integrated into an application or run as a standalone tool application.

[0036] See Figure 1 , which provides a flow chart of an agent adaptive decision-making method based on multimodal semantic alignment for an embodiment of the present application. Figure 1 As shown, the method of the embodiment of the present application includes the following steps:

[0037] S101, collecting voice information, visual information, and tactile information through a multimodal sensor array pre-deployed on the intelligent body, performing time stamp synchronization, and obtaining time-aligned multimodal data;

[0038] A multimodal sensor array is a collection of sensors of various types, used to simultaneously collect data from multiple modalities (such as voice, vision, and touch). The multimodal sensor array includes voice sensors, vision sensors, and tactile sensors. The voice sensor is a microphone array that collects sound signals. The vision sensor is an RGBD camera (color camera and depth camera) that captures images and depth information. The tactile sensor is a pressure sensor array that collects pressure distribution on an object's surface.

[0039] Voice information is audio signals collected by a microphone array, visual information is images captured by an RGBD camera, and tactile information is surface pressure distribution data collected by a pressure sensor array. Timestamp synchronization ensures temporal consistency of data from different modalities, meaning that all modal data is collected at the same time.

[0040] For example Figure 2 As shown, a large number of sensors need to be deployed on the intelligent agent. For the core algorithm of this application, a microphone array, such as a ring array of multiple microphones, is deployed on the intelligent agent to collect voice signals from all directions. An RGBD camera, such as the Intel RealSense D455, is then deployed to capture high-resolution color images and depth information. Finally, a flexible pressure sensor array, such as the Tekscan FlexiForce, is deployed to collect pressure distribution on the surface of objects.

[0041] After the hardware sensors are deployed, a microphone array is used to collect voice signals, an RGBD camera is used to capture images, and a flexible pressure sensor array is used to collect tactile signals. After data collection, software checks the timestamps of each modality's data and corrects for time deviations. For example, this can be achieved by interpolating or discarding some data to align time. Finally, the collected voice, visual, and tactile data is stored in a circular buffer, which supports dynamic reading.

[0042] S102: Performing fine-grained semantic alignment on the multimodal data according to a preset hierarchical attention mechanism to obtain a unified semantic representation. The preset hierarchical attention mechanism includes an intra-modal attention layer and a cross-modal attention layer. The intra-modal attention layer is used to capture the dependencies of the multimodal data and extract key features within each modal data based on the dependencies. The cross-modal attention layer is used to perform semantic alignment and semantic fusion on each modal data based on the key features.

[0043] Among them, the preset hierarchical attention mechanism is a pre-designed multimodal data processing mechanism that performs attention calculations on data of different modalities in a hierarchical manner to achieve fine-grained semantic alignment. Fine-grained semantic alignment is to perform detailed semantic analysis and alignment on the data of each modality in multimodal data to ensure that data of different modalities can accurately correspond at the semantic level. The unified semantic representation is used to represent the semantic alignment and fusion of data of different modalities to generate a unified semantic representation that can comprehensively reflect the semantic information of all modal data. Key features are the most representative and informative parts extracted from modal data. These features can effectively reflect the core information of modal data.

[0044] For example Figure 3As shown in the figure, the core preset hierarchical attention mechanism proposed in the scheme includes intra-modal attention layer and cross-modal attention layer.

[0045] Among them, multimodal data includes visual sequences, speech sequences and tactile sequences.

[0046] In some embodiments of the present application, the specific process of capturing the dependency relationship of multimodal data includes: extracting the visual features of each image frame in the visual sequence to obtain a visual feature vector sequence; using the Mel-frequency cepstral coefficient method to process each audio signal in the speech sequence to obtain an acoustic feature vector sequence; extracting the pressure change trend characteristics of each tactile signal in the tactile sequence to obtain a tactile feature vector sequence; applying the self-attention mechanism to the visual feature vector sequence, the acoustic feature vector sequence, and the tactile feature vector sequence respectively to extract the interaction between the elements in different sequences, and obtain a context vector sequence group as the dependency relationship corresponding to the multimodal data.

[0047] The visual sequence is a series of continuous image frames captured by an RGBD camera. The visual feature vector sequence is a collection of feature vectors extracted from the visual sequence, each vector representing the visual features of an image frame. Deep learning models (such as ResNet and VGG) can be used to extract image feature vectors. The speech sequence is a series of continuous audio signals captured by a microphone array. The acoustic feature vector sequence is a collection of feature vectors extracted from the speech sequence, each vector representing the acoustic features of an audio signal. The Mel-Frequency Cepstral Coefficient (MFCC) method can be used to extract feature vectors of audio signals. The tactile feature vector sequence is a collection of feature vectors extracted from the tactile sequence, each vector representing the pressure variation trend characteristics of a tactile signal. Feature vectors are extracted by analyzing the pressure variation trend of the tactile signal.

[0048] In the embodiments of the present application, this process can effectively capture the dependencies and contextual information within each modality in multimodal data, enabling the agent to more comprehensively and accurately understand the semantic content of each modal data. Through the self-attention mechanism, the agent can not only focus on the key information in each modality, but also understand the interrelationships between this information. This allows for more accurate semantic alignment and fusion in the subsequent multimodal fusion and decision-making process, improving the accuracy and reliability of decisions.

[0049] The context vector sequence group includes a visual context vector sequence, an acoustic context vector sequence, and a tactile context vector sequence.

[0050] In some embodiments of the present application, the specific process of generating a visual context vector sequence, an acoustic context vector sequence, and a tactile context vector sequence includes: applying a self-attention mechanism to the visual feature vector sequence, the acoustic feature vector sequence, and the tactile feature vector sequence, respectively, to generate a query vector, a key vector, and a value vector for the visual feature vector sequence, the acoustic feature vector sequence, and the tactile feature vector sequence; performing a linear transformation on the query vector, the key vector, the value vector, and the preset learnable weight matrix to obtain a self-attention matrix for the visual feature vector sequence, the acoustic feature vector sequence, and the tactile feature vector sequence; calculating an attention score representing the dependency strength between each information in the visual sequence, the speech sequence, and the tactile sequence based on the self-attention matrix; based on the attention score, separately aggregating the dependency relationships of the visual sequence, the speech sequence, and the tactile sequence to obtain a visual context vector sequence, an acoustic context vector sequence, and a tactile context vector sequence.

[0051] Among them, the expression of the self-attention matrix is:

[0052] , , ;

[0053] in, is the query vector, is the key vector, is a value vector, is a sequence of visual feature vectors, acoustic feature vectors, or tactile feature vectors, It is the preset learnable weight matrix under the self-attention mechanism;

[0054] Among them, the calculation expression of the attention score is:

[0055]

[0056] in, is the attention score, is the scaling factor, is the transpose operation;

[0057] The expressions for separately aggregating the dependencies of visual sequence, speech sequence and tactile sequence are:

[0058]

[0059] in, It is a sequence of visual context vectors, acoustic context vectors, or tactile context vectors.

[0060] The dependency relationship includes a context vector sequence group, and the context vector sequence group includes a visual context vector sequence, an acoustic context vector sequence, and a tactile context vector sequence.

[0061] In some embodiments of the present application, the specific process of extracting key features within each modal data based on dependency relationships includes: applying maximum pooling or average pooling operations to the visual context vector sequence to aggregate vectors in the time dimension to obtain text descriptions of objects in the dependency relationship as visual key features; applying time pooling to the acoustic context vector sequence to obtain command keywords in the dependency relationship as voice key features; applying pooling operations to the tactile context vector sequence to obtain pressure peaks in the dependency relationship as tactile key features; and using visual key features, voice key features, and tactile key features as key features within each modal data.

[0062] In the embodiments of this application, through the key feature extraction process described above, the intelligent agent is able to extract the most representative features from multimodal data. These features can reflect the core information of each modal data, which can significantly improve the accuracy and reliability of decision-making. For example, in complex environments, the intelligent agent can more accurately identify objects, understand voice commands, and perceive tactile information, thereby making more reasonable decisions.

[0063] Among them, key features include visual key features, voice key features and tactile key features.

[0064] In some embodiments of the present application, the specific process of semantically aligning and semantically fusing data of each modality based on key features includes: taking the speech key features as the query dominant modality, and taking the visual key features and tactile key features as auxiliary modalities; calculating the target query vector based on the query dominant modality, and calculating the target key vector and target value vector respectively based on the auxiliary modalities; calculating the cross-modal attention matrix that characterizes the fine-grained dependency relationship between modalities based on the target query vector and the target key vector; performing context feature extraction based on the cross-modal attention matrix and the target value vector to obtain aggregated visual and tactile key information related to the voice command; performing residual connection and normalization on the speech key features and the aggregated visual and tactile key information related to the voice command to obtain cross-modal semantic alignment features with the voice command as the anchor point; and performing semantic fusion through the cross-modal semantic alignment features with the voice command as the anchor point.

[0065] Among them, the calculation expression of the cross-modal attention matrix is:

[0066]

[0067] in, is the cross-modal attention matrix, () is the normalization function, is the target query vector, is the target key vector, is the scaling factor, express have Line and Column; among them,

[0068] Target query vector Target key vector Target value vector

[0069] in, is the key feature of speech, is the key visual feature, It is a key feature of touch;

[0070] The expression of the aggregated visual and tactile key information related to the voice command is:

[0071]

[0072] in, It is the aggregated visual and tactile key information related to the voice command;

[0073] The expression of cross-modal semantic alignment features with voice commands as anchors is:

[0074]

[0075] To align features of cross-modal semantics with voice commands as anchors, It is a residual connection and normalization layer.

[0076] In some embodiments of the present application, the specific process of performing semantic fusion through cross-modal semantic alignment features anchored by voice instructions includes: establishing a visual gate based on the cross-modal semantic alignment features and visual key features anchored by voice instructions; establishing a tactile gate based on the cross-modal semantic alignment features and tactile key features anchored by voice instructions; performing weighted fusion of the visual gate and visual key features, and the tactile gate and tactile key features to obtain fused features; and projecting the fused features into a unified semantic space to obtain a unified semantic representation.

[0077] The expression of the visual gate is:

[0078] The expression of the tactile gate is:

[0079] in, For visual door, For the tactile door, is the normalization function, is the learnable weight matrix, To align features for cross-modal semantics using voice commands as anchors;

[0080] The expression of fusion features is:

[0081]

[0082] in, It is a fusion feature;

[0083] The expression of unified semantic representation is:

[0084]

[0085] in, To unify the semantic representation, is the activation function, is the learnable parameter matrix, is a learnable bias term used to adjust the reference position of the linear transformation, To unify the dimensions.

[0086] For example Figure 4 As shown, the collected multimodal data includes visual sequences, speech sequences, and tactile sequences, and the collected multimodal data is input into a preset hierarchical attention mechanism, which includes an intra-modal attention layer and a cross-modal attention layer. The intra-modal attention layer is used to capture the dependency of multimodal data and extract the key features within each modal data based on the dependency. The cross-modal attention layer is used to perform semantic alignment and semantic fusion of each modal data according to the key features. Finally, the mechanism outputs a unified semantic representation.

[0087] S103: Generate a decision result based on the unified semantic representation and the pre-trained generative adversarial network.

[0088] In some embodiments of the present application, the specific process of generating a decision result based on the unified semantic representation and the pre-trained generative adversarial network includes: inputting the unified semantic representation into the pre-trained generative adversarial network, and outputting the semantic alignment features optimized by adversarial training; based on the semantic alignment features, generating a preliminary decision result and displaying it on a display preset in the intelligent body; if no user feedback information is received within a preset time period, taking the preliminary decision result as the final decision result; or, if user feedback information is received within a preset time period, dynamically adjusting the alignment parameters and decision parameters of the pre-trained generative adversarial network based on the feedback information and the current environmental state, and continuing to execute the step of inputting the unified semantic representation into the pre-trained generative adversarial network until the final decision result is obtained.

[0089] In one possible implementation, a preliminary decision result is generated based on the optimized semantically aligned features. This result can be an action instruction, a state description, or other decision information. A preset display device on the intelligent agent is used to display the preliminary decision result to the user. The user's feedback on the preliminary decision result can be "correct," "incorrect," or other forms of feedback. If no user feedback is received within a preset time, the system will use the preliminary decision result as the final decision result. For example, the preliminary decision result is generated as "turn on the TV" and displayed on the robot's display. The robot waits for user feedback within 30 seconds. If the user does not provide feedback within 30 seconds, the robot will use "turn on the TV" as the final decision result and execute it. If the user responds with "No, do not turn on the TV" within 30 seconds, the robot will adjust the GAN parameters based on the feedback information and regenerate the semantically aligned features until a decision result that meets the user's expectations is generated.

[0090] For example Figure 5 As shown, Figure 5 This is a schematic diagram of the optimization process under a user feedback mechanism provided by this application. The unified semantic representation is input into a pre-trained generative adversarial network, and the semantic alignment features optimized by adversarial training are output. Based on the semantic alignment features, a preliminary decision result is generated and displayed on a display preset in the intelligent body. It is determined whether the user feedback information is received within a preset time. If so, the alignment parameters and decision parameters of the pre-trained generative adversarial network are dynamically adjusted according to the feedback information and the current environmental state, and the unified semantic representation is continued to be input into the pre-trained generative adversarial network to output the semantic alignment features optimized by adversarial training; if not, the preliminary decision result is used as the final decision result.

[0091] Among them, for example Figure 6 As shown, the pre-trained generative adversarial network includes a generator and a discriminator.

[0092] A generative adversarial network (GAN) is a deep learning model whose generator is a neural network whose goal is to generate fake data samples that are indistinguishable from real data. The generator typically starts with a random noise vector and generates data samples through a series of transformations. The discriminator is a neural network whose goal is to distinguish between fake data samples generated by the generator and real data samples. The discriminator is typically a binary classifier that outputs a probability value, indicating the probability that the input sample is real data.

[0093] In some embodiments of the present application, the unified semantic representation is input into a pre-trained generative adversarial network, and the specific process of outputting the semantic alignment feature optimized by adversarial training includes: injecting Gaussian noise into the unified semantic representation to obtain a conditional noise vector; inputting the conditional noise vector into the generator to output a decision action tensor; inputting the decision action tensor into the discriminator to obtain a discrimination result; when the discrimination result indicates that the decision action tensor is consistent with the true result, the unified semantic representation is used as the semantic alignment feature optimized by adversarial training.

[0094] In an embodiment of the present application, according to a preset hierarchical attention mechanism, the multimodal data is subjected to fine-grained semantic alignment. The intra-modal attention layer in the hierarchical attention mechanism can extract the key features within each modal data. These key features are the most representative and informative parts of the modal data, providing an accurate basis for subsequent semantic alignment. At the same time, the cross-modal attention layer further semantically aligns and fuses the modal data to generate a unified semantic representation, which solves the problem of differences in semantic expression between different modal data, enabling the intelligent agent to accurately grasp the semantic meaning of environmental information as a whole, and then accurately understand environmental information and user instructions, avoiding misunderstandings or erroneous operations caused by semantic conflicts or time inconsistencies between modalities, thereby improving the accuracy of the intelligent agent's decision-making.

[0095] See Figure 7 , which is a flow chart of a training method for generating adversarial networks according to an embodiment of the present application. Figure 7 As shown, the method of the embodiment of the present application may include the following steps:

[0096] S201, load the generative adversarial network;

[0097] S202, obtaining historical voice information, historical visual information, and historical tactile information collected by the agent to obtain multiple sample data;

[0098] S203, inputting each sample data and the preset noise vector into the generator of the generative adversarial network, and outputting the aligned cross-modal features;

[0099] S204: Input the real modal data of each sample data and the aligned cross-modal features into the discriminator, and output the binary classification probability;

[0100] S205, when the binary classification probability reaches the minimum, generate a pre-trained generative adversarial network.

[0101] In some embodiments of the present application, a deep learning framework is used to load a generative adversarial network model. Multiple sample data are extracted from the preprocessed data set. Each sample data contains corresponding speech, visual and tactile information. For each sample data, a preset noise vector is generated. The sample data and the noise vector are input into the generator of the generative adversarial network. The generator outputs aligned cross-modal features, which can represent the common features of data of different modalities. The real modal data of each sample data and the aligned cross-modal features are input into the discriminator. The discriminator outputs a binary classification probability, indicating whether the input data is real data. During the training process, the parameters of the generator and the discriminator are continuously adjusted so that the binary classification probability output by the discriminator is minimized. When the binary classification probability reaches the minimum value, the generative adversarial network has been fully trained and can generate high-quality cross-modal features.

[0102] For example Figure 8 As shown, Figure 8 The following diagram illustrates the adversarial training process. The generative model receives each sample data and a random noise vector (z) as input. Its task is to generate fake data (G(z)) that resembles real data. The discriminative model receives two types of input: real sample data (X) and fake data generated by the generative model (G(z)). Its task is to distinguish whether the input data is real or generated, and to output a probability value indicating the likelihood that the input data is real. During training, the generative model and the discriminative model compete with each other. The generative model attempts to generate increasingly realistic data to deceive the discriminative model, while the discriminative model attempts to more accurately distinguish between real and generated data. This adversarial training enables the generative model to gradually learn the distribution of real data.

[0103] In an embodiment of the present application, according to a preset hierarchical attention mechanism, the multimodal data is subjected to fine-grained semantic alignment. The intra-modal attention layer in the hierarchical attention mechanism can extract the key features within each modal data. These key features are the most representative and informative parts of the modal data, providing an accurate basis for subsequent semantic alignment. At the same time, the cross-modal attention layer further semantically aligns and fuses the modal data to generate a unified semantic representation, which solves the problem of differences in semantic expression between different modal data, enabling the intelligent agent to accurately grasp the semantic meaning of environmental information as a whole, and then accurately understand environmental information and user instructions, avoiding misunderstandings or erroneous operations caused by semantic conflicts or time inconsistencies between modalities, thereby improving the accuracy of the intelligent agent's decision-making.

[0104] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0105] See Figure 9 , which shows a schematic diagram of the structure of an intelligent agent adaptive decision-making device based on multimodal semantic alignment, provided by an exemplary embodiment of the present application. This intelligent agent adaptive decision-making device based on multimodal semantic alignment can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The device 1 includes a multimodal data acquisition module 10, a fine-grained semantic alignment module 20, and a decision result generation module 30.

[0106] The multimodal data acquisition module 10 is used to collect voice information, visual information and tactile information through a multimodal sensor array pre-deployed on the intelligent body, synchronize the timestamps and obtain time-aligned multimodal data;

[0107] A fine-grained semantic alignment module 20 is configured to perform fine-grained semantic alignment on multimodal data based on a preset hierarchical attention mechanism to obtain a unified semantic representation. The preset hierarchical attention mechanism includes an intra-modal attention layer and a cross-modal attention layer. The intra-modal attention layer is configured to capture the dependencies of multimodal data and extract key features within each modal data based on the dependencies. The cross-modal attention layer is configured to perform semantic alignment and semantic fusion on each modal data based on the key features.

[0108] The decision result generation module 30 is used to generate a decision result based on the unified semantic representation and the pre-trained generative adversarial network.

[0109] It should be noted that the intelligent agent adaptive decision-making device based on multimodal semantic alignment provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when executing the intelligent agent adaptive decision-making method based on multimodal semantic alignment. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the intelligent agent adaptive decision-making device based on multimodal semantic alignment provided in the above embodiment and the intelligent agent adaptive decision-making method based on multimodal semantic alignment embodiment belong to the same concept. The implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0110] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0111] In an embodiment of the present application, according to a preset hierarchical attention mechanism, the multimodal data is subjected to fine-grained semantic alignment. The intra-modal attention layer in the hierarchical attention mechanism can extract the key features within each modal data. These key features are the most representative and informative parts of the modal data, providing an accurate basis for subsequent semantic alignment. At the same time, the cross-modal attention layer further semantically aligns and fuses the modal data to generate a unified semantic representation, which solves the problem of differences in semantic expression between different modal data, enabling the intelligent agent to accurately grasp the semantic meaning of environmental information as a whole, and then accurately understand environmental information and user instructions, avoiding misunderstandings or erroneous operations caused by semantic conflicts or time inconsistencies between modalities, thereby improving the accuracy of the intelligent agent's decision-making.

[0112] The present application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implements the intelligent agent adaptive decision-making method based on multimodal semantic alignment provided by the above-mentioned various method embodiments.

[0113] The present application also provides a computer program product containing instructions, which, when executed on a computer, enables the computer to execute the agent adaptive decision-making method based on multimodal semantic alignment of the above-mentioned various method embodiments.

[0114] See Figure 10 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 10 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0115] The communication bus 1002 is used to implement the connection and communication between these components.

[0116] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0117] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0118] The processor 1001 may include one or more processing cores. The processor 1001 utilizes various interfaces and circuits to connect various components within the electronic device 1000. It executes instructions, programs, code sets, or instruction sets stored in the memory 1005, and accesses data stored in the memory 1005 to perform various functions and process data within the electronic device 1000. Optionally, the processor 1001 may be implemented in hardware using at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 1001 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing content displayed on the display; and the modem handles wireless communications. It is understood that the modem may also be implemented independently of the processor 1001 and implemented on a separate chip.

[0119] Among them, the memory 1005 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage system located away from the aforementioned processor 1001. As Figure 10 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an intelligent agent adaptive decision application based on multimodal semantic alignment.

[0120] exist Figure 10In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain user input data; and the processor 1001 can be used to call the intelligent agent adaptive decision application based on multimodal semantic alignment stored in the memory 1005 and specifically perform the following operations:

[0121] The multimodal sensor array pre-deployed on the intelligent body collects voice information, visual information and tactile information and synchronizes the timestamps to obtain time-aligned multimodal data;

[0122] According to the preset hierarchical attention mechanism, the multimodal data is semantically aligned in a fine-grained manner to obtain a unified semantic representation. The preset hierarchical attention mechanism includes an intra-modal attention layer and a cross-modal attention layer. The intra-modal attention layer is used to capture the dependencies of multimodal data and extract the key features within each modal data based on the dependencies. The cross-modal attention layer is used to semantically align and fuse the data of each modality based on the key features.

[0123] Generate decision results based on unified semantic representation and pre-trained generative adversarial network.

[0124] In one embodiment, when the processor 1001 generates a decision result based on the unified semantic representation and the pre-trained generative adversarial network, it specifically performs the following operations:

[0125] The unified semantic representation is input into a pre-trained generative adversarial network, which outputs semantically aligned features optimized through adversarial training.

[0126] Based on the semantic alignment features, a preliminary decision result is generated and displayed on the display preset in the intelligent agent;

[0127] If no user feedback is received within the preset time, the preliminary decision result is used as the final decision result; alternatively, if user feedback is received within the preset time, the alignment parameters and decision parameters of the pre-trained generative adversarial network are dynamically adjusted according to the feedback information and the current environmental state, and the step of inputting the unified semantic representation into the pre-trained generative adversarial network is continued until the final decision result is obtained.

[0128] In one embodiment, when executing the process of capturing the dependency relationship of multimodal data, the processor 1001 specifically performs the following operations:

[0129] Extract visual features of each image frame in the visual sequence to obtain a visual feature vector sequence;

[0130] Mel-frequency cepstral coefficient method is used to process each audio signal in the speech sequence to obtain an acoustic feature vector sequence;

[0131] Extracting the pressure change trend characteristics of each tactile signal in the tactile sequence to obtain a tactile feature vector sequence;

[0132] The self-attention mechanism is applied to the visual feature vector sequence, acoustic feature vector sequence, and tactile feature vector sequence respectively to extract the interactions between elements in different sequences and obtain a context vector sequence group as the dependency relationship corresponding to the multimodal data.

[0133] In one embodiment, when executing the generation of the visual context vector sequence, the acoustic context vector sequence, and the tactile context vector sequence, the processor 1001 specifically performs the following operations:

[0134] Applying a self-attention mechanism to the visual feature vector sequence, the acoustic feature vector sequence, and the tactile feature vector sequence, respectively, to generate a query vector, a key vector, and a value vector for the visual feature vector sequence, the acoustic feature vector sequence, and the tactile feature vector sequence;

[0135] Performing linear transformations on the query vector, key vector, and value vector with a preset learnable weight matrix to obtain self-attention matrices for the visual feature vector sequence, acoustic feature vector sequence, and tactile feature vector sequence;

[0136] According to the self-attention matrix, the attention score representing the dependency strength between each piece of information in the visual sequence, speech sequence, and tactile sequence is calculated;

[0137] Based on the attention scores, the dependencies of the visual sequence, speech sequence, and tactile sequence are aggregated respectively to obtain the visual context vector sequence, acoustic context vector sequence, and tactile context vector sequence.

[0138] In one embodiment, when extracting key features within each modality data based on dependency relationships, the processor 1001 specifically performs the following operations:

[0139] Applying a maximum pooling or average pooling operation to the visual context vector sequence to aggregate the vectors in the time dimension to obtain the object text description in the dependency relationship as the visual key feature;

[0140] Apply temporal pooling to the acoustic context vector sequence to obtain command keywords in the dependency relationship as key speech features;

[0141] Applying a pooling operation to the tactile context vector sequence to obtain the pressure peaks in the dependency as tactile key features;

[0142] Visual key features, speech key features and tactile key features are used as key features within each modality data.

[0143] In one embodiment, when performing semantic alignment and semantic fusion on each modality data according to the key features, the processor 1001 specifically performs the following operations:

[0144] The voice key features are used as the query dominant modality, and the visual key features and tactile key features are used as auxiliary modalities;

[0145] Calculate the target query vector based on the query dominant mode, and calculate the target key vector and target value vector based on the auxiliary mode respectively;

[0146] Based on the target query vector and the target key vector, a cross-modal attention matrix is ​​calculated to represent the fine-grained dependencies between modalities.

[0147] Contextual feature extraction is performed based on the cross-modal attention matrix and target value vector to obtain aggregated visual and tactile key information related to the voice command;

[0148] Perform residual connection and normalization on the key speech features and the aggregated visual and tactile key information related to the speech command to obtain cross-modal semantic alignment features with the speech command as the anchor point;

[0149] Semantic fusion is performed by cross-modal semantic alignment features using voice commands as anchors.

[0150] In one embodiment, when performing semantic fusion using cross-modal semantic alignment features anchored by voice commands, the processor 1001 specifically performs the following operations:

[0151] Establish visual gates based on cross-modal semantic alignment features and visual key features anchored by voice commands;

[0152] Establish a tactile gate based on cross-modal semantic alignment features and tactile key features anchored by voice commands;

[0153] The visual gate and visual key features, the tactile gate and tactile key features are weightedly fused to obtain the fused features.

[0154] In one embodiment, when the processor 1001 inputs the unified semantic representation into a pre-trained generative adversarial network and outputs the semantic alignment features optimized by adversarial training, the processor 1001 specifically performs the following operations:

[0155] For the unified semantic representation, inject Gaussian noise to obtain the conditional noise vector;

[0156] Input the conditional noise vector into the generator and output the decision action tensor;

[0157] Input the decision action tensor into the discriminator to obtain the discrimination result;

[0158] When the discrimination result indicates that the decision action tensor is consistent with the true result, the unified semantic representation is used as the semantic alignment feature after adversarial training optimization.

[0159] In one embodiment, when executing the generation of a pre-trained generative adversarial network, the processor 1001 specifically performs the following operations:

[0160] Load the generative adversarial network;

[0161] Obtain historical voice information, historical visual information, and historical tactile information collected by the intelligent agent to obtain multiple sample data;

[0162] Input each sample data and the preset noise vector into the generator of the generative adversarial network and output the aligned cross-modal features;

[0163] Input the real modal data of each sample data and the aligned cross-modal features into the discriminator and output the binary classification probability;

[0164] When the binary classification probability reaches the minimum, a pre-trained generative adversarial network is generated.

[0165] In an embodiment of the present application, according to a preset hierarchical attention mechanism, the multimodal data is subjected to fine-grained semantic alignment. The intra-modal attention layer in the hierarchical attention mechanism can extract the key features within each modal data. These key features are the most representative and informative parts of the modal data, providing an accurate basis for subsequent semantic alignment. At the same time, the cross-modal attention layer further semantically aligns and fuses the modal data to generate a unified semantic representation, which solves the problem of differences in semantic expression between different modal data, enabling the intelligent agent to accurately grasp the semantic meaning of environmental information as a whole, and then accurately understand environmental information and user instructions, avoiding misunderstandings or erroneous operations caused by semantic conflicts or time inconsistencies between modalities, thereby improving the accuracy of the intelligent agent's decision-making.

[0166] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program for adaptive decision-making of an intelligent agent based on multimodal semantic alignment can be stored in a computer-readable storage medium. When executed, the program can include the processes of the above-described method embodiments. The storage medium for the program for adaptive decision-making of an intelligent agent based on multimodal semantic alignment can be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0167] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An agent adaptive decision-making method based on multimodal semantic alignment, characterized in that: The method comprises: The multimodal sensor array pre-deployed on the intelligent body collects voice information, visual information and tactile information and synchronizes the timestamps to obtain time-aligned multimodal data; According to a preset hierarchical attention mechanism, the multimodal data is fine-grained semantically aligned to obtain a unified semantic representation. The preset hierarchical attention mechanism includes an intra-modal attention layer and a cross-modal attention layer. The intra-modal attention layer is used to capture the dependency relationship of the multimodal data and extract key features within each modal data based on the dependency relationship. The cross-modal attention layer is used to perform semantic alignment and semantic fusion on the modal data according to the key features. The preset hierarchical attention mechanism is a multimodal data processing mechanism that is used to perform attention calculations on data of different modalities in a hierarchical manner to achieve fine-grained semantic alignment; The multimodal data includes a visual sequence, a speech sequence, and a tactile sequence; Capturing the dependency relationship of the multimodal data includes: Extracting visual features of each image frame in the visual sequence to obtain a visual feature vector sequence; Using a Mel-frequency cepstral coefficient method to process each audio signal in the speech sequence to obtain an acoustic feature vector sequence; extracting a pressure variation trend feature of each tactile signal in the tactile sequence to obtain a tactile feature vector sequence; Applying a self-attention mechanism to the visual feature vector sequence, the acoustic feature vector sequence, and the tactile feature vector sequence respectively to extract the interactions between the elements in different sequences, thereby obtaining a context vector sequence group as the dependency relationship corresponding to the multimodal data; The key features include visual key features, voice key features and tactile key features; The performing semantic alignment and semantic fusion on the modal data according to the key features includes: The speech key feature is used as the query dominant modality, and the visual key feature and the tactile key feature are used as auxiliary modalities; Calculating a target query vector according to the query dominant modality, and respectively calculating a target key vector and a target value vector according to the auxiliary modality; Calculating a cross-modal attention matrix representing fine-grained dependencies between modalities based on the target query vector and the target key vector; performing context feature extraction based on the cross-modal attention matrix and the target value vector to obtain aggregated visual and tactile key information related to the voice command; Residual-connecting and normalizing the speech key features with the aggregated visual and tactile key information related to the speech command to obtain a cross-modal semantic alignment feature with the speech command as an anchor point; Semantic fusion is performed by cross-modal semantic alignment features using voice commands as anchors; A decision result is generated based on the unified semantic representation and a pre-trained generative adversarial network.

2. The method according to claim 1, characterized in that Generating a decision result based on the unified semantic representation and the pre-trained generative adversarial network includes: Inputting the unified semantic representation into a pre-trained generative adversarial network, and outputting semantic alignment features optimized by adversarial training; Generating a preliminary decision result based on the semantic alignment feature and displaying it on a display preset in the intelligent agent; If no user feedback is received within the preset time period, the preliminary decision result is used as the final decision result; or, if user feedback is received within the preset time period, the alignment parameters and decision parameters of the pre-trained generative adversarial network are dynamically adjusted according to the feedback information and the current environmental state, and the step of inputting the unified semantic representation into the pre-trained generative adversarial network is continued until the final decision result is obtained.

3. The method according to claim 1, characterized in that The context vector sequence group includes a visual context vector sequence, an acoustic context vector sequence, and a tactile context vector sequence; The visual context vector sequence, the acoustic context vector sequence, and the tactile context vector sequence are generated according to the following steps, including: Applying a self-attention mechanism to the visual feature vector sequence, the acoustic feature vector sequence, and the tactile feature vector sequence, respectively, to generate a query vector, a key vector, and a value vector for the visual feature vector sequence, the acoustic feature vector sequence, and the tactile feature vector sequence; Performing a linear transformation on the query vector, the key vector, the value vector, and a preset learnable weight matrix to obtain a self-attention matrix for the visual feature vector sequence, the acoustic feature vector sequence, and the tactile feature vector sequence; Calculating, based on the self-attention matrix, an attention score representing the dependency strength between each piece of information in the visual sequence, the speech sequence, and the tactile sequence; Based on the attention scores, the dependencies of the visual sequence, speech sequence, and tactile sequence are aggregated respectively to obtain a visual context vector sequence, an acoustic context vector sequence, and a tactile context vector sequence.

4. The method according to claim 1, wherein The dependency relationship includes a context vector sequence group, wherein the context vector sequence group includes a visual context vector sequence, an acoustic context vector sequence, and a tactile context vector sequence; The extracting key features within each modal data based on the dependency relationship includes: Applying a maximum pooling or average pooling operation to the visual context vector sequence to aggregate the vectors in the time dimension to obtain the object text description in the dependency relationship as the visual key feature; Applying time pooling to the acoustic context vector sequence to obtain command keywords in the dependency relationship as key speech features; Applying a pooling operation to the tactile context vector sequence to obtain pressure peaks in the dependency relationship as tactile key features; The visual key features, the speech key features, and the tactile key features are used as key features within each modality data.

5. The method according to claim 1, wherein The calculation expression of the cross-modal attention matrix is: in, is the cross-modal attention matrix, () is the normalization function, is the target query vector, is the target key vector, is the scaling factor, express have Line and Column; among them, The target query vector The target key vector The target value vector in, is the key feature of speech, is the key visual feature, It is a key feature of touch; It is the preset learnable weight matrix under the self-attention mechanism; The expression of the aggregated visual and tactile key information related to the voice command is: in, It is the aggregated visual and tactile key information related to the voice command; The expression of cross-modal semantic alignment features with voice commands as anchors is: To align features of cross-modal semantics with voice commands as anchors, It is a residual connection and normalization layer.

6. The method according to claim 1, wherein The semantic fusion is performed by using cross-modal semantic alignment features anchored by voice commands, including: Establishing a visual gate based on the cross-modal semantic alignment feature with the voice command as an anchor and the visual key features; Establishing a tactile gate based on the cross-modal semantic alignment feature anchored by the voice command and the tactile key feature; The visual gate and the visual key feature, the tactile gate and the tactile key feature are weightedly fused to obtain a fused feature.

7. The method according to claim 2, characterized in that The pre-trained generative adversarial network includes a generator and a discriminator; Inputting the unified semantic representation into a pre-trained generative adversarial network and outputting semantic alignment features optimized through adversarial training includes: Injecting Gaussian noise into the unified semantic representation to obtain a conditional noise vector; Input the conditional noise vector into the generator and output a decision action tensor; Inputting the decision action tensor into the discriminator to obtain a discrimination result; When the discrimination result indicates that the decision action tensor is consistent with the true result, the unified semantic representation is used as the semantic alignment feature after adversarial training optimization.

8. The method according to claim 1, characterized in that Follow these steps to generate a pre-trained generative adversarial network, including: Load the generative adversarial network; Acquiring historical voice information, historical visual information, and historical tactile information collected by the agent to obtain a plurality of sample data; Inputting each sample data and a preset noise vector into the generator of the generative adversarial network, and outputting aligned cross-modal features; Input the real modal data of each sample data and the aligned cross-modal features into the discriminator and output the binary classification probability; When the binary classification probability reaches the minimum, a pre-trained generative adversarial network is generated.

9. An intelligent agent adaptive decision-making device based on multimodal semantic alignment implemented using the method according to any one of claims 1 to 8, characterized in that: The device comprises: A multimodal data acquisition module is used to collect voice information, visual information, and tactile information through a multimodal sensor array pre-deployed on the intelligent agent, synchronize the timestamps, and obtain time-aligned multimodal data; A fine-grained semantic alignment module is configured to perform fine-grained semantic alignment on the multimodal data according to a preset hierarchical attention mechanism to obtain a unified semantic representation. The preset hierarchical attention mechanism includes an intra-modal attention layer and a cross-modal attention layer. The intra-modal attention layer is configured to capture the dependencies of the multimodal data and extract key features within each modal data based on the dependencies. The cross-modal attention layer is configured to perform semantic alignment and semantic fusion on the modal data according to the key features. A decision result generation module is used to generate a decision result based on the unified semantic representation and the pre-trained generative adversarial network.

Images