AI agent behavior prediction method based on multi-modal data

Abstract

The invention discloses an AI agent behavior prediction method based on multi-modal data, and particularly relates to the field of multi-modal data processing, and the method comprises the steps: S1, determining an agent data collection region, S2, obtaining agent multi-modal data, S3, processing the multi-modal data, S4, coding the multi-modal data, S5, fusing the multi-modal data, and S6, predicting the agent behavior. According to the method, multi-modal data such as vision, acceleration and voice are fused, the limitation of single data in description of agent behaviors is overcome, information is acquired from multiple dimensions, the behavior prediction accuracy is greatly improved, the potential relationship among different modal data is deeply mined through cooperative work of an attention mechanism and the graph convolutional network, and the behavior prediction accuracy is improved. According to the method, multi-modal data is subjected to multi-modal analysis, internal relations in the multi-modal data are accurately grasped, more powerful support is provided for prediction, in addition, targeted preprocessing is performed on the multi-modal data, the adaptability of the model to complex scenes is enhanced, and then the stability of prediction in different scenes is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing technology, and more specifically, to a method for predicting the behavior of AI agents based on multimodal data. Background Technology

[0002] In the field of artificial intelligence, agent behavior prediction is crucial for numerous applications, such as autonomous driving, intelligent security, and human-computer interaction. Traditional behavior prediction methods often rely on single-modal data, using only visual image data or sensor data. In autonomous driving scenarios, relying solely on visual images acquired by cameras is insufficient to accurately determine information such as vehicle speed and acceleration; while relying solely on sensor data fails to provide an intuitive understanding of the visual characteristics of the surrounding environment. With technological advancements, the concept of multimodal data fusion has gradually emerged. It integrates data from multiple sources, attempting to more comprehensively describe the agent and its environment. However, existing multimodal data fusion technologies still face many challenges in behavior prediction.

[0003] On the one hand, data fusion is ineffective. Traditional methods simply splice together data from different modalities without fully considering the complex interrelationships between the data, failing to effectively integrate information and reducing prediction accuracy. For example, visual data is high-dimensional image information, while speech data is a one-dimensional audio signal; effectively fusing these significantly different data becomes a challenge. On the other hand, models have poor adaptability to complex scenarios. Real-world scenarios contain interference factors such as data noise, occlusion, and changes in lighting. Existing methods lack robustness and cannot effectively handle these complex situations, leading to unstable prediction results. For instance, in autonomous driving, encountering severe weather can interfere with sensor data, making it difficult to predict vehicle behavior. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide an AI agent behavior prediction method based on multimodal data. The method addresses the problems mentioned in the background art, such as poor data fusion, inability to effectively integrate information, reduced prediction accuracy, poor model adaptability to complex scenarios, lack of robustness, inability to effectively handle complex real-world situations, and unstable prediction results.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an AI agent behavior prediction method based on multimodal data, comprising: S1: determining the agent data collection area: deploying information collection terminals for the agent; The information acquisition terminal includes a panoramic camera, a red light camera, an accelerometer, a gyroscope sensor, and a microphone array; S2: Acquire multimodal data of the intelligent agent: With the help of the deployed information collection terminal, acquire multimodal integrated data text related to the intelligent agent to be predicted. The multimodal integrated data text includes visual image data text, acceleration data text, and voice data text. S3: Multimodal data processing: Preprocess the acquired multimodal integrated data categories to obtain visual image preprocessed text, acceleration preprocessed text, and speech preprocessed text; S4: Multimodal data encoding: Input the preprocessed text of visual images, preprocessed text of acceleration, and preprocessed text of speech into the constructed behavior prediction model to obtain visual image feature text, acceleration feature text, and speech feature text; S5: Multimodal data fusion: Visual image feature text, acceleration feature text, and speech feature text are input into the attention mechanism to calculate the text. The attention mechanism calculates the attention score matrix. The attention score matrix is ​​used to weight and sum the features of each modality to obtain the weighted feature text. The graph convolutional network is used to process the weighted features to obtain the fused feature text. S6: Agent Behavior Prediction: Input the fused feature text into the agent behavior decoder to obtain the future behavior prediction results of the AI ​​agent.

[0006] The technical effects and advantages of this invention are as follows: This invention acquires visual, acceleration, and speech multimodal data, and uses a multimodal encoder to encode different modal data separately. This solves the problem that traditional single-modal data cannot fully describe the behavior of intelligent agents. It makes full use of the complementarity of multimodal data, enabling the model to learn richer and more representative features, enhancing the ability to express the behavior of intelligent agents, and thus improving the accuracy of behavior prediction. This invention employs a combination of attention mechanisms and graph convolutional networks for modal fusion and interaction. The attention mechanism calculates the correlation weights between features of different modalities, highlighting important features and enabling the model to dynamically focus on key information during fusion. The graph convolutional network processes the fused features, capturing the spatial relationships and interaction information between them. This fusion method solves the problems of existing multimodal data fusion methods that simply splice data together and cannot deeply explore the potential connections between data. It allows the model to more accurately grasp the complex intrinsic relationships between multimodal data, providing a more powerful basis for behavior prediction. To address issues such as data noise, occlusion, and lighting variations in complex scenarios, the multimodal data preprocessing steps of this invention perform targeted processing on data of each modality. Visual image data is cropped and normalized to remove irrelevant information and unify data scale; sensor data is denoised and filtered to reduce noise interference; and speech data is recognized and feature extracted to improve data quality. This enables the model to operate stably even in complex scenarios, effectively handles disturbed data, and ensures the stability and reliability of behavior prediction. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the overall structure of the present invention. Detailed Implementation

[0008] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0009] As attached Figure 1 The method for predicting the behavior of an AI agent based on multimodal data, as shown, includes: S1: Determining the data collection area for the intelligent agent: Deploying information collection terminals for the intelligent agent; The information acquisition terminal includes a panoramic camera, a red light camera, an accelerometer, a gyroscope sensor, and a microphone array; It should be further explained that the panoramic camera can acquire 360-degree panoramic images for the intelligent agent to fully perceive the surrounding environment, and the red light camera generates images by capturing infrared light emitted by objects, helping the intelligent agent to perform visual perception in no light or low light environments. The accelerometer sensor refers to a microelectromechanical system (MEMS) accelerometer sensor, which can be integrated into the main structure of the AI ​​agent. It measures the force generated by the internal mass block under acceleration, converts it into an electrical signal via a conversion circuit, and then converts the analog signal into a digital signal output via an analog-to-digital converter, thereby acquiring acceleration data. This sensor mainly acquires the acceleration information of the agent along the x, y, and z axes. The gyroscope sensor refers to a MEMS gyroscope sensor, which is used to acquire the angular velocity data of the agent around each axis, accurately measuring the agent's rotational motion and determining its attitude changes and turning rates. The microphone array consists of multiple microphones that can simultaneously collect sound signals. By processing these signals, not only can the voice content be acquired, but the voice source can also be located, allowing the acquisition of voice information in the AI ​​agent's surrounding environment, including dialogues between other individuals and the agent, and voice commands in the environment.

[0010] S2: Acquire multimodal data of the agent: With the help of the deployed information collection terminal, acquire multimodal integrated data text related to the agent to be predicted. The multimodal integrated data text includes visual image data text, acceleration data text, and voice data text.

[0011] S3: Multimodal data processing: Preprocess the acquired multimodal integrated data categories to obtain visual image preprocessed text, acceleration preprocessed text, and speech preprocessed text; This embodiment requires specific explanation; the preprocessing operation includes: Visual image data text preprocessing: Identify the agent in the image, using its bounding rectangle as the cropping region. Resize the cropped image to 224×224 pixels. Normalize the cropped image, reducing pixel values ​​from [0,255] to [-1,1] to obtain the preprocessed visual image text. The normalization formula is: ; in These are the original pixel values. These are the normalized pixel values. This normalization can speed up the convergence of CNN training and avoid gradient vanishing or exploding problems. Acceleration data text preprocessing: Use a Butterworth low-pass filter to remove high-frequency noise and retain low-frequency signals. Determine the cutoff frequency and order of the filter based on the sensor's sampling frequency. For example, for gyroscope sensor data, if the main focus is on low-frequency attitude change information, with a sampling frequency of 100Hz, the cutoff frequency can be set to 10Hz and the order to 4. Use the scipy.signal.butter and scipy.signal.lfilter functions to filter the data, making it smoother and obtaining the preprocessed acceleration text. Speech data text preprocessing: A wavelet transform-based denoising method is used to denoise the audio, resulting in denoised audio data. The audio data is divided into multiple short frames with a frame length of 20-30 milliseconds and a frame shift of 10 milliseconds. For each frame, its energy and zero-crossing rate are calculated. Energy thresholds and zero-crossing rate thresholds are determined based on historical data. If the energy of a frame is lower than the energy threshold and the zero-crossing rate is lower than the zero-crossing rate threshold, the frame is determined to be a silent frame and removed, thus obtaining an audio segment containing valid speech content. The audio segment data is converted to the Mel frequency domain, and cepstral coefficients are calculated as Mel frequency cepstral coefficient features. The extracted Mel frequency cepstral coefficient features are normalized using the min-max normalization method to obtain the preprocessed speech text.

[0012] S4: Multimodal data encoding: Input the preprocessed text of visual images, preprocessed text of acceleration, and preprocessed text of speech into the constructed behavior prediction model to obtain visual image feature text, acceleration feature text, and speech feature text; The behavior prediction model includes a visual image encoder, an acceleration data encoder, and a speech data encoder. The specific encoding steps of the visual image encoder are as follows: The preprocessed visual image text is input into the ResNet50 model. The ResNet50 model contains multiple convolutional layers, pooling layers, and residual blocks. The preprocessed visual image text first undergoes preliminary feature extraction through a 7x7 convolutional layer. Then, the data dimensionality is reduced through a max pooling layer. Subsequently, the preprocessed visual image text passes through multiple residual blocks, which can effectively learn complex features in the image. Finally, the spatial dimensional information is aggregated through a global average pooling layer to obtain a visual feature vector with a dimension of 1024. This vector is the visual image feature text. The acceleration data encoder refers to the Long Short-Term Memory (LSTM) network model. The specific encoding steps are as follows: the preprocessed acceleration text is input into the LSTM network step by step. The LSTM unit contains a forget gate, an input gate, and an output gate. These gating mechanisms can control the inflow and outflow of information and effectively handle the long-term dependency problem in time series data. The LSTM unit receives the acceleration data at the current time step and the hidden state at the previous time step. It processes and memorizes the information through the gating mechanism and updates the hidden state. When the data at all time steps has been input, the feature vector with a dimension of 512 output by the hidden state at the last time step is the acceleration feature text. The audio data encoder refers to the BERT model based on the Transformer architecture. The specific encoding steps are as follows: the preprocessed audio text is processed by word embedding, and each word is converted into a fixed-dimensional vector representation. At the same time, in order to enable the model to learn the position information of words in the sequence, position encoding is added. Then, the embedded vector is input into the BERT model. The BERT model consists of multiple Transformer blocks. Each Transformer block contains a multi-head attention mechanism and a multi-layer perceptron. After processing by multiple Transformer blocks, the average value of the word vectors output by the last layer of the BERT model is taken to obtain a feature vector with a dimension of 768, which is the audio feature text.

[0013] S5: Multimodal data fusion: Visual image feature text, acceleration feature text, and speech feature text are input into the attention mechanism to calculate the text. The attention mechanism calculates the attention score matrix. The attention score matrix is ​​used to weight and sum the features of each modality to obtain the weighted feature text. The graph convolutional network is used to process the weighted features to obtain the fused feature text. The specific calculation process of the attention mechanism is as follows: Visual image features text... Acceleration feature text Speech feature text Each is obtained by mapping to the same dimensional space through a linear transformation. , , Then calculate the attention score matrix. , , , They are , , The query vector and key vector obtained after linear transformation Refers to the index variable, here The value ranges from 1 to 3, representing the three modalities of vision, acceleration, and speech. This represents the transpose operation. Attention score matrix The elements in the expression represent the correlation score between the i-th modal eigenvector and the j-th modal eigenvector.

[0014] The specific steps of the linear transformation are as follows: A1: Determine the target dimension: Based on the computing resource requirements, determine a suitable common dimension. ; A2: Constructing the linear transformation matrix: For visual image feature text, its original dimension is... Build a size of weight matrix The elements in the matrix are initialized based on the normal distribution of visual image feature text. Similarly, for the original dimension... Acceleration feature text, construct weight matrix Size is For the original dimension is Construct a weight matrix from the speech features of the text. Size is ; A3: Visual image feature text With weight matrix Multiplication, that is , to obtain the mapping to the target dimension Visual feature vectors Similarly, for acceleration feature text ,calculate The mapped acceleration feature vector is obtained. For text with sound features ,calculate The mapped speech feature vector is obtained. ; It should be further explained that the weighted summation operation steps are as follows: For visual image features The weighting coefficients are calculated as follows: The corresponding matrix row vector Perform the Softmax operation, i.e. , obtained Each element in the vector corresponds to a modality, including visual image, acceleration, and speech, representing the relative importance of visual image features when correlated with other modalities. Similarly, for acceleration features... Corresponding row vectors and speech features Perform the same Softmax operation on the corresponding row vectors to obtain the following results: and ; For visual image features The weighted features are: ; For acceleration characteristics The weighted features are: ; For speech features The weighted features are: ; The weighted modal features are summed to obtain the weighted feature text: ; The graph convolutional network is used to perform convolution operations on graph-structured data, constructing a graph structure with modal features as nodes. Weighted visual image, acceleration, and speech features are used as nodes, and edges between nodes represent the relationships between features. This extracts features from the nodes and edges in the graph, resulting in fused feature text. The graph convolution operation is defined as follows: ,in It is the first The feature matrix of the layer, It adds a self-connected adjacency matrix. yes The degree matrix, It is the first The weight matrix of the layer, The mathematical expression for the activation function is: The meaning of this expression is that when the input value is greater than 0, the output is equal to the input value; when the input value is less than or equal to 0, the output is 0. In the graph convolutional network model, ReLU is used as the activation function. Its significance lies in introducing nonlinear transformation, breaking the limitations of the linear model, enabling the model to learn more complex functional relationships, and enhancing the model's expressive power.

[0015] S6: Agent Behavior Prediction: Input the fused feature text into the agent behavior decoder to obtain the future behavior prediction results of the AI ​​agent; In this embodiment, it should be specifically explained that the operation steps of the decoder include LSTM layer processing, fully connected MLP transformation, and trajectory prediction output, as follows: The LSTM layer is used to capture the relationship between the current position and the position and motion state in the past. At each time step, the LSTM unit receives the fused features of the current time step, as well as the hidden state and cell state of the previous time step. The forget gate decides to discard information from the previous cell state based on the input information. For example, if a pedestrian was previously stationary but is now making obvious movements, some historical information related to being stationary may be discarded. The input gate determines the new information to be added to the cell state, such as newly detected pedestrian movement changes. The output gate, based on the updated cell state and the hidden state, determines the output information, which participates in the update of the hidden state. After a series of time steps, the hidden state at the last time step of the LSTM layer contains the pedestrian's motion pattern and state information throughout the entire time series. It is a highly abstract representation of the fused features in the time dimension. The hidden state obtained through the LSTM layer... The data is then input into a Multilayer Perceptron (MLP) for further feature transformation and abstraction. The first hidden layer of the MLP... Perform a linear transformation using the weight matrix. and bias get Then, after activation by the activation function, we get... ; This process highlights important features and suppresses unimportant ones. Subsequent hidden layers repeat similar operations, with each layer further extracting and combining features to make the feature representation more suitable for trajectory prediction tasks. With each layer, the MLP gradually maps the fused features and the temporal features extracted by LSTM onto feature representations directly related to the pedestrian trajectory, such as the pedestrian's displacement and direction changes. The result y obtained by linear transformation of the output of the last hidden layer of the MLP is the prediction parameter of the pedestrian trajectory. For example, y can directly represent the coordinate value of the pedestrian in space in the next few time steps, or the change in coordinates, speed, etc. After appropriate transformation, these values ​​can be used to obtain the future trajectory prediction of the pedestrian. Since the input data has a time dimension, the decoder output can be the trajectory points of the pedestrian in the next multiple time steps, thus forming a predicted trajectory. Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other. In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for predicting the behavior of an AI agent based on multimodal data, characterized in that, include: S1: Determining the data collection area for the intelligent agent: Deploying information collection terminals for the intelligent agent; The information acquisition terminal includes a panoramic camera, a red light camera, an accelerometer, a gyroscope sensor, and a microphone array; S2: Acquire multimodal data of the intelligent agent: With the help of the deployed information collection terminal, acquire multimodal integrated data text related to the intelligent agent to be predicted. The multimodal integrated data text includes visual image data text, acceleration data text, and voice data text. S3: Multimodal data processing: Preprocess the acquired multimodal integrated data categories to obtain visual image preprocessed text, acceleration preprocessed text, and speech preprocessed text; S4: Multimodal data encoding: Input the preprocessed text of visual images, preprocessed text of acceleration, and preprocessed text of speech into the constructed behavior prediction model to obtain visual image feature text, acceleration feature text, and speech feature text; S5: Multimodal data fusion: Visual image feature text, acceleration feature text, and speech feature text are input into the attention mechanism to calculate the text. The attention mechanism calculates the attention score matrix. The attention score matrix is ​​used to weight and sum the features of each modality to obtain the weighted feature text. The graph convolutional network is used to process the weighted features to obtain the fused feature text. S6: Agent Behavior Prediction: Input the fused feature text into the agent behavior decoder to obtain the future behavior prediction results of the AI ​​agent.

2. The AI ​​agent behavior prediction method based on multimodal data according to claim 1, characterized in that: The preprocessing operations include: Visual image data text preprocessing: Identify the agent in the image, using its bounding rectangle as the cropping region. Resize the cropped image to 224×224 pixels. Normalize the cropped image, reducing pixel values ​​from [0,255] to [-1,1] to obtain the preprocessed visual image text. The normalization formula is: ； in These are the original pixel values. These are the normalized pixel values; Acceleration data text preprocessing: Use a Butterworth low-pass filter to remove high-frequency noise and retain low-frequency signals. Determine the cutoff frequency and order of the filter based on the sensor's sampling frequency. Use the scipy.signal.butter and scipy.signal.lfilter functions to perform filtering and obtain the preprocessed acceleration text. Speech data text preprocessing: A wavelet transform-based denoising method is used to denoise the audio, resulting in denoised audio data. The audio data is divided into multiple short frames with a frame length of 20-30 milliseconds and a frame shift of 10 milliseconds. For each frame, its energy and zero-crossing rate are calculated. Energy thresholds and zero-crossing rate thresholds are determined based on historical data. If the energy of a frame is lower than the energy threshold and the zero-crossing rate is lower than the zero-crossing rate threshold, the frame is determined to be a silent frame and removed, thus obtaining an audio segment containing valid speech content. The audio segment data is converted to the Mel frequency domain, and cepstral coefficients are calculated as Mel frequency cepstral coefficient features. The extracted Mel frequency cepstral coefficient features are normalized using the min-max normalization method to obtain the preprocessed speech text.

3. The AI ​​agent behavior prediction method based on multimodal data according to claim 1, characterized in that: The behavior prediction model includes a visual image encoder, an acceleration data encoder, and a speech data encoder. The specific encoding steps of the visual image encoder are as follows: the visual image preprocessed text is input into the ResNet50 model. The visual image preprocessed text first goes through a 7x7 convolutional layer for preliminary feature extraction, then through a max pooling layer to reduce the data dimension, then through a residual block, and finally through a global average pooling layer to aggregate the spatial dimension information to obtain a visual feature vector. The acceleration data encoder refers to the Long Short-Term Memory (LSTM) network model. The speech data encoder refers to the BERT model based on the Transformer architecture.

4. The AI ​​agent behavior prediction method based on multimodal data according to claim 1, characterized in that: The specific calculation process of the attention mechanism is as follows: Visual image features text... Acceleration feature text Speech feature text Each is obtained by mapping to the same dimensional space through a linear transformation. , , Then calculate the attention score matrix. , , , They are , , The query vector and key vector obtained after linear transformation Refers to the index variable, here The value ranges from 1 to 3, representing the three modalities of vision, acceleration, and speech. This represents the transpose operation. Attention score matrix The elements in the expression represent the correlation score between the i-th modal eigenvector and the j-th modal eigenvector.

5. The AI ​​agent behavior prediction method based on multimodal data according to claim 4, characterized in that: The specific steps of the linear transformation are as follows: A1: Determine the target dimension: Based on the computing resource requirements, determine a suitable common dimension. ; A2: Constructing the linear transformation matrix: For visual image feature text, its original dimension is... Build a size of weight matrix The elements in the matrix are initialized based on the normal distribution of visual image feature text. Similarly, for the original dimension... Acceleration feature text, construct weight matrix Size is For the original dimension is Construct a weight matrix from the speech features of the text. Size is ; A3: Visual image feature text With weight matrix Multiplication, that is , to obtain the mapping to the target dimension Visual feature vectors Similarly, for acceleration feature text ,calculate The mapped acceleration feature vector is obtained. For text with sound features ,calculate The mapped speech feature vector is obtained. .

6. The AI ​​agent behavior prediction method based on multimodal data according to claim 1, characterized in that: The steps for the weighted summation are as follows: For visual image features The weighting coefficients are calculated as follows: The corresponding matrix row vector Perform the Softmax operation, i.e. , obtained Each element in the vector corresponds to a modality, including visual image, acceleration, and speech, representing the relative importance of visual image features when correlated with other modalities. Similarly, for acceleration features... Corresponding row vectors and speech features Perform the same Softmax operation on the corresponding row vectors to obtain the following results: and ; For visual image features The weighted features are: ； For acceleration characteristics The weighted features are: ； For speech features The weighted features are: ； The weighted modal features are summed to obtain the weighted feature text: 。 7. The AI ​​agent behavior prediction method based on multimodal data according to claim 1, characterized in that: The graph convolutional network is used to perform convolution operations on graph-structured data, constructing a graph structure with modal features as nodes. Weighted visual image, acceleration, and speech features are used as nodes, and edges between nodes represent the relationships between features. This extracts features from the nodes and edges in the graph, resulting in fused feature text. The graph convolution operation is defined as follows: ,in It is the first The feature matrix of the layer, It adds a self-connected adjacency matrix. yes The degree matrix, It is the first The weight matrix of the layer, The mathematical expression for the activation function is: .

8. The AI ​​agent behavior prediction method based on multimodal data according to claim 1, characterized in that: The decoder's operation steps include LSTM layer processing, fully connected MLP transformation, and trajectory prediction output.

Images