A multimodal fusion method, system, device, and medium for human intent recognition

Abstract

This invention provides a multimodal fusion method, system, device, and medium for human intent recognition, belonging to the fields of human-machine collaboration and intelligent manufacturing technology. The method includes: in a human-machine collaboration scenario, acquiring RGB video data of an operator through a visual sensor and inertial sensing data through an inertial sensor, serving as the operator's intent data; selecting data from the operator's intent data, preprocessing it, and generating training and testing sets; constructing a human intent recognition model that fuses RGB video and inertial sensing modalities based on a decision-level fusion strategy or a feature-level fusion strategy, and training the human intent recognition model using an action sample training set; inputting the action sample testing set into the human intent recognition model to perform human intent recognition, obtaining the human intent recognition result. This invention utilizes decision-level and feature-level fusion strategies, combining the advantages of two modalities of data, thus improving the accuracy of human intent recognition.

Description

Technical Field

[0001] This invention belongs to the field of human-machine collaboration and intelligent manufacturing technology, and more specifically relates to a multimodal fusion method, system, device and medium for human intent recognition. Background Technology

[0002] Human-robot collaboration, with its high flexibility, efficiency, adaptability, personalization, and safety, has become a significant driving force for industrial transformation towards intelligent manufacturing. In human-robot collaboration, the high precision, strength, and repeatability of robots are combined with the high flexibility and adaptability of operators to achieve optimal productivity. One of the keys to achieving this goal is the robot's ability to dynamically plan safe responses based on the operator's intentions. Therefore, human intention recognition, as a prerequisite for dynamic robot decision-making, is crucial for achieving efficient human-robot manufacturing collaboration, including human-robot collaborative assembly and human-robot collaborative welding.

[0003] Currently, person intent recognition is mainly achieved through unimodal methods. In particular, RGB video-based and inertial sensor-based person intent recognition are the two main approaches. RGB video data is suitable for visually sensitive person recognition, while inertial sensor data is suitable for fine-grained person recognition. However, despite the considerable progress these two unimodal methods have made in person intent recognition, they each have significant limitations.

[0004] 1. Person Intent Recognition Based on RGB Video: Video can capture complex human actions and contextual information, but this method is limited by physical factors such as limited field of view (occlusions, viewing angle limitations), lighting environment, and complex backgrounds. In particular, when the person is not within the camera's field of view, intent recognition cannot be determined.

[0005] 2. HAR based on wearable inertial sensors: The wearability, versatility, and independence of inertial sensors make them unrestricted by specific field of view and lighting conditions, overcoming the limitations of video-based methods. However, this method mainly captures human movement and posture information and cannot provide rich contextual information.

[0006] Therefore, single-modal sensing cannot fully encompass all information related to human intent recognition. In human-machine manufacturing collaboration, operators not only exhibit complex movements, but sometimes these movements also possess subtle, similar motion patterns (e.g., inserting a pin, tightening a bolt) or the same motion pattern occurring in different operational sequences (which, from a manufacturing perspective, belong to two types of motion intent). In this application scenario, human intent recognition based on single-modal data cannot comprehensively understand human intent, resulting in poor performance in human intent recognition. Summary of the Invention

[0007] To address the above problems, the present invention aims to provide a multimodal fusion method, system, device, and medium for human intent recognition. By utilizing decision-level fusion and feature-level fusion strategies, it integrates the advantages of RGB video and inertial sensing modalities, overcomes the limitations of single-modality human intent recognition, and improves the accuracy of human intent recognition in human-machine collaborative manufacturing scenarios.

[0008] To achieve the above objectives, the present invention employs the following technical solution:

[0009] In a first aspect, embodiments of this application provide a multimodal fusion method for recognizing human intent, including:

[0010] In human-machine collaboration scenarios, RGB video data of the operator is collected through a visual sensor, and inertial sensing data is collected through a wearable inertial sensor as the operator's intention data; the inertial sensing data includes, but is not limited to: acceleration, angular velocity, joint quaternion, and center of mass displacement.

[0011] Data is selected from the operator's intention data, and after preprocessing, a training set and a test set of action samples are generated.

[0012] Based on a decision-level fusion strategy or a feature-level fusion strategy, a human intention recognition model that fuses RGB video and inertial sensing modalities is constructed, and the human intention recognition model is trained using an action sample training set.

[0013] The action sample test set is input into the human intention recognition model to perform human intention recognition and obtain the human intention recognition result.

[0014] In an optional implementation, the step of selecting operator intent data, preprocessing it, and generating an action sample training set and an action sample test set includes:

[0015] Select RGB video data of a preset duration, divide the selected RGB video data into multiple video sequences of equal length, select one frame image from each video sequence, and crop the image to a fixed size to generate an image sequence as preprocessed RGB video data;

[0016] Select inertial sensing data collected within a preset time period and organize the selected inertial sensing data into an action sequence X={x1, x... 2, ..., x T}, as preprocessed inertial sensing data; where T represents the period length of the action sequence, x t It represents the personnel behavior state at time step t, where each x t It contains m basic features, x t =(x t1 , xt2 , …,x tm );

[0017] The pre-processed RGB video data and pre-processed inertial sensor data are used as a motion sample;

[0018] Select M action samples as the action sample training set and N action samples as the action sample test set.

[0019] In an optional implementation, the step of constructing a person intention recognition model based on decision-level fusion strategy or feature-level fusion strategy by fusing RGB video and inertial sensing modalities includes:

[0020] A person intent recognition model based on the fusion of RGB video and inertial sensing modalities is constructed using a decision-level fusion strategy. The specific implementation process is as follows:

[0021] A first personnel intent recognition sub-model is constructed by modeling the temporal and spatial features of preprocessed RGB video data based on a 3D convolutional network; the first personnel intent recognition sub-model is used to generate a first operator intent recognition result based on the preprocessed RGB video data.

[0022] Based on a bidirectional long short-term memory-bidirectional gated loop-converter network, the temporal and global features of the preprocessed inertial sensing data are modeled to construct a second personnel intention recognition sub-model; the second personnel intention recognition sub-model is used to generate the second operator intention recognition result based on the preprocessed inertial sensing data;

[0023] A calculation module is constructed based on a decision fusion algorithm; the calculation module is used to obtain the first operator's intention recognition result and the second operator's intention recognition result, and to calculate the operator's intention recognition result using the decision fusion algorithm;

[0024] The decision fusion algorithm includes:

[0025]

[0026] Where RGB_output[i] is the first operator intent recognition result for the i-th action sample, IS_output[i] is the second operator intent recognition result for the i-th action sample, and W RGB W represents the weight of the first operator's intent recognition result. IS It is the weight of the result of the first operator's intent recognition. It is the result of the human intent recognition for the i-th action sample.

[0027] In an optional implementation, the step of constructing a person intention recognition model based on decision-level fusion strategy or feature-level fusion strategy by fusing RGB video and inertial sensing modalities includes:

[0028] A human intent recognition model based on the fusion of RGB video and inertial sensor features was constructed using a feature-level fusion strategy based on a 3D convolutional network, a bidirectional long short-term memory-bidirectional gated recurrent converter network, and a feature analysis network.

[0029] The feature-level fusion strategy includes:

[0030] The preprocessed RGB video data is read through a 3D convolutional network, and after feature extraction, the video feature vector is extracted.

[0031] The preprocessed inertial sensor data is read through a bidirectional long short-term memory-bidirectional gated loop-converter network, and after feature extraction, the sensor feature vector is extracted.

[0032] The feature analysis network concatenates video feature vectors and sensor feature vectors into a fused feature vector, and generates personnel intent recognition results based on the fused feature vector.

[0033] In an optional implementation, the feature analysis network includes at least two fully connected layers and introduces nonlinearity through the ReLU activation function;

[0034] The feature analysis network is used to implement the following feature fusion algorithm:

[0035]

[0036]

[0037] Where Z represents the fused feature, [;] denotes the concatenation operation, W1 and W2 represent the weight matrices of the two fully connected layers, and b1 and b2 are bias vectors. For video feature vectors, For sensing feature vectors, The result is for identifying the intent of the person.

[0038] In an optional implementation, the first person intent recognition sub-model includes: a C3D core module and an output module;

[0039] The C3D core module is used to capture spatiotemporal features representing operator activities in preprocessed RGB video information by performing 3D convolution and 3D pooling operations. The C3D core module includes five 3D sub-modules, each of which includes one or two 3D convolutional layers and one 3D max pooling layer.

[0040] The output module is used to map the spatiotemporal features to the classification space by deploying fully connected layers, generating a score distribution covering the overall categories of human actions; the output module includes two fully connected layers and one dropout layer.

[0041] In an optional implementation, the second person intent recognition sub-model includes a BiLSTM-BiGRU module, a Transformer module, and an output module;

[0042] The BiLSTM-BiGRU module is used to capture the temporal characteristics of human behavior in preprocessed inertial sensing data; the BiLSTM-BiGRU module contains a stacked bidirectional long short-term memory network and a bidirectional gated loop unit;

[0043] The Transformer module contains two stacked Transformer encoders used to capture long-term dependencies of the temporal features and enhance global features, generating enhanced features.

[0044] The output module contains a fully connected layer that maps the enhanced features generated by the Transformer module to the classification space, generating scores that cover all categories of human actions.

[0045] Secondly, embodiments of this application also provide a multimodal fusion-based human intent recognition system, including:

[0046] The multimodal data acquisition module is used to collect RGB video data of the operator through a visual sensor and inertial sensing data through a wearable inertial sensor in human-machine collaboration scenarios, as the operator's intention data; the inertial sensing data includes, but is not limited to: acceleration, angular velocity, joint quaternion and center of mass displacement.

[0047] The multimodal data preprocessing module is used to select data from the operator's intention data. After preprocessing, it generates a training set of action samples and a test set of action samples.

[0048] The personnel intention recognition model construction module is used to construct a personnel intention recognition model based on decision-level fusion strategy or feature-level fusion strategy, which fuses RGB video and inertial sensing modalities, and to train the personnel intention recognition model using an action sample training set.

[0049] The personnel intent recognition module is used to input the action sample test set into the personnel intent recognition model to perform personnel intent recognition and obtain personnel intent recognition results.

[0050] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal fusion method for human intent recognition as described in any of the above.

[0051] Fourthly, embodiments of this application also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the multimodal fusion method for human intent recognition as described in any of the above claims.

[0052] As can be seen from the above technical solutions, the present invention has the following advantages:

[0053] The multimodal fusion method for personnel intent recognition provided in this application presents two fusion strategies: decision-level fusion and feature-level fusion. By combining the recognition advantages of the two modalities, the limitations of single-modal personnel intent recognition (personnel image recognition based on RGB video or personnel intent recognition based on inertial sensing) can be overcome, and accurate recognition of operator intent can be achieved in human-machine manufacturing collaboration scenarios.

[0054] This application employs decision-level fusion of RGB video and inertial sensing models. From a decision-level fusion perspective, it integrates the probabilistic decision results of two modal branch models to improve the accuracy of human intent recognition. The RGB video and inertial sensing modalities proceed along two independent paths, involving data acquisition, data preprocessing, feature extraction, and decision / classification recognition. Finally, probabilistic decision fusion technology is used to fuse the decision results of the two modalities together, generating a more accurate recognition result for the operator's intent.

[0055] This application can also improve the accuracy of human intent recognition by fusing the feature vectors of the two modal branches through RGB video and inertial sensing feature-level fusion. The data acquisition, preprocessing, and feature extraction of the two branches, as well as their feature fusion and feature analysis (decision recognition), constitute a complete feature fusion model, generating more accurate recognition results for operator intent.

[0056] This application comprehensively utilizes visual sensors and wearable inertial sensors to collect RGB video data and inertial sensing data from operators, respectively, as key information sources for intent recognition. In the data processing stage, the method cleverly selects and preprocesses these data to generate action sample sets for training and testing. Subsequently, based on decision-level or feature-level fusion strategies, a human intent recognition model fusing RGB video and inertial sensing modalities is constructed. This model not only fully leverages the complementary advantages of the two modalities but also employs advanced deep learning algorithms, such as 3D convolutional networks and bidirectional long short-term memory-bidirectional gated recurrent-transformer networks, to accurately identify the operator's intent. This application significantly improves the accuracy and robustness of intent recognition, providing strong support for the intelligent and efficient human-machine collaboration. Attached Figure Description

[0057] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 A flowchart illustrating the multimodal fusion method for human intent recognition provided in this application.

[0059] Figure 2 A flowchart illustrating the decision-level fusion strategy provided in this application.

[0060] Figure 3 A flowchart illustrating the feature-level fusion strategy provided in this application.

[0061] Figure 4 A flowchart illustrating another multimodal fusion method for human intent recognition provided in this application.

[0062] Figure 5 A schematic diagram of the structure of the first person intent recognition sub-model provided in this application.

[0063] Figure 6 A schematic diagram of the structure of the second person intent recognition sub-model provided in this application.

[0064] Figure 7 This is a schematic diagram of the structure of the BiLSTM-BiGRU module provided in this application.

[0065] Figure 8 A schematic diagram of the structure of the personnel intent recognition model constructed based on the feature fusion strategy provided in this application.

[0066] Figure 9A comparison chart of the recognition results of different models (single-modal branch model and fusion model) provided for this application on the test dataset.

[0067] Figure 10 A schematic diagram of the structure of the multimodal fusion human intent recognition system provided in this application.

[0068] Figure 11 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0069] The various embodiments of this disclosure will be described more fully in the following detailed description of the specific steps of the multimodal fusion method for human intent recognition. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0070] In the following, the terms “comprising” or “may include”, which may be used in various embodiments of this disclosure, indicate the presence of the disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in various embodiments of this disclosure, the terms “comprising,” “having,” and their cognates are intended only to indicate a particular feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more combinations of the foregoing.

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] Please see Figure 1 The diagram shows a flowchart of a multimodal fusion method for recognizing human intent in a specific embodiment. The method includes:

[0073] S101: In a human-machine collaboration scenario, RGB video data of the operator is collected through a visual sensor, and inertial sensing data is collected through a wearable inertial sensor as the operator's intention data; the inertial sensing data includes, but is not limited to: acceleration, angular velocity, joint quaternion, and center of mass displacement.

[0074] In this step, inertial sensing data refers to the operator posture information that inertial sensors can capture in human-machine collaborative manufacturing scenarios, including acceleration, angular velocity, skeleton quaternion / Euler angle, joint quaternion / Euler angle, joint vector angle, and center of mass displacement.

[0075] S102: Select data from the operator's intention data, and after preprocessing, generate a training set and a test set of action samples.

[0076] In a specific implementation, the purpose of this step is to preprocess the operator's intention data. This includes: sampling, scaling, cropping, and normalizing the original RGB video data to ensure that the input RGB data meets the model's training requirements; normalizing and partitioning the original inertial sensor data to ensure that the input inertial sensor data meets the model's training requirements; and uniformly truncating the RGB video and inertial sensor data to ensure that the data length of all actions input to the model is consistent, since the duration of each operator's actions varies.

[0077] Specifically, for the preprocessing of RGB video data, firstly, RGB video data of a preset duration is selected, and the selected RGB video data is divided into multiple video sequences of equal length. From each video sequence, one frame is selected and the image is cropped to a fixed size to generate an image sequence, which serves as the preprocessed RGB video data.

[0078] For the preprocessing of inertial sensing data, firstly, inertial sensing data collected within a preset time period is selected, and the selected inertial sensing data is organized into an action sequence X={x1, x...} 2, ..., x T}, as preprocessed inertial sensing data; where T represents the period length of the action sequence, x t It represents the personnel behavior state at time step t, where each x t It contains m basic features, x t =(x t1 , x t2 , …,x tm ).

[0079] Then, the preprocessed RGB video data and preprocessed inertial sensor data are used as a motion sample. This allows for the creation of multiple different motion samples based on different operators.

[0080] At this point, select M action samples as the action sample training set and select N action samples as the action sample test set.

[0081] S103: Based on a decision-level fusion strategy or a feature-level fusion strategy, construct a human intention recognition model that fuses RGB video and inertial sensing modalities, and train the human intention recognition model using an action sample training set.

[0082] In this step, two fusion strategies can be used to construct a person intent recognition model that integrates RGB video and inertial sensing modalities, depending on different needs. Details are as follows:

[0083] 1. The decision-level fusion strategy refers to constructing a decision-fusion-based human intention recognition model using probabilistic decision fusion technology, which is used to fuse the human intention recognition results of two single-modal branch models. The two single-modal branch models refer to human intention recognition models constructed separately for RGB video and inertial sensing, respectively. This includes modeling the temporal and spatial features of preprocessed video data in the RGB video branch using a 3D convolutional network (C3D); and modeling the temporal and global features of preprocessed inertial sensing data in the inertial sensing branch using a bidirectional long short-term memory-bidirectional gated recurrent-transformer network (BiLSTM-BiGRU-Transformer).

[0084] It is important to note that probabilistic fusion techniques include methods such as majority voting, summation rule, Naive Bayes, weighted average, and DS evidence theory. The chosen method determines the decision fusion model for RGB video and inertial sensing.

[0085] As an example, the specific process of building a person intent recognition model using a decision-level fusion strategy is as follows:

[0086] On the one hand, a first personnel intention recognition sub-model is constructed by modeling the temporal and spatial features of the preprocessed RGB video data based on a three-dimensional convolutional network; the first personnel intention recognition sub-model is used to generate the first operator intention recognition result based on the preprocessed RGB video data.

[0087] Another method is to model the temporal and global features of the preprocessed inertial sensing data based on a bidirectional long short-term memory-bidirectional gated loop-converter network, and construct a second personnel intention recognition sub-model; the second personnel intention recognition sub-model is used to generate a second operator intention recognition result based on the preprocessed inertial sensing data.

[0088] Simultaneously, a calculation module is constructed based on a decision fusion algorithm; the calculation module is used to obtain the first operator's intention recognition result and the second operator's intention recognition result, and to calculate the operator's intention recognition result using the decision fusion algorithm.

[0089] The decision fusion algorithm includes:

[0090]

[0091] In the formula, RGB_output[i] represents the first operator intent recognition result for the i-th action sample, IS_output[i] represents the second operator intent recognition result for the i-th action sample, and W RGB W represents the weight of the first operator's intent recognition result. IS It is the weight of the result of the first operator's intent recognition. It is the result of the human intent recognition for the i-th action sample.

[0092] As can be seen, the personnel intent recognition model constructed using the decision-level fusion strategy includes: a first personnel intent recognition sub-model, a second personnel intent recognition sub-model, and a calculation module. The first and second personnel intent recognition sub-models each generate single-dimensional operator intent recognition results. The calculation module then uses probabilistic decision fusion technology to fuse the recognition results of the two sub-models, obtaining the final personnel intent recognition result.

[0093] 2. Feature-level fusion strategy refers to concatenating the feature vectors output by two single-modal branch networks on top of a single-modal branch network, and then connecting them to a feature analysis network to construct a feature-fused person intent recognition model. The feature analysis network can be a complex sub-neural network or a simple sub-neural network, such as two fully connected layers. The first fully connected layer receives the deeply fused feature vectors and then introduces non-linearity using the ReLU activation function. The second fully connected layer generates the final person intent recognition result.

[0094] As can be seen, the human intent recognition model constructed by the feature-level fusion strategy includes a three-dimensional convolutional network, a bidirectional long short-term memory-bidirectional gated recurrent-transformer network, and a feature analysis network.

[0095] As examples, feature-level fusion strategies include:

[0096] First, a 3D convolutional network is used to read preprocessed RGB video data, and after feature extraction, video feature vectors are extracted. Simultaneously, a bidirectional long short-term memory-bidirectional gated recurrent converter network is used to read preprocessed inertial sensor data, and after feature extraction, sensor feature vectors are extracted.

[0097] Then, the video feature vector and the sensor feature vector are concatenated into a fused feature vector through a feature analysis network, and the person intention recognition result is generated based on the fused feature vector.

[0098] The feature analysis network includes at least two fully connected layers and introduces nonlinearity through the ReLU activation function;

[0099] The feature analysis network is used to implement the following feature fusion algorithm:

[0100]

[0101]

[0102] In the formula, Z represents the fused feature, [;] denotes the concatenation operation, W1 and W2 represent the weight matrices of the two fully connected layers, and b1 and b2 are bias vectors. For video feature vectors, For sensing feature vectors, The result is for identifying the intent of the person.

[0103] S104: Input the action sample test set into the human intention recognition model to perform human intention recognition and obtain the human intention recognition result.

[0104] In a specific implementation, a test set of motion samples, including preprocessed RGB video data and inertial sensor data, or real-time data, is input into two modal branches of the human intention recognition model. The human intention is then recognized through a decision fusion-based human intention recognition model or a feature fusion-based human intention recognition model to obtain the final human intention recognition result.

[0105] In this embodiment, by fusing RGB video data collected by a visual sensor and inertial sensing data collected by a wearable inertial sensor, a decision-level or feature-level fusion model is constructed, which effectively improves the accuracy and reliability of operator intent recognition in human-machine collaboration scenarios.

[0106] In one embodiment of the present invention, based on the decision-level fusion strategy disclosed in step S103, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0107] like Figure 2 As shown, the decision-level fusion involves three independent models: a first person intent recognition sub-model that identifies intent using the RGB video branch, a second person intent recognition sub-model that identifies intent using the inertial sensing branch, and a computational module that fuses the decisions from the two branches. The RGB video and inertial sensing modes proceed along two independent paths, involving data acquisition, data preprocessing, feature extraction, and decision / classification recognition. Finally, the decision results from the two modes are fused together to generate a more accurate recognition result for person intent. The final result of the decision fusion model depends only on the decision results of the single-modal branch model and is unaffected by its internal state.

[0108] Specifically, the human intent recognition that achieves decision-level fusion of RGB video and inertial sensing includes:

[0109] Based on visual sensors (cameras) and wearable inertial sensors, the intention data of operators in human-machine manufacturing collaboration scenarios are collected, including parts assembly actions, welding actions, etc.

[0110] The operator intent data is preprocessed to generate feasible preprocessed data;

[0111] Based on a 3D convolutional network (C3D), a human intention recognition model for RGB video is constructed, namely the first human intention recognition sub-model; based on BiLSTM-BiGRU-Transformer, a human intention recognition model for inertial sensing is constructed, namely the second human intention recognition sub-model.

[0112] The preprocessed RGB data is input into the first human intention recognition sub-model to obtain the understanding of the operator's intention by the video branch, i.e., the operator intention recognition result; the preprocessed inertial sensing data is input into the second human intention recognition sub-model to obtain the understanding of the operator's intention by the inertial sensing branch, i.e., the operator intention recognition result.

[0113] By using probabilistic decision fusion technology, the recognition results of the two branch models are combined to obtain the final result of human intent recognition.

[0114] In one embodiment of the present invention, based on the feature-level fusion strategy disclosed in step S103, a possible embodiment will be given below, and its specific implementation will be described in a non-limiting manner.

[0115] like Figure 3 As shown, feature-level fusion involves an associated fusion model. The final result of the model is influenced not only by the fused features but also by the internal state of the branch models. The data acquisition, preprocessing, and feature extraction of the two branches, along with their feature fusion and feature analysis (decision recognition), constitute a complete feature fusion model, generating more accurate recognition results for operator behavior. The branch models can be fine-tuned based on the gradient feedback information of the entire model, thereby improving the overall performance of the model.

[0116] Specifically, the human intent recognition that achieves feature-level fusion of RGB video and inertial sensing includes:

[0117] Based on visual sensors (cameras) and wearable inertial sensors, the intention data of operators in human-machine manufacturing collaboration scenarios are collected, including parts assembly actions, welding actions, etc.

[0118] The operator intent data is preprocessed to generate feasible preprocessed data;

[0119] A human intent recognition model is constructed by fusing RGB video and inertial sensing features based on a 3D convolutional network (C3D), BiLSTM-BiGRU-Transformer, and feature analysis network.

[0120] The preprocessed RGB data and inertial sensing data are input into the human intention recognition model that fuses RGB video and inertial sensing features to perform human intention recognition and obtain the final human intention recognition result.

[0121] The process of obtaining the person's intention recognition result based on the feature-level fusion model of the RGB video and inertial sensing includes:

[0122] The preprocessed RGB data is input into the C3D model to obtain the output vector of the video branch, which is the feature vector of the video branch's understanding of the operator's intention;

[0123] Meanwhile, the preprocessed inertial sensing data is input into the BiLSTM-BiGRU-Transformer model to obtain the output vector of the inertial sensing branch, which is the feature vector of the inertial sensing branch's understanding of the operator's intention.

[0124] The feature vectors of the two branches are concatenated or combined to form a feature vector that is deeply fused between the two modalities;

[0125] The deeply fused feature vectors are passed to the feature analysis network to generate the final human intent recognition result.

[0126] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, and to fully illustrate the specific implementation process in this embodiment, taking the assembly of an RV reducer as an example, another modal fusion method for human intent recognition is provided, such as... Figure 4 As shown, the method includes the following steps:

[0127] S201: Determine the type of assembly action for the operator based on the RV reducer assembly task.

[0128] Specifically, the assembly actions of the RV reducer are divided into 13 categories according to the assembly sequence, namely actions 1-13: install large shims; lubricate needle roller bearings; install small shims; install snap rings; lubricate main bearings; assemble output shaft and main bearings; install large shims; assemble support flange and main bearings; install tapered pins; lubricate needle roller bearings; install limit caps; install small bolts; install large bolts.

[0129] S202: Configure video sensors and inertial sensors according to the assembly task and assembly layout.

[0130] Specifically, the operator's assembly activities were within the field of view of the vision sensor, and RGB video of the operator's assembly movements was captured. Simultaneously, seven wearable inertial sensors were fixed to the operator's upper limbs to collect assembly posture information, including acceleration, angular velocity, and joint vector angles. The video sensor sampling rate was 30 frames / s, and the inertial sensor sampling frequency was 100 Hz. In this embodiment, the movements of seven individuals were collected, with each person repeating each movement 12 times, resulting in 84 data sets per category, totaling 1092 samples.

[0131] S203: Collect RGB video data and inertial sensor data, and construct a motion sample training set and a motion sample test set after preprocessing.

[0132] Specifically, the original RGB video data was sampled and image cropped; the original inertial sensor data was normalized and partitioned; the RGB video and inertial sensor data were standardized in terms of time length, and the first 2 seconds of data were selected for subsequent model training and testing. Data collected from 5 people was selected as the action sample training set, and data from another 2 people was selected as the action sample test set.

[0133] In this step, sampling refers to preprocessing the raw RGB video data using a frame-skipping strategy. Specifically, the raw video is divided into a series of equal sequences. Then, one frame is selected from each sequence. Finally, each raw video can be divided into non-overlapping T-frame segments as input to the network. In this embodiment, each raw video is sampled into a 15-frame video sequence, and the image size is uniformly cropped to 112×112. For data partitioning, this embodiment selects data collected from 5 people as the action sample training set, and data from another 2 people as the action sample test set.

[0134] Furthermore, since the duration of each person's operation varies, approximately between 6 and 10 seconds, a uniform time length was applied to both the RGB video and inertial sensor data. The first 2 seconds of data were selected for subsequent model training and testing. Therefore, within the set 2-second timeframe, 15 frames were sampled from each sample of RGB video data, and each frame was cropped to 112×112 pixels. For the inertial sensor data, only the first 2 seconds of data were extracted, meaning each sample contained 200 data points.

[0135] S204: Construct a human intent recognition model based on a decision fusion strategy or feature fusion strategy, and train the model.

[0136] Specifically, a human intention recognition model is constructed by fusing RGB video and inertial sensing modal decision-making / feature-based fusion. The action sample training set data is then input into the human intention recognition model for training.

[0137] In this step, the personnel intention recognition model constructed based on the decision fusion strategy includes a first personnel intention recognition sub-model constructed based on the C3D network and a second personnel intention recognition sub-model constructed based on the BiLSTM-BiGRU-Transformer network.

[0138] See Figure 5 The diagram shows the structure of the first personnel intention recognition sub-model constructed according to this invention. This model consists of two modules: a C3D core module and an output module. The C3D core module primarily captures the spatiotemporal features representing operator activities in RGB video by performing 3D convolution and 3D pooling operations. The output module maps the extracted spatiotemporal features to a classification space by deploying fully connected layers, thereby obtaining a score distribution covering all personnel action categories.

[0139] The specific parameter configuration of the first-person intention recognition sub-model is shown in Table 1. The dimensions of the input RGB video data are C×L×H×W, where C is the number of RGB image channels, L is the number of frames extracted from each video segment, H is the image height, and W is the image width. In this embodiment, it is set to 3×15×112×112. The core module consists of five 3D sub-modules, each containing one or two 3D convolutional layers and one 3D max-pooling layer. The output module consists of two fully connected layers and one dropout layer. The output of the last 3D max-pooling layer is flattened and passed to the first fully connected layer. Based on ReLU activation, the output of the first fully connected layer is passed to the 50% dropout layer, and then connected to the final fully connected layer to generate the recognition result of the operator's behavior. The number of filters in the 3D convolutional layers of the five 3D sub-modules are 64, 128, 256, 512, and 512, respectively. All 3D convolutional filters are 3×3×3 with a stride of 1×1×1. Except for the first layer, all 3D max-pooling layers are 2×2×2 with a stride of 2×2×2. The first pooling layer has a 1×2×2 kernel with a stride of 1×1×1. The two fully connected layers have 4096 output units.

[0140] Table 1: Specific Parameter Information of the First Person Intent Recognition Submodel

[0141]

[0142] See Figure 6The diagram illustrates the structure of the second person intention recognition sub-model constructed according to an embodiment of the present invention. This model consists of three core modules: a BiLSTM-BiGRU module, a Transformer module, and an output module. In the BiLSTM-BiGRU module, BiLSTM and BiGRU are stacked together to capture the temporal features representing person behavior in inertial sensing signals. In the Transformer module, Transformer encoders are stacked to capture long-term dependencies and enhance global features. Simultaneously, to mitigate the risk of overfitting during model training, a 50% dropout layer is introduced after this module. The output module maps the Transformer-enhanced features to the classification space by deploying a fully connected layer, thereby obtaining a score covering all person action categories.

[0143] In the BiLSTM-BiGRU module, the inertial sensing signals characterizing human behavior are typically time-series data. BiLSTM and BiGRU consist of two memory networks with opposite propagation directions, which can capture forward (historical) and backward (future) information of the sequence data.

[0144] Therefore, in order to fully extract the temporal characteristics of inertial sensing data, the BiLSTM-BiGRU module stacks two layers of BiLSTM and two layers of BiGRU. (See [link]). Figure 7 The details are as follows:

[0145] Suppose the preprocessed action sequence is X = {x1, x2, ..., x...} T}, where T represents the period length of the sequence. x t (tϵ{1, 2, ..., T}) represents the personnel behavior state at time step t, where each x t It contains m basic features, including acceleration, angular velocity, and joint vector angle, specifically x t =(x t1 , x t2 , …,x tm Accordingly, after the BiLSTM-BiGRU model processes the entire sequence X, the overall features F={f1, f2, ..., fT} are obtained. In the BiLSTM-BiGRU model of this embodiment, both BiLSTM and BiGRU have 2 layers and a hidden dimension of 128. The first layer of BiLSTM receives the preprocessed inertial sensing data, and the output of the second layer of BiLSTM is the input of the first layer of BiGRU.

[0146] The core of the Transformer module is its self-attention mechanism, which can process the input sequence in parallel, effectively capturing long-term dependencies and global information, offering significant advantages in handling long sequences and large-scale data. Therefore, to capture long-term dependencies and enhance global features, two Transformer encoder modules are stacked, and their specific implementation process is as follows:

[0147] Taking the first-layer Transformer encoder as an example, its embedding dimension (d_model) is 128, corresponding to the output dimension of the second-layer BiGRU. The temporal features generated by the BiLSTM BiGRU module are passed to the multi-head self-attention mechanism of the first Transformer encoder layer. In this mechanism, these features are divided into four parts, each processed independently by an attention head. Finally, the outputs of the four attention heads are concatenated and linearly transformed to produce enhanced temporal features with the same dimension as the output of the second-layer BiGRU.

[0148] Specifically, in implementing the decision fusion strategy, this invention selects a weighted average method for decision-level fusion of RGB video and inertial sensing modes; that is, the weighted average method serves as the decision fusion model for both modes. For each sample, the calculation formula for the decision fusion result is as follows:

[0149]

[0150] Where IS_output[i] and RGB_output[i] represent the decision results of the i-th sample in the RGB video and inertial sensing branches, respectively. W IS and W RGB These are the weights of the two branch results. In this embodiment of the invention, both weights are 0.5.

[0151] In this step, the person intent recognition model constructed based on the feature fusion strategy includes a 3D convolutional network, a bidirectional long short-term memory-bidirectional gated recurrent-transformer network, and a feature analysis network.

[0152] The feature analysis network can be a complex sub-neural network or a simple sub-neural network, such as two fully connected layers. (See [reference]). Figure 8 The diagram illustrates a person intention recognition model constructed based on a feature fusion strategy according to an embodiment of the present invention. First, the decision results from the RGB video and the inertial sensing branch are obtained as feature information. Then, the output features of these two branches are concatenated to form a fused feature. Subsequently, the fused feature is passed to a first fully connected layer with a hidden dimension of 128, and a ReLU activation function is applied to introduce non-linearity. Finally, a second fully connected layer is connected to generate the final person recognition result.

[0153] The calculation formula for feature fusion is as follows.

[0154]

[0155]

[0156] Where Z is the fused feature. [;] indicates the concatenation operation. W1 and W2 represent the weight matrices of the two fully connected layers, and b1 and b2 are the bias vectors.

[0157] In summary, in this step, for decision-level fusion, firstly, the decision recognition results for human actions are obtained using the BiLSTM-BiGRU-Transformer model in the inertial sensing branch and the C3D model in the RGB video branch, respectively. Then, a weighted average method is used to fuse the recognition results of the two branches to achieve the final human action recognition. For feature-level fusion, the branch models, the feature fusion of the two branches, and feature analysis (decision recognition) constitute a feature fusion model integrating RGB video and inertial sensing modalities. Therefore, the human intention recognition model for decision fusion has no training parameters; it is a weighted average calculation module.

[0158] In this step, the training parameters of the two branch sub-models used for personnel assembly action recognition and the personnel intention recognition model for feature fusion are shown in Table 2.

[0159] Table 2: Comparison of training parameters for different models of personnel assembly motion recognition

[0160]

[0161] S205: Input the data of the action sample test set into the trained personnel intention recognition model to perform personnel assembly action recognition and obtain the recognition result.

[0162] As can be seen, this invention compares the proposed method for human intent recognition by fusing RGB video and inertial sensing data with a single-modal intent recognition method. That is, it compares the recognition results of decision-level fusion or feature-level fusion of the two modalities with the recognition results of the C3D model of the RGB video branch and the BiLSTM-BiGRU-Transformer model of the inertial sensing branch.

[0163] Figure 9The results show the recognition outcomes of different models (single-modal branch model and fusion model) on the test dataset when the data length is 2 seconds. It can be seen that the fusion model integrating two modalities used in this invention produces better recognition accuracy than the single-modal branch model. When the data length is 2 seconds, the optimal recognition accuracy of the two fusion models reaches 94%: decision fusion (95.51%) and feature fusion (94.23%), both higher than the accuracy of the RGB branch model (91.35%) and the inertial sensing branch model (79.49%). Therefore, the RGB video and inertial sensing fusion method for human intention recognition proposed in this invention can overcome the limitations of single-modality methods and significantly improve the robustness of human intention recognition. The RGB video modality has the ability to capture contextual information of the manufacturing scene, while the inertial sensing modality excels at capturing detailed operator postures. The two complement each other and can provide strong support for human intention recognition in human-machine manufacturing collaboration scenarios.

[0164] like Figure 10 As shown, the following are embodiments of the multimodal fusion human intention recognition system provided in this disclosure. This system and the multimodal fusion human intention recognition method of the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the multimodal fusion human intention recognition system, please refer to the embodiments of the multimodal fusion human intention recognition method described above.

[0165] A multimodal fusion system for recognizing human intent includes: a multimodal data acquisition module, a multimodal data preprocessing module, a human intent recognition model construction module, and a human intent recognition module.

[0166] The multimodal data acquisition module is used in human-machine collaboration scenarios to collect RGB video data of the operator through a visual sensor and inertial sensing data through a wearable inertial sensor, serving as the operator's intent data. The inertial sensing data includes, but is not limited to, acceleration, angular velocity, joint quaternions, and center-of-mass displacement. The intent data may include part assembly actions, welding actions, etc.

[0167] The multimodal data preprocessing module is used to select data from the operator's intention data. After preprocessing, it generates a training set of action samples and a test set of action samples.

[0168] The personnel intention recognition model construction module is used to construct a personnel intention recognition model based on decision-level fusion strategy or feature-level fusion strategy, which fuses RGB video and inertial sensing modalities, and to train the personnel intention recognition model using an action sample training set.

[0169] The personnel intent recognition module is used to input the action sample test set into the personnel intent recognition model to perform personnel intent recognition and obtain personnel intent recognition results.

[0170] The multimodal fusion-based human intention recognition system provided in this embodiment fully leverages the complementary advantages of visual and motion information by integrating RGB video data and inertial sensing data. Based on decision-level or feature-level fusion strategies, an accurate human intention recognition model is constructed, effectively improving the accuracy and real-time performance of operator intention recognition in human-computer collaboration scenarios, and providing strong support for intelligent human-computer interaction.

[0171] Figure 11 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0172] The multimodal fusion method for human intent recognition provided in this application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure described in the embodiments of this invention does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0173] Electronic devices may include processors, external memory interfaces, internal memory, universal serial bus (USB) interfaces, charging management modules, power management modules, batteries, wireless communication modules, audio modules, speakers, microphones, sensor modules, buttons, cameras, displays, and SIM card interfaces, etc.

[0174] A processor may include one or more processing units, such as: a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0175] The processor can serve as the nerve center and command center of an electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0176] The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0177] An external storage interface (ESI) can be used to connect external memory cards, such as microSD cards, to expand the storage capacity of electronic devices. The external memory card communicates with the processor through the ESI to perform data storage functions, such as saving music and video files on the external memory card.

[0178] Internal memory can be used to store computer executable program code, which includes instructions. The processor executes various functional applications and data processing of electronic devices by running the instructions stored in internal memory. Internal memory can include a program storage area and a data storage area. Internal memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0179] Wireless communication functionality in electronic devices can be achieved through antennas, wireless communication modules, modem processors, and baseband processors.

[0180] Wireless communication modules can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0181] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0182] Electronic devices can achieve shooting functions through ISPs, cameras, video codecs, GPUs, displays, and application processors.

[0183] Electronic devices can achieve display functions through GPUs, displays, and application processors.

[0184] A GPU is a microprocessor for image processing, connected to the display screen and application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0185] A display screen is used to display images, videos, etc. A display screen includes a display panel.

[0186] The aforementioned electronic device implements the multimodal fusion method for human intent recognition proposed in this application. By combining RGB video data and inertial sensing data, and utilizing decision-level or feature-level fusion strategies to construct a model, it achieves accurate recognition of the operator's intent. This method has significant advantages in human-machine collaboration scenarios, achieving the beneficial effect of greatly improving the accuracy and efficiency of intent recognition.

[0187] The storage medium provided in this application stores a program product capable of realizing a person intent recognition method through multimodal fusion.

[0188] The multimodal fusion method for human intent recognition includes: in a human-machine collaboration scenario, acquiring RGB video data of the operator through a visual sensor and inertial sensing data through a wearable inertial sensor, which serve as the operator's intent data; selecting data from the operator's intent data, and generating an action sample training set and an action sample test set after preprocessing; constructing a human intent recognition model that fuses RGB video and inertial sensing modalities based on a decision-level fusion strategy or a feature-level fusion strategy, and training the human intent recognition model using the action sample training set; inputting the action sample test set into the human intent recognition model to perform human intent recognition, and obtaining the human intent recognition result.

[0189] In some possible implementations, the multimodal fusion method for identifying human intent disclosed herein can be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0190] The storage medium disclosed herein may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0191] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal fusion method for recognizing human intent, characterized in that, include: In human-machine collaboration scenarios, RGB video data of the operator is collected through a visual sensor, and inertial sensing data is collected through a wearable inertial sensor as the operator's intention data; the inertial sensing data includes, but is not limited to: acceleration, angular velocity, joint quaternion, and center of mass displacement. Data is selected from the operator's intention data, and after preprocessing, a training set and a test set of action samples are generated. Based on a decision-level fusion strategy or a feature-level fusion strategy, a human intention recognition model that fuses RGB video and inertial sensing modalities is constructed, and the human intention recognition model is trained using an action sample training set. The action sample test set is input into the human intention recognition model to perform human intention recognition and obtain the human intention recognition result; The method for constructing a person intention recognition model based on decision-level fusion strategy or feature-level fusion strategy, which integrates RGB video and inertial sensing modalities, includes: A person intent recognition model based on the fusion of RGB video and inertial sensing modalities is constructed using a decision-level fusion strategy. The specific implementation process is as follows: A first personnel intent recognition sub-model is constructed by modeling the temporal and spatial features of preprocessed RGB video data using a 3D convolutional network; the first personnel intent recognition sub-model is used to generate a first operator intent recognition result based on the preprocessed RGB video data. Based on a bidirectional long short-term memory-bidirectional gated loop-converter network, the temporal and global features of the preprocessed inertial sensing data are modeled to construct a second personnel intention recognition sub-model; the second personnel intention recognition sub-model is used to generate the second operator intention recognition result based on the preprocessed inertial sensing data; A calculation module is constructed based on a decision fusion algorithm; the calculation module is used to obtain the first operator's intention recognition result and the second operator's intention recognition result, and to calculate the operator's intention recognition result using the decision fusion algorithm; The decision fusion algorithm includes: Where RGB_output[i] is the first operator intent recognition result for the i-th action sample, IS_output[i] is the second operator intent recognition result for the i-th action sample, and W RGB W represents the weight of the first operator's intent recognition result. IS It is the weight of the result of the first operator's intent recognition. It is the result of the personnel intent recognition for the i-th action sample; A human intent recognition model based on the fusion of RGB video and inertial sensor features was constructed using a feature-level fusion strategy based on a 3D convolutional network, a bidirectional long short-term memory-bidirectional gated recurrent converter network, and a feature analysis network. The feature-level fusion strategy includes: The preprocessed RGB video data is read through a 3D convolutional network, and after feature extraction, the video feature vector is extracted. The preprocessed inertial sensor data is read through a bidirectional long short-term memory-bidirectional gated loop-converter network, and after feature extraction, the sensor feature vector is extracted. The feature analysis network concatenates video feature vectors and sensor feature vectors into a fused feature vector, and generates personnel intent recognition results based on the fused feature vector.

2. The multimodal fusion method for human intent recognition according to claim 1, characterized in that, The process of selecting and preprocessing the operator's intention data to generate a training set and a test set of action samples includes: Select RGB video data of a preset duration, divide the selected RGB video data into multiple video sequences of equal length, select one frame image from each video sequence, and crop the image to a fixed size to generate an image sequence as preprocessed RGB video data; Select inertial sensing data collected within a preset time period and organize the selected inertial sensing data into an action sequence X={x1,x2}. 2, ..., x T }, as preprocessed inertial sensing data; where T represents the period length of the action sequence, x t It represents the personnel behavior state at time step t, where each x t It contains m basic features, x t =(x t1 , x t2 , …,x tm ); The pre-processed RGB video data and pre-processed inertial sensor data are used as a motion sample; Select M action samples as the action sample training set and N action samples as the action sample test set.

3. The multimodal fusion method for human intent recognition according to claim 1, characterized in that, The feature analysis network includes at least two fully connected layers and introduces nonlinearity through the ReLU activation function; The feature analysis network is used to implement the following feature fusion algorithm: Where Z represents the fused feature, [;] denotes the concatenation operation, W1 and W2 represent the weight matrices of the two fully connected layers, and b1 and b2 are bias vectors. For video feature vectors, For sensing feature vectors, The result is for identifying the intent of the person.

4. The multimodal fusion method for human intent recognition according to claim 1, characterized in that, The first person intent recognition sub-model includes: a C3D core module and an output module; The C3D core module is used to capture spatiotemporal features representing operator activities in preprocessed RGB video information by performing 3D convolution and 3D pooling operations. The C3D core module includes five 3D sub-modules, each of which includes one or two 3D convolutional layers and one 3D max pooling layer. The output module is used to map the spatiotemporal features to the classification space by deploying fully connected layers, generating a score distribution covering the overall categories of human actions; the output module includes two fully connected layers and one dropout layer.

5. The multimodal fusion method for human intent recognition according to claim 1, characterized in that, The second person intent recognition sub-model includes a BiLSTM-BiGRU module, a Transformer module, and an output module; The BiLSTM-BiGRU module is used to capture the temporal characteristics of human behavior in preprocessed inertial sensing data; the BiLSTM-BiGRU module contains a stacked bidirectional long short-term memory network and a bidirectional gated loop unit; The Transformer module contains two stacked Transformer encoders used to capture long-term dependencies of the temporal features and enhance global features, generating enhanced features. The output module contains a fully connected layer that maps the enhanced features generated by the Transformer module to the classification space, generating scores that cover all categories of human actions.

6. A multimodal fusion system for recognizing human intent, characterized in that, The system employs the multimodal fusion method for human intent recognition as described in any one of claims 1 to 5; The system includes: The multimodal data acquisition module is used to collect RGB video data of the operator through a visual sensor and inertial sensing data through a wearable inertial sensor in human-machine collaboration scenarios, as the operator's intention data; the inertial sensing data includes, but is not limited to: acceleration, angular velocity, joint quaternion and center of mass displacement. The multimodal data preprocessing module is used to select data from the operator's intention data. After preprocessing, it generates a training set of action samples and a test set of action samples. The personnel intention recognition model construction module is used to construct a personnel intention recognition model based on decision-level fusion strategy or feature-level fusion strategy, which fuses RGB video and inertial sensing modalities, and to train the personnel intention recognition model using an action sample training set. The personnel intent recognition module is used to input the action sample test set into the personnel intent recognition model to perform personnel intent recognition and obtain personnel intent recognition results.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal fusion method for human intent recognition as described in any one of claims 1 to 5.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal fusion method for human intent recognition as described in any one of claims 1 to 5.

Images