Multi-sensor fusion human intention recognition method, system, electronic device and medium

Abstract

The embodiment of the present invention provides a multi-sensor fusion human intention recognition method, system, electronic device and medium, which belongs to the field of human-machine collaboration and intelligent manufacturing. The method includes: using inertial sensors worn on the upper limbs of personnel to collect continuous action signals and perform online segmentation to obtain a set of candidate segmentation points; based on information gain, the continuous action signals are divided into segments with minimum entropy to determine the optimal segmentation point in the set of candidate segmentation points, and the legitimacy of the segmented action segments is checked based on meta-actions; based on a fully convolutional bidirectional long short-term memory network, the dependency relationship between the action segments is determined, and the behavioral intention of the personnel is identified by matching according to the action library. By combining the action library and the fully convolutional bidirectional long short-term memory network, the segmentation of complex continuous human actions in human-machine collaboration scenarios is realized, the accuracy and versatility of personnel intention recognition in the process of human-machine collaboration are improved, and the basis for the real-time implementation and later optimization of human-machine collaboration tasks is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of human-machine collaboration and intelligent manufacturing, and in particular to a multi-sensor fusion human intention recognition method, system, electronic equipment and medium. Background Art

[0002] Collaborative robots have good application prospects in the fields of intelligent manufacturing, smart home, medical health, etc. due to their flexible deployment, safe design, and reliable performance, and have become one of the important research directions in the field of robotics at this stage. Human-machine interaction is an important prerequisite for achieving human-machine collaborative tasks. Collaborative robots need to efficiently and accurately infer human behavioral intentions in various scenarios and respond to subsequent actions in advance, which is of great significance for the efficient and smooth execution of human-machine collaborative tasks.

[0003] At present, human intention recognition methods in human-machine collaboration scenarios mainly rely on visual sensors, but such sensors have great restrictions on scene layout and the range of personnel activities. If the background of the work site is complex, occlusion, viewpoint changes and different lighting conditions will inevitably affect the use effect. Human continuous action segmentation and intention recognition methods that only use inertial sensors are mostly offline and the application scenarios are relatively simple. They are mostly repeated behavior mode switching or there are obvious boundaries between actions. The algorithms involved are highly general but less targeted, and there is a lack of research on the online segmentation of complex continuous actions, which makes it difficult to directly apply them to various manufacturing and service industry scenarios. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a multi-sensor fusion human intention recognition method, system, electronic device and medium, which are used to fully or at least partially solve the technical problems existing in the above-mentioned prior art.

[0005] In order to achieve the above object, an embodiment of the present invention provides a multi-sensor fusion human intention recognition method, which is applied to a human-machine collaboration process. The multi-sensor fusion human intention recognition method includes:

[0006] Using an inertial sensor worn on the upper limbs of a person to collect continuous motion signals, and performing online segmentation on the continuous motion signals to obtain a set of candidate segmentation points;

[0007] An online hybrid segmentation method based on information gain is used to divide the continuous action signal into a preset number of segments with minimum entropy to determine the optimal segmentation point in the candidate segmentation point set, and the legitimacy of the segmented action segments is checked based on predefined meta-actions;

[0008] Based on the pre-built fully convolutional bidirectional long short-term memory network, the dependencies between the segmented action fragments are determined, and matched according to the pre-built action library to identify the behavioral intentions of people.

[0009] Optionally, an inertial sensor with a built-in three-axis accelerometer, three-axis gyroscope, and three-axis magnetometer is pre-worn on the upper arm, forearm, and hand of the personnel;

[0010] When people perform actions, they use inertial sensors to collect continuous action signals in real time, and then clean the continuous action signals and convert them into joint vector angle data and Euler angle data.

[0011] Based on the action types involved in the human-machine collaboration process, the continuous action signals are defined and decomposed, where the continuous action sequence is defined. ,in It is an indivisible meta-action, and the meta-action is defined based on the Euler angle data as ,in is the continuous action sequence collected by the i-th inertial sensor.

[0012] Optionally, an inertial sensor worn on an upper limb of a person is used to collect a continuous motion signal, and the continuous motion signal is segmented online to obtain a set of candidate segmentation points, including:

[0013] The collected joint vector angle data are processed by online piecewise linear representation to obtain a preliminary set of candidate action segmentation points. The online piecewise linear representation uses the maximum vertical distance metric as the segmentation criterion to reduce time complexity.

[0014] Optionally, an online hybrid segmentation method based on information gain is used to divide the continuous action signal into a preset number of segments with minimum entropy to determine the optimal segmentation point in the candidate segmentation point set, and perform a legitimacy check on the segmented action segments based on predefined meta-actions, including:

[0015] A detection window with a fixed step length starting from the initial point of the time series is predefined, and the point that reduces the entropy of the time series the most in the candidate segmentation point set is identified through the cost function of the time series information gain, and is used as the optimal segmentation point.

[0016] Optionally, the cost function is as follows:

[0017] ;

[0018] ;

[0019] In the formula, After the time series is divided j part, is the length operator, X is a multidimensional time series, k is the set of candidate segmentation fragments in the previous joint vector angle segmentation process, is the entropy of the current time series, which has a constant value in this segment. For fragment The entropy of m is the number of groups of total data points in the current time series, is the first The probability of the segment area divided by the time series X.

[0020] Optionally, the definition of a meta-action includes:

[0021] Based on the Euler angle of bones, define the corresponding relationship between the Euler angles of the child bones connected to the upper limb joints of the personnel and the parent bones. For the shoulder joint and elbow joint of the personnel, define the rotation order of the Euler angle as yxz;

[0022] The positive and negative values ​​of the three types of skeletal Euler angles correspond to different postures of the joints, where different postures are represented by abduction and adduction, flexion and extension, and external rotation and internal rotation of the joints. The three types of skeletal Euler angles represent the Euler angles of the shoulder joint, elbow joint, and wrist joint.

[0023] Optionally, the action library consists of three parts: an action template entry script, an assembly element action template library, and an assembly action sequence library. The action template entry script repeatedly enters actions of the same category and performs unified preprocessing. The processed actions are saved as action templates of this type for training. The assembly action sequence library contains flexible assembly sequences involved in assembly tasks to perform assembly sequence matching and predict the operator's intentions.

[0024] Optionally, based on a pre-built fully convolutional bidirectional long short-term memory network, the dependencies between the segmented action fragments are determined, including:

[0025] The multivariate time series collected during the human-machine collaboration are classified using a fully convolutional bidirectional long short-term memory network. The fully convolutional bidirectional long short-term memory network consists of two branches. The first branch contains three time convolution blocks as feature extractors for time series data, three filters, and a global average pooling layer. The second branch includes a dimension shuffle layer, a BiLSTM block, and a dropout layer:

[0026] The multivariate time series collected during the human-machine collaboration process are respectively input into the first branch to obtain the characteristic parameters of the multivariate time series;

[0027] The multivariate time series collected during the human-machine collaboration process are input into the dimension shuffling layer of the second branch for multivariate time series processing. The BiLSTM block receives the transformed time series after the dimension shuffling layer, transposes the time dimension, and then performs dropout to avoid overfitting.

[0028] The outputs of the first and second branches are connected and fed into the SoftMax classifier to obtain the dependencies between the segmented action fragments.

[0029] On the other hand, the present invention further provides a multi-sensor fusion human intention recognition system, which is applied to a human-machine collaboration process. The multi-sensor fusion human intention recognition system comprises:

[0030] A collection unit, used to collect continuous motion signals using inertial sensors worn on upper limbs of personnel, and to perform online segmentation on the continuous motion signals to obtain a set of candidate segmentation points;

[0031] A verification unit, configured to divide the continuous action signal into a preset number of segments with minimum entropy based on an online hybrid segmentation method of information gain, so as to determine the optimal segmentation point in the candidate segmentation point set, and to perform a legality check on the segmented action segments based on predefined meta-actions;

[0032] The recognition unit is used to determine the dependencies between the segmented action fragments based on the pre-built fully convolutional bidirectional long short-term memory network, and match them according to the pre-built action library to identify the behavioral intentions of the personnel.

[0033] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-sensor fusion human intention recognition method as described above when executing the program.

[0034] Through the above technical solution, combined with the action library and the fully convolutional bidirectional long short-term memory network, the segmentation of complex continuous human actions in human-machine collaboration scenarios is realized, the accuracy and versatility of personnel intention recognition in the human-machine collaboration process are improved, and the basis is provided for the real-time implementation and subsequent optimization of human-machine collaboration tasks.

[0035] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0037] Figure 1 is an implementation flow chart of a multi-sensor fusion human intention recognition method provided by an embodiment of the present invention;

[0038] Figure 2 It is a model architecture of a fully convolutional bidirectional long short-term memory network provided by an embodiment of the present invention;

[0039] Figure 3 It is a block diagram of an operator intention recognition method based on an inertial sensor in an assembly scenario provided by an embodiment of the present invention;

[0040] Figure 4 The present invention also provides a structural schematic diagram of a multi-sensor fusion human intention recognition system. DETAILED DESCRIPTION

[0041] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0042] See also Figure 1 As shown, it is a flowchart of an implementation of a multi-sensor fusion human intention recognition method provided by an embodiment of the present invention, which is applied to a human-machine collaboration process and includes the following execution steps:

[0043] Step 100: Continuous motion signals are collected using inertial sensors worn on the upper limbs of a person, and the continuous motion signals are segmented online to obtain a set of candidate segmentation points.

[0044] Before executing step 100, the personnel are pre-equipped with inertial sensors with built-in three-axis accelerometers, three-axis gyroscopes, and three-axis magnetometers on their upper arms, forearms, and hands; when the personnel perform actions, the inertial sensors are used to collect continuous action signals in real time, and the continuous action signals are cleaned and converted into joint vector angle data and Euler angle data; the continuous action signals are defined and decomposed based on the action types involved in the human-machine collaboration process, wherein a continuous action sequence is defined. ,in It is an indivisible meta-action, and the meta-action is defined based on the Euler angle data as ,in is the continuous action sequence collected by the i-th inertial sensor.

[0045] The joint vector angle data is used for the preliminary segmentation of the assembly action, and the preliminary segmentation of the assembly action is completed based on the idea of ​​online piecewise linear representation of the time series. The piecewise linear representation of the time series uses the maximum vertical distance (MVD) metric as the segmentation criterion to reduce the time complexity. The first point of the time series is selected as the initial point, and the previous segmentation point is defined. With the new segmentation point , all points between them The line connecting the previous segment point , arrive The vertical distance between ,but It should be within the set maximum error limit.

[0046] Specifically, when executing step 100, the following steps can be specifically performed: apply online piecewise linear representation processing to the collected joint vector angle data to obtain a preliminary set of candidate action segmentation points, wherein the online piecewise linear representation uses the maximum vertical distance measurement method as the segmentation criterion to reduce time complexity.

[0047] Furthermore, when a type of action is completed, the trend of the joint vector angle data changes. Based on the step-by-step strategy, the segmentation points of the time series are further optimized. The existence of the maximum error makes the trend turning point when the action is completed earlier than the segmentation point on the time axis. The segmentation points are modified based on the maximum tolerance of the six types of trend turning points in the time series and the local segments, and the piecewise linear representation of the joint vector angle data is completed, and a preliminary set of candidate points for action segmentation is obtained.

[0048] Step 101: Based on an online hybrid segmentation method of information gain, the continuous action signal is divided into a preset number of segments with minimum entropy to determine the optimal segmentation point in the candidate segmentation point set, and the legitimacy of the segmented action segments is checked based on predefined meta-actions.

[0049] Specifically, when executing step 101, the following steps may be specifically performed: pre-define a detection window of a fixed step length starting from the initial point of the time series, and identify the point in the candidate segmentation point set that reduces the entropy of the time series the most through the cost function of the time series information gain, and use it as the optimal segmentation point.

[0050] Specifically, the current time series is divided into non-overlapping segments based on information gain, and the divided segments are a set of sub-segments between two consecutive segmentation candidate points. Among them, the segment obtained by the method based on information gain has the smallest entropy, and the entropy reduction of the signal after the time series is divided is the largest. The total reduction in entropy is measured by information gain, and the cost function is as follows:

[0051] ;

[0052] ;

[0053] In the formula, After the time series is divided j part, is the length operator, X is a multidimensional time series, k is the set of candidate segmentation fragments in the previous joint vector angle segmentation process, is the entropy of the current time series, which has a constant value in this segment. For fragment The entropy of m is the number of groups of total data points in the current time series, is the first The probability of the segment area divided by the time series X.

[0054] Furthermore, a detection window with a fixed step length starting from the initial point of the time series is defined, and the fixed time step length should be set to be greater than the time required to complete the first assembly action. The point in the candidate segmentation point set that reduces the entropy of the time series the most is identified through the cost function of the time series information gain.

[0055] Furthermore, a detection window is defined that starts from the initial point of the time series and grows with the time series, and the original time series segments input after the piecewise linear representation are sequentially processed, and the cost of the newly added points in the candidate segmentation point set for the time series segmentation is calculated. Then, the importance of the action segmentation points obtained in sequence through the cost function of the time series flow is evaluated. With each addition of a new segmentation point in a fixed time series, the information gain tends to increase, that is: based on the size of the ratio of the slope values ​​of the candidate action segmentation point and its previous and subsequent candidate points as a metric, if the former is greater than the latter, the candidate action segmentation point is regarded as a true segmentation point, and the corresponding combined action fragment is obtained.

[0056] In some embodiments, the definition of meta-actions includes: based on the skeletal Euler angle method, defining the Euler angle correspondence between the child bones connected to the upper limb joints of the person and the parent bones, and defining the rotation order of the Euler angles as yxz for the shoulder joints and elbow joints of the person; the positive and negative values ​​of the three types of skeletal Euler angles (shoulder joint, elbow joint and wrist joint) correspond to different postures of the joints, where different postures are characterized by abduction and adduction, flexion and extension, and external rotation and internal rotation of the joints.

[0057] Furthermore, the obtained combined action fragments are analyzed, and it is checked whether the combined action fragments are missing or redundant according to the above definition.

[0058] Step 102: Based on the pre-built fully convolutional bidirectional long short-term memory network, the dependency relationship between the segmented action segments is determined, and matching is performed according to the pre-built action library to identify the behavioral intention of the person.

[0059] In some embodiments, the action library consists of three parts: an action template entry script, an assembly element action template library, and an assembly action sequence library. The action template entry script repeatedly enters actions of the same category and performs unified preprocessing. The processed actions are saved as action templates of this type for training. The assembly action sequence library contains flexible assembly sequences involved in assembly tasks to perform assembly sequence matching and predict the operator's intentions.

[0060] Furthermore, the action signal dataset consists of a multivariate time series, and the multivariate time series dataset of the action signal is defined as a tensor of shape (N, Q, M), where N is the number of samples in the dataset, Q is the maximum time step of all variables, and M is the number of variables processed for each time step.

[0061] Specifically, when executing step 102, the following steps may be specifically performed:

[0062] The multivariate time series collected during the human-machine collaboration are classified using a fully convolutional bidirectional long short-term memory network. The fully convolutional bidirectional long short-term memory network consists of two branches. The first branch contains three time convolution blocks as feature extractors for time series data, three filters, and a global average pooling layer. The second branch includes a dimension shuffle layer, a BiLSTM block, and a dropout layer:

[0063] The multivariate time series collected during the human-machine collaboration process are respectively input into the first branch to obtain the characteristic parameters of the multivariate time series;

[0064] The multivariate time series collected during the human-machine collaboration process are input into the dimension shuffling layer of the second branch for multivariate time series processing. The BiLSTM block receives the transformed time series after the dimension shuffling layer, transposes the time dimension, and then performs dropout to avoid overfitting.

[0065] The outputs of the first and second branches are connected and fed into the SoftMax classifier to obtain the dependencies between the segmented action fragments.

[0066] Specifically, the multivariate time series collected during the human-machine collaboration process are classified. The proposed model architecture consists of two branches (see the model architecture for details). Figure 2As shown in Figure 1), the branch of the full convolutional block contains 3 temporal convolutional blocks as feature extractors for time series data. The full convolutional block has three 1D convolutional kernels of size 8, 5, and 3. Batch normalization is performed after each layer to improve the speed, performance, and stability of the network. The nonlinear expression ability of the model is enhanced by the ReLU layer to alleviate the vanishing gradient problem in the network. The first branch is constructed by stacking three convolutional blocks with filter sizes of 128, 256, and 128 and subsequent global average pooling layers. The second branch consists of a dimension shuffle layer, a BiLSTM block, and a dropout layer. The dimension shuffle layer is used to process multivariate time series inputs. The BiLSTM block receives the transformed time series after the dimension shuffle layer, transposes the time dimension, and then performs dropout to avoid overfitting. Finally, the outputs of the two branches are connected and fed to the SoftMax classifier.

[0067] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0068] Embodiment 1:

[0069] As described in the background technology, in the current industrial scenario of human-machine collaborative assembly work, it is urgent for the collaborative robot to have the ability to recognize the operator's intention and flexibly provide assistance to the operator in the assembly work. In order to solve the above problems and improve the production efficiency of human-machine collaboration, this embodiment provides a flowchart of an operator intention recognition method based on an inertial sensor in an assembly scenario, such as Figure 3 As shown, the method includes an online hybrid segmentation mode of assembly actions, construction of an assembly action library, and recognition of assembly actions.

[0070] This embodiment is based on the collaborative robot AUBO-I5 and the inertial sensor FOHEART® MAGIC. Its optimization goal is to accurately segment the single-step assembly action and correctly identify the operator's intention in the process of human-machine collaborative assembly of the RV reducer, and then cooperate with the operator to improve production efficiency. The method mainly includes the following steps:

[0071] Step 1: predefine the assembly sequences that may occur during the assembly process, and define the meta-actions such as picking up\placing tools\workpieces, tapping with a rubber hammer, and tightening with a torque wrench in assembling the RV reducer using the above-mentioned description method based on the skeleton Euler angle.

[0072] Step 2: Based on the six inertial sensors FOHEART®MAGIC worn on the operator's upper arms, forearms, and hands, the operator's assembly movements are perceived and monitored, and the joint vector angle data and bone Euler angle data are collected and input into the host computer in the form of data streams.

[0073] Step 3: Apply online piecewise linear representation processing to the collected multivariate joint vector angle time series and obtain a preliminary set of assembly action segmentation candidate points. Merge the adjacent action segmentation candidate points of each joint of the left and right arms on the time axis, and set the maximum time tolerance to 0.5 seconds.

[0074] Step 4: Based on the information gain combination, the action segments are assembled without overlapping to obtain the action segmentation point with the minimum entropy in the existing time series in the initialized fixed step window. In the variable length window, the remaining candidate action segmentation points are determined based on the idea of ​​the changing trend of information gain.

[0075] Step 5: Based on the predefined assembly meta-actions, detect whether the assembly action segments composed of the time series initial point and the candidate action segmentation points have missing or redundant action segments, such as tools taken but not placed, workpieces taken but not assembled, and determine the real assembly action segmentation points.

[0076] Step 6: Input the original time series fragments that have passed the inspection into the BiLSTM-FCN network, capture the bidirectional dependencies in the sequence data through the cell state and gating mechanism, and infer the operator's assembly intention.

[0077] Step 7: Inferring the operator's further intention based on the predefined assembly sequences in the constructed assembly action library.

[0078] The above steps are steps for realizing operator intention recognition in an embodiment of the present invention.

[0079] In step 1, the assembly of a certain model of RV reducer is used as an example for analysis and explanation. The example includes 14 assembly intentions, and the assembly workpieces are detected and classified according to the scheme shown in the assembly task. Among them, the assembly action information of the operator in step 2 is collected at a sampling frequency of 100hz of the inertial sensor FOHEART® MAGIC, and the software Motion Venus used in conjunction with the inertial sensor is used to monitor whether the inertial sensor is abnormal during the assembly task. In step 3, the maximum allowable error of the piecewise linear representation based on the MVD measurement method is set to 6°, which can characterize that the operator's joint angle fluctuation within 6° during the assembly process is considered as unconscious upper limb shaking.

[0080] Compared with the prior art, the beneficial effects of this application are:

[0081] 1. In this application, in order to address the problems of poor versatility of visual sensors in various human-machine collaboration scenarios, such as large restrictions on equipment layout and personnel activity scenes, complex on-site backgrounds, occlusions, viewpoint changes, and lighting conditions, a human intention recognition process based on inertial sensors was constructed to improve the flexibility and generalization of intention recognition in personnel operations.

[0082] 2. In this application, in view of the current situation that the human continuous motion segmentation process based on inertial sensor data is mostly in offline mode and the application scenarios are mostly simple human motion patterns, an online hybrid segmentation method based on important points of time series and information gain is proposed, which realizes the segmentation of complex continuous human motion in human-machine collaboration scenarios, and provides a basis for the real-time implementation and later optimization of human-machine collaboration tasks.

[0083] 3. In this application, in view of the flexible and variable characteristics of on-site collaborative tasks in different tasks, a class of meta-action libraries is defined and constructed, and a human action classification model based on a fully convolutional bidirectional long short-term memory neural network (MBiLSTM-FCN) is proposed, which realizes the whole process of collecting actions, processing actions, extracting action features, and training meta-action templates. Through the action fragments obtained through online segmentation and the preset task sequences defined in the action library, the intention recognition and prediction of personnel in human-machine collaborative scenarios are realized, which improves the flexibility and efficiency of collaborative tasks.

[0084] See also Figure 4 As shown, the present invention also provides a structural schematic diagram of a multi-sensor fusion human intention recognition system, which is applied to a human-machine collaboration process, including:

[0085] The collecting unit 40 is used to collect continuous motion signals using inertial sensors worn on the upper limbs of the personnel, and to perform online segmentation on the continuous motion signals to obtain a set of candidate segmentation points;

[0086] A verification unit 41 is used to divide the continuous action signal into a preset number of segments with minimum entropy based on an online hybrid segmentation method of information gain, so as to determine the optimal segmentation point in the candidate segmentation point set, and perform a legality check on the segmented action segments based on predefined meta-actions;

[0087] The recognition unit 42 is used to determine the dependency between the segmented action segments based on a pre-built fully convolutional bidirectional long short-term memory network, and to match them according to a pre-built action library to recognize the behavioral intention of the person.

[0088] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-sensor fusion human intention recognition method described in any one of the above embodiments when executing the program.

[0089] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multi-sensor fusion human intention recognition method described in any of the above embodiments.

[0090] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0091] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0094] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0095] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0096] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0097] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0098] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A multi-sensor fusion human intention recognition method, applied to human-machine collaboration process, characterized in that: The multi-sensor fusion human intention recognition method includes: Using an inertial sensor worn on the upper limbs of a person to collect continuous motion signals, and performing online segmentation on the continuous motion signals to obtain a set of candidate segmentation points; An online hybrid segmentation method based on information gain is used to divide the continuous action signal into a preset number of segments with minimum entropy to determine the optimal segmentation point in the candidate segmentation point set, and the legitimacy of the segmented action segments is checked based on predefined meta-actions; Based on the pre-built fully convolutional bidirectional long short-term memory network, the dependency between the segmented action fragments is determined, and the pre-built action library is used to match and identify the behavior intention of the person; Among them, the online hybrid segmentation method based on information gain divides the continuous action signal into a preset number of segments with minimum entropy to determine the optimal segmentation point in the candidate segmentation point set, and performs a legitimacy check on the segmented action segments based on predefined meta-actions, including: A fixed-step detection window starting from the initial point of the time series is predefined, and the point that reduces the entropy of the time series the most in the candidate segmentation point set is identified through the cost function of the time series information gain, and is used as the optimal segmentation point; The cost function is as follows: ； ； In the formula, is the jth segment after the time series is divided, is the length operator, is the entropy of the current time series, k represents the set of candidate segmentation segments in the previous joint vector angle segmentation process, X represents the multidimensional time series, For fragment The entropy of m is the number of groups of total data points in the current time series, is the first The probability of the segment area divided by the time series X.

2. The multi-sensor fusion human intention recognition method according to claim 1, characterized in that: Inertial sensors with built-in three-axis accelerometers, three-axis gyroscopes, and three-axis magnetometers are pre-installed on the upper arms, forearms, and hands of the personnel; When people perform actions, they use inertial sensors to collect continuous action signals in real time, and then clean the continuous action signals and convert them into joint vector angle data and Euler angle data. Based on the action types involved in the human-machine collaboration process, the continuous action signals are defined and decomposed, where the continuous action sequence is defined. ,in It is an indivisible meta-action, and the meta-action is defined based on the Euler angle data as ,in is the continuous action sequence collected by the i-th inertial sensor.

3. The multi-sensor fusion human intention recognition method according to claim 2 is characterized in that: The continuous motion signal is collected by using the inertial sensor worn by the upper limb of the person, and the continuous motion signal is segmented online to obtain a set of candidate segmentation points, including: The collected joint vector angle data are processed by online piecewise linear representation to obtain a preliminary set of candidate action segmentation points. The online piecewise linear representation uses the maximum vertical distance metric as the segmentation criterion to reduce time complexity.

4. The multi-sensor fusion human intention recognition method according to claim 1, characterized in that: The definition of a meta-action includes: Based on the Euler angle of bones, define the corresponding relationship of the Euler angles of the child bones connected to the upper limb joints of the personnel relative to the parent bones. For the shoulder joint and elbow joint of the personnel, define the rotation order of the Euler angle as yxz; The positive and negative values ​​of the three types of skeletal Euler angles correspond to different postures of the joints, where different postures are represented by abduction and adduction, flexion and extension, and external rotation and internal rotation of the joints. The three types of skeletal Euler angles represent the Euler angles of the shoulder joint, elbow joint, and wrist joint.

5. The multi-sensor fusion human intention recognition method according to claim 1, characterized in that: The action library consists of three parts: an action template entry script, an assembly element action template library, and an assembly action sequence library. The action template entry script repeatedly enters actions of the same category and performs unified preprocessing. The processed actions are saved as action templates of this type for training. The assembly action sequence library contains flexible assembly sequences involved in assembly tasks to perform assembly sequence matching and predict the operator's intentions.

6. The multi-sensor fusion human intention recognition method according to claim 1, characterized in that: Based on the pre-built fully convolutional bidirectional long short-term memory network, the dependencies between the segmented action fragments are determined, including: The multivariate time series collected during the human-machine collaboration are classified using a fully convolutional bidirectional long short-term memory network. The fully convolutional bidirectional long short-term memory network consists of two branches. The first branch contains three time convolution blocks as feature extractors for time series data, three filters, and a global average pooling layer. The second branch includes a dimension shuffle layer, a BiLSTM block, and a dropout layer: The multivariate time series collected during the human-machine collaboration process are respectively input into the first branch to obtain the characteristic parameters of the multivariate time series; The multivariate time series collected during the human-machine collaboration process are input into the dimension shuffling layer of the second branch for multivariate time series processing. The BiLSTM block receives the transformed time series after the dimension shuffling layer, transposes the time dimension, and then performs dropout to avoid overfitting. The outputs of the first and second branches are connected and fed into the SoftMax classifier to obtain the dependencies between the segmented action fragments.

7. A multi-sensor fusion human intention recognition system applied to the multi-sensor fusion human intention recognition method according to any one of claims 1 to 6, applied to a human-machine collaboration process, characterized in that: The multi-sensor fusion human intention recognition system includes: A collection unit, used to collect continuous motion signals using inertial sensors worn on upper limbs of personnel, and to perform online segmentation on the continuous motion signals to obtain a set of candidate segmentation points; A verification unit, configured to divide the continuous action signal into a preset number of segments with minimum entropy based on an online hybrid segmentation method of information gain, so as to determine the optimal segmentation point in the candidate segmentation point set, and to perform a legality check on the segmented action segments based on predefined meta-actions; The recognition unit is used to determine the dependencies between the segmented action fragments based on the pre-built fully convolutional bidirectional long short-term memory network, and match them according to the pre-built action library to identify the behavioral intentions of the personnel.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the multi-sensor fusion human intention recognition method as described in any one of claims 1-6 are implemented.

Images