Human-machine collaboration method, device, intelligent terminal and storage medium

Abstract

The human-machine collaboration method, device, intelligent terminal and storage medium provided by the present invention specifically relate to the field of human-machine collaboration technology, and the scheme includes: using the three-dimensional posture parameterization network in the human digital twin model to perform three-dimensional posture parameterization on the target depth image to construct the human posture; based on the human posture, extract the human joint point sequence and perform feature extraction to obtain the global joint point feature; use the risk assessment network in the human digital twin model to perform risk assessment on the global joint point feature to obtain the ergonomic risk index; use the behavior intention recognition network in the human digital twin model to identify and classify the global joint point feature and the target video stream, obtain the classification result of the behavior intention, so as to make behavioral decisions and plan the motion trajectory. The scheme can realize real-time and comprehensive understanding and response to the human body state, ensure that the robot performs tasks efficiently and safely, and thus realize smoother human-machine collaboration.

Description

Technical Field

[0001] The present invention relates to the field of human-machine collaboration technology, and in particular to a human-machine collaboration method, device, intelligent terminal and storage medium. Background Art

[0002] As one of the most promising paradigms in Industry 5.0, human-robot collaboration (HRC) is gradually being widely used in the field of intelligent manufacturing. In this mode, human operators and robots work closely together on the production line to complete tasks together. Compared with the traditional model that only focuses on profit and production efficiency, human-robot collaboration emphasizes human well-being and comfort, trying to create a more humane working environment, and has great potential and application value. However, in order to achieve efficient collaboration between humans and machines, robots need to have the ability to accurately and timely perceive human movements and postures.

[0003] Digital twin technology is a technology that can create virtual copies of physical objects and simulate and predict the behavior of physical objects in a virtual environment. In the field of human-machine collaboration technology, the human digital twin (HDT) model can serve as a digital representation of the human body to help robots better understand and predict the actions and intentions of human operators. However, existing technologies for implementing human-machine collaboration solutions based on HDT models using visual perception technology usually only focus on processing a single task, such as posture recognition or action recognition, and are likely to cause discomfort to operators and pose safety risks. Therefore, it is impossible to design a comprehensive and safe human-machine collaboration method planning solution. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a human-computer collaboration method, device, intelligent terminal and storage medium, aiming to solve the problem that the human-computer collaboration scheme based on the HDT model in the prior art cannot design a comprehensive and safe human-computer collaboration method due to the lack of comprehensive understanding and analysis of the human body state.

[0005] In order to achieve the above-mentioned object, the first aspect of the present invention provides a human-machine collaboration method based on a human digital twin model, the method comprising:

[0006] Obtain target depth images and target video streams in human-machine collaboration scenarios;

[0007] Using a three-dimensional posture parameterization network in a human digital twin model to perform three-dimensional posture parameterization on the target depth image to obtain three-dimensional posture parameters, and constructing a human posture based on the three-dimensional posture parameters;

[0008] Based on the human body posture, a human body joint point sequence is extracted, and a preset spatiotemporal graph convolutional network is used to perform feature extraction on the human body joint point sequence to obtain global joint point features;

[0009] Using the risk assessment network in the human digital twin model to conduct risk assessment on the global joint point features to obtain an ergonomic risk index;

[0010] Using a behavior intention recognition network in a human digital twin model to identify and classify the global joint point features and the target video stream, and obtain a classification result of the behavior intention;

[0011] Based on the classification results of the human body posture, the ergonomic risk index and the behavioral intention, a behavioral decision is made and a motion trajectory is planned.

[0012] Optionally, the three-dimensional posture parameterization network in the human body digital twin model is used to perform three-dimensional posture parameterization on the target depth image to obtain three-dimensional posture parameters, and construct a human body posture based on the three-dimensional posture parameters, including:

[0013] Performing feature extraction on the target depth image to obtain target human features;

[0014] The three-dimensional posture parameterization network in the human body digital twin model is used to perform three-dimensional posture parameterization on the target human body features to obtain three-dimensional posture parameters, and the three-dimensional posture parameters at least include spatial rotation parameters, spatial translation parameters, posture parameters and shape parameters;

[0015] Using a preset parameterized human body model to perform parameterization processing on the posture parameters and the shape parameters to obtain a dense human body grid;

[0016] Based on the dense grid of the human body, the spatial rotation parameters and the spatial translation parameters, a trained three-dimensional human body posture perception network is used to perform three-dimensional posture and joint estimation to obtain the human body posture.

[0017] Optionally, the using a preset spatiotemporal graph convolutional network to extract features from the human joint point sequence to obtain global joint point features includes:

[0018] Splitting the human joint point sequence into a plurality of human joint point subsequences, and using a matching spatiotemporal graph convolutional network to process the human joint point subsequences respectively to obtain corresponding local spatiotemporal graphs;

[0019] The preset aggregation network is used to fuse the features of all the local spatiotemporal graphs to obtain the global joint point features.

[0020] Optionally, the use of a behavior intention recognition network in a human digital twin model to identify and classify the global joint point features and the target video stream to obtain a classification result of the behavior intention includes:

[0021] Extracting features from the target video stream to obtain spatiotemporal data features;

[0022] Superimposing the global joint point feature and the spatiotemporal data feature to obtain a fusion feature;

[0023] The behavior intention recognition network in the human digital twin model is used to identify and classify the fusion features to obtain the classification results of the behavior intention.

[0024] Optionally, making a behavior decision and planning a motion trajectory based on the classification result of the human body posture, the ergonomic risk index and the behavior intention includes:

[0025] Based on the classification result of the behavioral intention, obtaining a behavioral decision result;

[0026] Based on the behavior decision result, motion path planning is performed using the human body posture and the ergonomic risk index to obtain a motion trajectory that matches the behavior decision result.

[0027] Optionally, the method further includes the step of training the human digital twin model, specifically including:

[0028] Collecting depth image samples and video stream samples in a human-machine integration scenario to obtain a training set, and manually annotating the training set to obtain a manual annotation result;

[0029] Inputting the depth image samples into an initial three-dimensional posture parameterization network to generate human body posture samples;

[0030] Comparing the human body posture sample with the manual annotation result and optimizing the training to obtain the three-dimensional posture parameterized network;

[0031] Using the three-dimensional posture parameterization network, updating the human body posture samples, and generating global joint point feature samples based on the human body posture samples;

[0032] Inputting the global joint feature sample into an initial risk assessment network to obtain an ergonomic risk index sample;

[0033] Inputting the global joint point features and the video stream samples into an initial behavior intention recognition network to obtain a classification result sample of the behavior intention;

[0034] Comparing the ergonomic risk index sample and the behavioral intention classification result sample with the manual annotation result at the same time and optimizing the training to obtain the risk assessment network and the behavioral intention recognition network;

[0035] The human body digital twin model is constructed based on the three-dimensional posture parameterization network, the risk assessment network and the behavior intention recognition network.

[0036] Optionally, the target video stream is preprocessed, including:

[0037] Based on the target processing accuracy, preset feature map dimensions;

[0038] At least one of segmentation, sampling and / or scaling is performed on the target video stream according to the feature map dimension to update the target video stream.

[0039] A second aspect of the present invention provides a human-machine collaboration device based on a human digital twin model, the model comprising:

[0040] The data acquisition module is used to obtain the target depth image and target video stream in the human-machine integration scene;

[0041] A human posture reconstruction module is used to perform three-dimensional posture parameterization on the target depth image using the three-dimensional posture parameterization network in the human digital twin model, obtain three-dimensional posture parameters, and construct a human posture based on the three-dimensional posture parameters;

[0042] A joint point feature extraction module is used to extract a human joint point sequence based on the human body posture, and use a preset spatiotemporal graph convolutional network to extract features of the human joint point sequence to obtain global joint point features;

[0043] An ergonomic risk index assessment module, used to perform risk assessment on the global joint point features using a risk assessment network in a human digital twin model to obtain an ergonomic risk index;

[0044] A behavior intention recognition module is used to use the behavior intention recognition network in the human digital twin model to identify and classify the global joint point features and the target video stream to obtain a classification result of the behavior intention;

[0045] A behavior decision-making and planning module is used to make behavior decisions and plan movement trajectories based on the classification results of the human body posture, the ergonomic risk index and the behavior intention.

[0046] The third aspect of the present invention provides an intelligent terminal, which includes a memory, a processor, and human-computer collaboration based on a human digital twin model stored in the memory and run on the processor. When the human-computer collaboration based on the human digital twin model is executed by the processor, it implements any step of the above-mentioned human-computer collaboration method based on the human digital twin model.

[0047] A fourth aspect of the present invention provides a computer-readable storage medium, on which is stored human-computer collaboration based on a digital twin model of the human body. When the human-computer collaboration based on the digital twin model of the human body is executed by a processor, the human-computer collaboration based on the digital twin model of the human body implements any step of the above-mentioned human-computer collaboration method based on the digital twin model of the human body.

[0048] Compared with the prior art, the beneficial effects of this solution are as follows:

[0049] The present invention uses the human digital twin model to process the target depth image and target video stream, and obtains three processing results: human posture, ergonomic risk index, and classification results of behavioral intention. Then the human body can make behavioral decisions and plan motion trajectories based on all the processing results. This method is based on the human digital twin model. By real-time identification and synchronization of the 3D posture, behavioral intention, and ergonomic risk index of the human body in the human-machine collaboration scene to the cyberspace, it can achieve a real-time and comprehensive understanding and response to the human body state, ensuring that the robot can achieve efficient and safe execution of tasks, thereby achieving smoother human-machine collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 A brief flow chart of the human-machine collaboration method based on the human digital twin model of the present invention;

[0052] Figure 2 This is a flow chart of the human-machine collaboration method based on the human digital twin model of the present invention;

[0053] Figure 3 This is a schematic diagram of the functional modules of the human digital twin model of the present invention;

[0054] Figure 4 Schematic diagram of the structure of the human digital twin model of the present invention;

[0055] Figure 5 It is a schematic diagram of the structure of the intelligent terminal of the present invention. DETAILED DESCRIPTION

[0056] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0057] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0058] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0059] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0060] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0061] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0063] In view of the problem that the human-computer collaboration scheme based on the HDT model in the prior art cannot design a comprehensive and safe human-computer collaboration method due to the lack of comprehensive understanding and analysis of the human body state, the present invention proposes a human-computer collaboration method based on a human digital twin model. In response to the target depth image and target video stream received in real time in the human-computer integration scene, the human digital twin model is used to process the target depth image and the target video stream to obtain the processing result. The method is based on the human digital twin model, and can reflect various aspects of the human body in the physical space in real time by identifying and synchronizing the 3D posture, behavioral intention, and ergonomic risk index of the human body in the human-computer collaboration scene to the cyberspace in real time, and can be used to guide the decision-making and task execution of human-computer collaboration.

[0064] The embodiment of the present invention provides a human-machine collaboration method based on a human digital twin model, which is deployed on electronic devices such as computers and servers, and the electronic devices control the robot to perform actions, or is directly deployed on the operating robot, and is applied to human-machine collaboration scenarios. Specifically, Figure 1 and Figure 2 As shown, the method of this embodiment includes:

[0065] Step S1000: Acquire a target depth image and a target video stream in a human-machine integration scene;

[0066] Specifically, the target depth image refers to an RGB-D image, which contains not only color information but also the distance of each pixel to the acquisition terminal (such as a camera). The target video stream refers to an RGB video stream or an RGB-D video stream.

[0067] Step S2000: using the three-dimensional posture parameterization network in the human body digital twin model to perform three-dimensional posture parameterization on the target depth image, obtain three-dimensional posture parameters, and construct the human body posture based on the three-dimensional posture parameters;

[0068] Specifically, a 3D posture parameterization network in the human digital twin model is used to accurately extract 3D posture parameters from the target depth image, and the human posture is constructed based on the obtained 3D posture parameters. The 3D posture parameterization network includes a residual deep learning network, such as ResNet50.

[0069] Step S3000: extracting a human joint point sequence based on human body posture, and using a preset spatiotemporal graph convolutional network to extract features of the human joint point sequence to obtain global joint point features;

[0070] Specifically, a 3D human joint point sequence is extracted by performing linear regression on the human body posture.

[0071] Step S4000: using the risk assessment network in the human digital twin model to conduct risk assessment on global joint point features to obtain an ergonomic risk index;

[0072] Specifically, by regressing the REBA score, a quantitative risk assessment is performed on the human body posture corresponding to the global joint point features. Different scores represent different ergonomic risk indexes, and the larger the score, the higher the risk, and the more unsafe the human body posture.

[0073] For example, according to the REBA ergonomics evaluation standard, the human posture risk is scored, the training data is labeled, and the standard REBA calculation process is used to calculate the angles between joints, etc., to obtain the ergonomic risk score of the whole body (for example, the score is 1-12). The higher the score, the higher the risk, and the more unsafe the human posture is. These scores are used for labeling to train the risk assessment network to regress and estimate these scores. After the training is completed, there is no need to label and train again during use. Instead, the trained risk assessment network can be directly used for a forward propagation to obtain the estimated ergonomic risk index.

[0074] Step S5000: using the behavior intention recognition network in the human digital twin model to identify and classify global joint point features and target video streams to obtain the classification results of behavior intentions;

[0075] Step S6000: Make behavioral decisions and plan movement trajectories based on the classification results of human posture, ergonomic risk index and behavioral intention.

[0076] In this embodiment, after processing the target depth image and the target video stream using the human digital twin model, three processing results are obtained, namely, the classification results of human posture, ergonomic risk index and behavioral intention. Then the human body can make behavioral decisions and plan the motion trajectory according to all the processing results. Based on the human digital twin model, the method can reflect the various aspects of the human body in the physical space in real time by identifying and synchronizing the 3D posture, behavioral intention and ergonomic risk index of the human body in the human-machine collaboration scene to the cyberspace in real time, realize the real-time and comprehensive understanding and response of the human body state, accurately guide the decision-making of human-machine collaboration, and ensure the safety of the human body in the process of performing tasks. The robot can perform tasks efficiently and safely, while minimizing conflicts with people, thereby achieving smoother human-machine collaboration. It can also ensure that the robot stays away from possible obstacles or risk areas, avoids damage to human bones or muscles and avoids accidents. Further, it can also provide early warnings based on the ergonomic risk index to provide real-time feedback for human-machine collaboration scenes.

[0077] like Figure 2 and Figure 3 As shown, in one implementation, the step S2000 uses a 3D posture parameterization network to perform 3D posture parameterization on the target depth image to obtain 3D posture parameters, and constructs a human body posture based on the 3D posture parameters, specifically including:

[0078] Step S2100: extracting features from the target depth image to obtain target human features;

[0079] Step S2200: using a 3D posture parameterization network to perform 3D posture parameterization on target human features to obtain 3D posture parameters, where the 3D posture parameters at least include space rotation parameters, space translation parameters, posture parameters and shape parameters;

[0080] Step S2300: using a preset parameterized human body model to perform parameterization processing on posture parameters and shape parameters to obtain a dense mesh of the human body;

[0081] Step S2400: Based on the dense grid of the human body, the spatial rotation parameters and the spatial translation parameters, a trained three-dimensional human posture perception network is used to perform three-dimensional posture and joint estimation to obtain the human body posture.

[0082] For example, the SMPL model in the parameterized human body model is a classic human 3D model with good representation capabilities. It can greatly simplify the reconstruction process because it can stretch and deform the human mesh template according to the estimated few postures and shape joints to reconstruct the human 3D representation in the input image.

[0083] The SMPL model is used to process the spatial rotation parameters, spatial translation parameters, posture parameters and shape parameters in the three-dimensional posture parameters. The process of reconstructing the human posture M can be expressed as:

[0084] M=R·SMPL(θ,β)+T,

[0085] Among them, R is the estimated spatial rotation parameter, T is the spatial translation parameter, θ is the estimated posture parameter of the SMPL model, and β is the estimated shape parameter of the SMPL model.

[0086] It can be seen that this embodiment adopts the SMPL model to use the estimated three-dimensional posture parameters of the human body to identify and reconstruct the human body posture and 3D mesh, which simplifies the 3D reconstruction process and can reduce the calculation time cost.

[0087] In one embodiment, the step S3000 uses a preset spatiotemporal graph convolutional network to extract features from a human joint sequence to obtain global joint features, specifically including:

[0088] Step S3100: splitting the human joint point sequence into several human joint point subsequences, and using a matching spatiotemporal graph convolutional network to process the human joint point subsequences respectively to obtain corresponding local spatiotemporal graphs;

[0089] Specifically, the human joint sequence is split into human joint subsequences such as head, torso, arms and legs, and the human joint subsequences are processed using a spatiotemporal graph convolutional network that matches each human joint subsequence to obtain a local spatiotemporal graph corresponding to the human joint subsequence. The spatiotemporal graph convolutional network includes but is not limited to the ST-GCN network.

[0090] Step S3200: Use a preset aggregation network to fuse features of all local spatiotemporal graphs to obtain global joint point features.

[0091] Specifically, all local spatiotemporal graphs of the same dimension are superimposed using a preset aggregation network, and then one or more linear layers (also called fully connected layers) with the same dimension as the local spatiotemporal graph are passed to obtain aggregated global joint point features with the same dimension. It should be noted that in actual applications, local spatiotemporal graphs of specific dimensions are often generated according to specific needs.

[0092] For example, assuming that the human joint sequence has a total of 15 frames and the time span is 5 seconds, a single-frame joint only contains the spatial joint data of the corresponding frame of the image, and when the entire sequence is added, there is information in the time dimension. Then the joint sequence is split into four human joint subsequences of head, torso, arms and legs from the spatial dimension. The topological connection of each human joint subsequence includes both the connection between different joints in the single-frame space and the connection of the same joint in the time dimension. Here, the joints in each local space-time graph are topologically connected to form an undirected space-time graph. An undirected space-time graph is constructed for each human joint subsequence, and these four undirected space-time graphs are respectively input into four ST-GCN branch networks, such as Figure 3 As shown. The undirected time-space graph is represented by G = (V, E), where G represents a graph, V represents a set of joint points, and E represents a set of edges connecting the points. Each human key point in each frame is regarded as a joint point, and the natural connection and time domain connection between the human key points are used as edges. Each time-space graph includes a subsequence of joint points of several frames of several joint points in the human skeleton sequence, that is, it has both in-body and inter-frame connections, that is, a time-space graph.

[0093] In this embodiment, by fusing each local spatiotemporal graph of consecutive frames, temporal consistency can be ensured, thereby accurately estimating the human body posture.

[0094] like Figure 2 and Figure 3 As shown, in one embodiment, the behavior intention recognition network in step S5000 is used to identify and classify the global joint point features and the target video stream to obtain the classification result of the behavior intention, which specifically includes:

[0095] Step S5100: extracting features from the target video stream to obtain spatiotemporal data features;

[0096] Specifically, a single image only contains the spatial information of the scene at a moment, while an RGB video stream segment contains not only spatial information but also change information in the time dimension. Here, the RGB video stream segment is called spatiotemporal data, so that feature extraction is performed on the RGB video stream to obtain spatiotemporal data features.

[0097] For example, given a segment of RGB video stream, it is first preprocessed to obtain a B*C*H*W dimensional RGB video segment, where B is the number of samples in the batch (usually determined by the video memory of the training GPU during training, and 1 during deployment), C is the length of the RGB video stream segment sequence, and H and W are the length and width of the image, respectively; then the BxCxHxW dimensional RGB video segment is subjected to feature extraction through the 3D ResNet50 network to obtain spatiotemporal data features. 3D ResNet50 performs forward propagation during the processing, and converts the spatiotemporal data feature vector into a preset dimension, such as 1024 dimensions, through the last global connection layer. It should be stated that the difference between the model structure of the 3D ResNet50 network here and the model structure of the ResNet50 network mentioned above is only the addition of the input of the time dimension.

[0098] Step S5200: superimpose the global joint point features and the spatiotemporal data features to obtain fusion features;

[0099] Specifically, the dimensions of the global joint point features and the spatiotemporal data features are the same. The two are superimposed and fused to obtain the fused features. It can be seen that the fused features include both the connections between different joint points of the human body in multi-frame space, and the connections between the same joint point in the continuous time dimension, which can reflect the position distribution of each joint point of the human skeleton in the continuous time frame.

[0100] Step S5300: Use the behavior intention recognition network to identify and classify the fused features to obtain the classification results of the behavior intention.

[0101] Specifically, the behavior intention recognition network is used to identify the fusion features, and the recognition results are matched with the pre-set behavior intention categories to obtain the classification results of the behavior intention. The behavior intention recognition network here is similar to the aggregation network, which is composed of one or more linear layers. The input is the fusion features, and the output is the classification category of the behavior intention. The dimension of the output data changes with the number of classification categories of the behavior intention. It is easy to understand that the classification of specific behavior intentions can be divided accordingly according to factors such as the human-machine collaboration scenario and the complexity of the human body's operation of the machine. Moreover, in the process of human-machine collaboration, the fusion features within the continuous time period (such as 5 seconds or 10 seconds, etc.) can be identified and classified by the behavior intention recognition network according to the fusion features of the continuous time period to obtain one or more classification results.

[0102] In this embodiment, the fusion features generated by superimposing the global joint point features and the spatiotemporal data features include more comprehensive information about the human body posture in the three-dimensional space, which can make the recognition and classification of behavioral intentions more accurate.

[0103] In one embodiment, the classification results of human body posture, ergonomic risk index and behavioral intention in step S6000 are used to make behavioral decisions and plan the motion trajectory, which specifically includes:

[0104] Step S6100: obtaining a behavior decision result based on the classification result of the behavior intention;

[0105] Specifically, based on one or more classification results obtained, a behavior decision result corresponding to each classification result is obtained, which may be a single behavior decision result or a continuous decision result.

[0106] Step S6200: Based on the behavior decision result, motion path planning is performed using human body posture and ergonomic risk index to obtain a motion trajectory that matches the behavior decision result.

[0107] Specifically, a relatively simple motion trajectory can be obtained based on one behavior decision result, or a relatively complex motion trajectory can be obtained based on multiple behavior decision results.

[0108] For example, a human-machine collaboration scenario is known in which a robot is required to operate a machine to place a target object from the ground to a forklift. The corresponding classification results of the behavioral intentions are grasping, lifting, and placing. By using human body posture and ergonomic risk index rules, motion path planning is performed for the three behavioral decision results of grasping, lifting, and placing, and a motion trajectory that can continuously execute grasping, lifting, and placing the target object is obtained.

[0109] In this embodiment, the classification results of behavioral intentions are used to guide more accurate and reasonable behavioral decisions, and a precise and safe motion trajectory is planned for the robot based on the behavioral decisions, thereby achieving smoother human-machine collaboration.

[0110] like Figure 2 As shown, in one embodiment, step S7000 is also included, i.e., a step of training a human digital twin model, specifically including:

[0111] Step S7100: collecting depth image samples and video stream samples in a human-machine integration scene to obtain a training set, and manually annotating the training set to obtain a manual annotation result;

[0112] Step S7200: input the depth image sample into the initial three-dimensional posture parameterization network to generate a human posture sample;

[0113] Step S7300: Compare the human body posture sample with the manual annotation result and optimize the training to obtain a three-dimensional posture parameterized network;

[0114] Specifically, by comparing the human posture sample with the manual annotation result, the value of the loss function of the constructed human posture sample is calculated, and the parameter gradient of the back propagation model is calculated, and the random gradient descent is used for optimization training to obtain the model parameters that meet the preset first loss function threshold, thereby obtaining a trained three-dimensional posture parameterized network. The first loss function here can use the L2 loss function, also called the mean squared error loss (MSE), or different types of loss functions can be flexibly selected according to the different accuracy requirements of the actual application.

[0115] Step S7400: using the three-dimensional posture parameterization network to update the human posture samples, and based on the human posture samples, generate global joint feature samples;

[0116] Step S7500: inputting the global joint feature samples into the initial risk assessment network to obtain ergonomic risk index samples;

[0117] Step S7600: inputting the global joint point features and the video stream samples into the initial behavior intention recognition network to obtain the classification result samples of the behavior intention;

[0118] Step S7700: Compare the ergonomic risk index samples and the classification result samples of behavioral intentions with the manual annotation results and optimize the training to obtain a risk assessment network and a behavioral intention recognition network;

[0119] Specifically, by comparing the ergonomic risk index samples and the classification result samples of behavioral intentions with the manual annotation results at the same time, that is, respectively calculating the values ​​of the loss functions corresponding to the ergonomic risk index samples or the classification result samples of behavioral intentions, and summing the values ​​of the two loss functions and comparing them with the preset second loss function threshold, and calculating the parameter gradient of the back propagation model, using stochastic gradient descent for optimization training, obtaining the model parameters that meet the preset second loss function threshold, and thus obtaining a trained three-dimensional posture parameterized network. For example, the ergonomic risk index uses the L2 loss function to judge the error loss, and the human behavioral intention recognition and classification uses the cross entropy loss function to judge the error loss, or different types of loss functions can be flexibly selected according to the different accuracy requirements of actual applications.

[0120] It is easy to understand that the human digital twin model constructed by the present invention includes three sub-network modules: a three-dimensional posture parameterized network, a risk assessment network, and a behavior intention recognition network. The human posture constructed using the three-dimensional posture parameterized network can provide input data such as global joint point features for the risk assessment network and the behavior intention recognition network, and the risk assessment network and the behavior intention recognition network are trained as a whole, and the total loss error of the two is used as an indicator for evaluating the two. It can be seen that the three sub-network modules have a close relationship of complementing each other, relying on each other and influencing each other. Obviously, the trained human digital twin model has good robustness and stability.

[0121] Step S7800: Construct a human digital twin model based on the three-dimensional posture parameterization network, the risk assessment network and the behavior intention recognition network.

[0122] In this embodiment, by decoupling the entire human digital twin model module, the visual recognition functions with large differences are split and divided into two networks for step-by-step recognition. This can avoid insufficient training and decreased accuracy caused by the conflict between the loss functions of multiple tasks with large differences in the training process of a single network, thereby more accurately capturing the state of the human body.

[0123] In one implementation, step S8000 is further included, i.e., preprocessing the target video stream, specifically including:

[0124] Step S8100: Presetting feature map dimensions based on target processing accuracy;

[0125] Step S8200: Perform at least one of segmentation, sampling and / or scaling on the target video stream according to the feature map dimension to update the target video stream.

[0126] For example, the target video stream is divided into small video clips of 5 seconds in length, and each video clip is sampled at equal intervals to extract 15 frames of depth images to form an input clip. Each frame of the image is preprocessed by noise removal, filtering, normalization, and scaling to make the image more suitable for subsequent 3D posture reconstruction and motion analysis.

[0127] In summary, the human-machine collaboration method proposed in the present invention has the characteristics of real-time, flexibility, safety and accuracy, and the specific analysis is as follows;

[0128] 1) Real-time: Through this invention, the status and behavior of human operators can be captured and analyzed in real time, and a human digital twin model can be constructed to provide real-time feedback for human-machine collaboration scenarios. This model can promote the robot's real-time understanding and response to human behavior and intentions, thereby achieving smoother human-machine collaboration.

[0129] 2) Flexibility: The model can adapt to a variety of human-machine collaboration scenarios and tasks, and has strong versatility and adaptability.

[0130] 3) Safety: The present invention can more accurately capture and analyze the posture and movement of the human body by comprehensively utilizing multiple features and modules, such as 3D posture reconstruction and joint point recognition, thereby more effectively ensuring the safety of operators in human-machine collaboration.

[0131] 4) Accuracy: Compared with the existing technology that often uses a feature extraction network to simultaneously identify multiple aspects of the human body, the present invention splits the visual recognition functions with great differences through module decoupling, and divides them into two networks for step-by-step recognition. This avoids insufficient training and decreased accuracy caused by the conflict between the loss functions of multiple tasks with great differences in the training process of a single network, and can capture the state of the human body more accurately.

[0132] like Figure 4 As shown, corresponding to the above-mentioned human-machine collaboration method based on the human digital twin model, the embodiment of the present invention further provides a human-machine collaboration device based on the human digital twin model, and the above-mentioned human digital twin model includes:

[0133] The data acquisition module 410 is used to obtain a target depth image and a target video stream in a human-machine integration scene;

[0134] A human posture reconstruction module 420 is used to perform three-dimensional posture parameterization on the target depth image using the three-dimensional posture parameterization network in the human digital twin model, obtain three-dimensional posture parameters, and construct a human posture based on the three-dimensional posture parameters;

[0135] The joint feature extraction module 430 is used to extract a human joint sequence based on human posture, and use a preset spatiotemporal graph convolutional network to extract features of the human joint sequence to obtain global joint features;

[0136] The ergonomic risk index assessment module 440 is used to perform risk assessment on global joint point features using a risk assessment network in the human digital twin model to obtain an ergonomic risk index;

[0137] The behavior intention recognition module 450 is used to recognize and classify the global joint point features and the target video stream using the behavior intention recognition network in the human digital twin model to obtain the classification result of the behavior intention;

[0138] The behavior decision planning module 460 is used to make behavior decisions and plan movement trajectories based on the classification results of human posture, ergonomic risk index and behavior intention.

[0139] Specifically, in this embodiment, the specific functions of the above-mentioned human-machine collaboration device based on the digital twin model of the human body can also refer to the corresponding description in the above-mentioned human-machine collaboration method based on the digital twin model of the human body, and will not be repeated here.

[0140] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 5 As shown. The above-mentioned intelligent terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a human-computer collaboration program based on a human digital twin model. The internal memory provides an environment for the operation of the operating system and the human-computer collaboration program based on the human digital twin model in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the human-computer collaboration program based on the human digital twin model is executed by the processor, the steps of any one of the above-mentioned human-computer collaboration methods based on the human digital twin model are implemented. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0141] Those skilled in the art will understand that Figure 5 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] In one embodiment, a smart terminal is provided, which includes a memory, a processor, and a human-computer collaboration program based on a human digital twin model stored in the memory and executable on the processor. When the human-computer collaboration program based on the human digital twin model is executed by the processor, the steps of any one of the human-computer collaboration methods based on the human digital twin model provided in an embodiment of the present invention are implemented.

[0143] An embodiment of the present invention also provides a computer-readable storage medium, on which a human-computer collaboration program based on a human digital twin model is stored. When the human-computer collaboration program based on a human digital twin model is executed by a processor, the steps of any one of the human-computer collaboration methods based on a human digital twin model provided in an embodiment of the present invention are implemented.

[0144] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0145] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0146] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0147] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0148] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal equipment and method can be implemented in other ways. For example, the apparatus / terminal equipment embodiments described above are only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, other division methods can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0149] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A human-machine collaboration method based on a human digital twin model, characterized in that: include: Obtain target depth images and target video streams in human-machine collaboration scenarios; Using a three-dimensional posture parameterization network in a human digital twin model to perform three-dimensional posture parameterization on the target depth image to obtain three-dimensional posture parameters, and constructing a human posture based on the three-dimensional posture parameters; Based on the human body posture, a human body joint point sequence is extracted, and a preset spatiotemporal graph convolutional network is used to perform feature extraction on the human body joint point sequence to obtain global joint point features; Using the risk assessment network in the human digital twin model to conduct risk assessment on the global joint point features to obtain an ergonomic risk index; Using a behavior intention recognition network in a human digital twin model to identify and classify the global joint point features and the target video stream, and obtain a classification result of the behavior intention; Based on the classification results of the human body posture, the ergonomic risk index and the behavioral intention, make a behavioral decision and plan a motion trajectory; The method of using a preset spatiotemporal graph convolutional network to extract features from the human joint point sequence to obtain global joint point features includes: The human joint point sequence is split into several human joint point sub-sequences, and the human joint point sub-sequences are processed respectively by using a matching spatiotemporal graph convolutional network to obtain corresponding local spatiotemporal graphs; and the features of all the local spatiotemporal graphs are fused using a preset aggregation network to obtain global joint point features.

2. The human-machine collaboration method based on the human digital twin model according to claim 1 is characterized in that: The method uses a three-dimensional posture parameterization network in a human digital twin model to perform three-dimensional posture parameterization on the target depth image to obtain three-dimensional posture parameters, and constructs a human posture based on the three-dimensional posture parameters, including: Performing feature extraction on the target depth image to obtain target human features; The three-dimensional posture parameterization network in the human body digital twin model is used to perform three-dimensional posture parameterization on the target human body features to obtain three-dimensional posture parameters, and the three-dimensional posture parameters at least include spatial rotation parameters, spatial translation parameters, posture parameters and shape parameters; Using a preset parameterized human body model to perform parameterization processing on the posture parameters and the shape parameters to obtain a dense human body grid; Based on the dense grid of the human body, the spatial rotation parameters and the spatial translation parameters, a trained three-dimensional human body posture perception network is used to perform three-dimensional posture and joint estimation to obtain the human body posture.

3. The human-machine collaboration method based on the human digital twin model according to claim 1 is characterized in that: The behavior intention recognition network in the human digital twin model is used to identify and classify the global joint point features and the target video stream to obtain the classification result of the behavior intention, including: Extracting features from the target video stream to obtain spatiotemporal data features; Superimposing the global joint point feature and the spatiotemporal data feature to obtain a fusion feature; The behavior intention recognition network in the human digital twin model is used to identify and classify the fusion features to obtain the classification results of the behavior intention.

4. The human-machine collaboration method based on the human digital twin model according to claim 1 is characterized in that: The step of making a behavior decision and planning a motion trajectory based on the classification result of the human body posture, the ergonomic risk index and the behavior intention includes: Based on the classification result of the behavioral intention, obtaining a behavioral decision result; Based on the behavior decision result, motion path planning is performed using the human body posture and the ergonomic risk index to obtain a motion trajectory that matches the behavior decision result.

5. The human-machine collaboration method based on the human digital twin model according to claim 1 is characterized in that: The method further includes the step of training the human digital twin model, specifically including: Collecting depth image samples and video stream samples in a human-machine integration scenario to obtain a training set, and manually annotating the training set to obtain a manual annotation result; Inputting the depth image samples into an initial three-dimensional posture parameterization network to generate human body posture samples; Comparing the human body posture sample with the manual annotation result and optimizing the training to obtain the three-dimensional posture parameterized network; Using the three-dimensional posture parameterization network, updating the human body posture samples, and generating global joint point feature samples based on the human body posture samples; Inputting the global joint feature sample into an initial risk assessment network to obtain an ergonomic risk index sample; Inputting the global joint point features and the video stream samples into an initial behavior intention recognition network to obtain a classification result sample of the behavior intention; Comparing the ergonomic risk index sample and the behavioral intention classification result sample with the manual annotation result at the same time and optimizing the training to obtain the risk assessment network and the behavioral intention recognition network; The human body digital twin model is constructed based on the three-dimensional posture parameterization network, the risk assessment network and the behavior intention recognition network.

6. The human-machine collaboration method based on the human digital twin model according to claim 1 is characterized in that: The target video stream is preprocessed, including: Based on the target processing accuracy, preset feature map dimensions; At least one of segmentation, sampling and / or scaling is performed on the target video stream according to the feature map dimension to update the target video stream.

7. A human-machine collaboration device based on a human digital twin model, characterized in that: include: The data acquisition module is used to obtain the target depth image and target video stream in the human-machine integration scene; A human posture reconstruction module, used to perform three-dimensional posture parameterization on the target depth image using a three-dimensional posture parameterization network in a human digital twin model, obtain three-dimensional posture parameters, and construct a human posture based on the three-dimensional posture parameters; A joint point feature extraction module is used to extract a human joint point sequence based on the human body posture, and use a preset spatiotemporal graph convolutional network to extract features of the human joint point sequence to obtain global joint point features; The method of extracting features from the human joint point sequence using a preset spatiotemporal graph convolutional network to obtain global joint point features includes: splitting the human joint point sequence into a plurality of human joint point subsequences, and using a matching spatiotemporal graph convolutional network to process the human joint point subsequences respectively to obtain corresponding local spatiotemporal graphs; and using a preset aggregation network to perform feature fusion on all the local spatiotemporal graphs to obtain global joint point features. An ergonomic risk index assessment module, used to perform risk assessment on the global joint point features using a risk assessment network in a human digital twin model to obtain an ergonomic risk index; A behavior intention recognition module is used to use the behavior intention recognition network in the human digital twin model to identify and classify the global joint point features and the target video stream to obtain a classification result of the behavior intention; A behavior decision-making and planning module is used to make behavior decisions and plan movement trajectories based on the classification results of the human body posture, the ergonomic risk index and the behavior intention.

8. Intelligent terminal, characterized in that: The intelligent terminal includes a memory, a processor, and a human-computer collaboration program based on a human digital twin model stored in the memory and executable on the processor. When the human-computer collaboration program based on the human digital twin model is executed by the processor, the steps of the human-computer collaboration method based on the human digital twin model as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a human-computer collaboration program based on a human digital twin model. When the human-computer collaboration program based on a human digital twin model is executed by a processor, the steps of the human-computer collaboration method based on a human digital twin model as described in any one of claims 1 to 6 are implemented.

Images