Man-machine interaction system and device
Through real-time security perception module and multimodal interaction module, combined with deep learning and sensor data, accurate detection and prediction of human behavior is achieved, and the safety hazards in the human-computer interaction environment in the existing technology are solved, and the safety is improved and energy efficiency.
Patent Information
- Application Number
- CN202411356824.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2024-11-29
AI Technical Summary
The prior art is difficult to accurately perceive human position and motion trajectory in a human-computer interactive environment, resulting in safety risks. The data obtained by sensors is complex and difficult to predict the subsequent motion trend of humans.
The real-time security perception module is used to fusion of multiple sensor data to perform dynamic 3D reconstruction, and the deep learning model is used to extract features, combine hand and eye focus information, predict human behavior, and adjust the state of the robot to avoid collisions and misoperation.
Accurate detection and prediction of human motion trajectories is achieved, the risk of human-machine collision is reduced, the safety of the human-machine coexistence environment is improved, and the long-term high computing power state of the robot is reduced through an energy-saving mechanism, saving energy.
Smart Images

Figure CN119036483A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of information data processing, and in particular relates to a human-computer interaction system and device. Background Art
[0002] With the rapid development of AI and robotics technology, robots are appearing in more and more work and life scenarios. Although robots can assist humans in completing work, there are also certain safety hazards in an environment where humans and robots coexist. These safety hazards mainly include collision risks and misjudgment / misoperation risks. If the robot cannot accurately sense the position of a person and predict the motion trajectory in real time, a human-machine collision is likely to occur, or an incorrect operation will be performed, thereby threatening the safety of personnel.
[0003] In the prior art, the means to solve this kind of safety hazard mainly include installing sensors, such as laser scanners, cameras, radars and infrared sensors, to monitor the environment around the robot in real time, including people, obstacles, etc. However, the data obtained by these sensors are relatively independent, and the data provided to the robot is too complicated, which can easily lead to the robot's misjudgment; and the data obtained by the sensors is also difficult to predict the next movement trend of humans.
[0004] Therefore, if a new human-computer interaction system can be provided that can not only detect a person's position in real time but also make certain predictions about a person's movement trajectory, it is expected to reduce safety risks and improve the safety of the human-computer coexistence environment. Summary of the invention
[0005] The purpose of the present invention is to provide a human-computer interaction system and device that facilitates human-computer collaboration.
[0006] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a human-computer interaction system, the system includes a real-time safety perception module; the real-time safety perception module is used to perceive, collect information and predict human behavior in real time, collect information through sensors, and fuse the collected information, and use the fused data to perform dynamic 3D reconstruction;
[0007] Data fusion methods include:
[0008] 1-1 Data synchronization and preprocessing: synchronize the data in time and align it in space.
[0009] 1-2 Feature extraction and fusion: Use deep learning models to extract features from data, and use feature fusion technology to combine the features to form a comprehensive feature vector.
[0010] Preferably, the system includes a multimodal interaction module; the multimodal interaction module recognizes commands and intentions through human motion and posture tracking; the motion and posture tracking includes hand posture tracking, and the hand focus is obtained in real time by allocating weights to the joints of the human hand.
[0011] Preferably, the hand focus obtaining method includes:
[0012] 2-1 Define the coordinate system of the hand joints: Each joint of the finger is represented by three coordinates (x, y, z). In three-dimensional space, the set of hand joints is:
[0013] P = {p i |i=1,2,…,n}, where p i =(x i ,y i ,z i ), represents the coordinates of the i-th joint point;
[0014] 2-2 Define hand focus FP:
[0015] w i is related to each joint point p i The relevant weights;
[0016] 2-3 Calculate the hand focus position:
[0017] At time t, the hand focus is: w i (t) and p i (t) represent the weight and coordinate of the joint point at time t respectively.
[0018] Preferably, the movement and posture tracking includes eye focus tracking.
[0019] Preferably, according to the obtained coordinates of the hand focus and the eye focus, it is determined whether the hand focus and the eye focus are currently in a convergence or separation state;
[0020] If in the separated state, the system enters an energy-saving state, and there is no need to separately warn humans that the robot is approaching;
[0021] If the robot is in the converging state, the system enters the warning state and needs to warn humans to inform them that the robot is approaching. The warning method can be selected as sound warning, light warning, etc.
[0022] Preferably, the system includes a workspace management module, which divides the area into functions and controls the robot according to the human-machine distance and movement trend; when it is detected that the human-machine distance is less than a first set value and the human-machine movement trend is approaching each other, the system real-time data acquisition frequency is increased; when it is detected that the human-machine distance is less than a second set value, the robot is controlled to shut down in whole or in part; the first set value>the second set value.
[0023] Accordingly, a hand gesture recognition method comprises the following steps:
[0024] 2-1 Define the coordinate system of the hand joints: Each joint of the finger is represented by three coordinates (x, y, z). In three-dimensional space, the set of hand joints is:
[0025] P = {p i |i=1,2,…,n}, where p i =(x i ,y i ,z i ), represents the coordinates of the i-th joint point;
[0026] 2-2 Define hand focus FP:
[0027] w i is related to each joint point p i The relevant weights;
[0028] 2-3 Calculate the hand focus position:
[0029] At time t, the hand focus is: w i (t) and p i (t) represent the weight and coordinate of the joint point at time t respectively.
[0030] Correspondingly, a computer device / equipment / system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the hand gesture recognition method.
[0031] Accordingly, a computer-readable storage medium stores a computer program / instruction thereon, which implements the steps of the system or the hand gesture recognition method when executed by a processor.
[0032] Accordingly, a computer program product comprises a computer program / instruction, which implements the steps of the system or the hand gesture recognition method when executed by a processor.
[0033] The present invention has the following beneficial effects: The present invention provides a new human-machine exchange system, which can collect data in real time through multiple sensors, perform data fusion, and realize dynamic 3D reconstruction based on the fused data to detect and predict the movement trajectory and possible behavior of people. The system also includes a multimodal interaction module, which can detect and predict human behavior in real time, and judge whether the person is currently in a focused state or a non-focused state based on the person's behavior, so as to adjust the state of the robot to avoid the robot being in a high computing power state / warning state for a long time to save energy. The system also includes a workspace management module, which divides the area and issues instructions based on the distance between people and machines and the movement trend, further avoiding the robot being in a high computing power state / warning state for a long time. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 This is a schematic diagram of the system components;
[0035] Figure 2 Schematic diagram of the hand joints. DETAILED DESCRIPTION
[0036] The present invention provides a human-machine interaction system and device that facilitates human-machine collaboration. The human-machine interaction system and device are mounted on a robot, and the human-machine interaction system can collect current state information in real time, identify, analyze, and judge the information, and issue instructions. The robot performs current and subsequent operations based on the instructions, thereby preventing the robot from accidentally injuring an operator when performing operations in a human-machine collaborative environment, such as a laboratory where humans and machines coexist.
[0037] like Figure 1 As shown, the human-computer interaction system includes: a real-time safety perception module and a multimodal interaction module, and may also include a workspace management module. It should be noted that the human-computer interaction system may also add other known modules of the prior art as needed, such as a map entry and recognition module, a direction recognition and control module, a voice module, a sound / light control module, etc.
[0038] 1. Real-time security perception module
[0039] For robots working in a human-machine collaborative (or coexistence) environment, the primary issue that needs to be considered is how to perceive and predict human behavior in real time in a complex and dynamic working environment and make safe decisions to avoid potential collisions and accidents.
[0040] In order to solve this problem, the human-computer interaction system is equipped with a real-time security perception module to collect data, realize 3D reconstruction of the environment, and perceive the environmental conditions in real time, so as to realize real-time updating and correction of the environment and realize dynamic 3D reconstruction of the environment.
[0041] The perception module can capture information. The captured information includes the distance between the physical boundary between the human and the robot (such as the boundary between the human's hand, head, etc. and the robot), the trend of the human and the robot approaching or leaving each other (this information is used to predict whether the human and the robot are about to interact with each other), and the rate of approach or departure between the human and the robot. The perception module can optionally use a high-speed camera to capture information. Through the real-time safety perception module, the robot can predict the actions that humans may make, thereby avoiding collisions with humans.
[0042] The perception module also includes a real-time perception system of multi-sensor fusion to build a dynamic 3D model and correct and update the model in real time according to the currently collected data and information. The real-time perception system includes a lidar, an RGB camera, a depth camera, an ultrasonic sensor, an inertial measurement unit, and an infrared sensor (hereinafter referred to as "sensor" unless otherwise specified).
[0043] Among them, the laser radar is used to scan and generate high-precision three-dimensional point cloud data of the robot's working environment (such as a laboratory) to achieve environmental modeling and obstacle detection. The robot's working environment may change, so every time the robot starts working, it will re-scan and detect, and correct the previous simulated environment. The depth camera is used to obtain the depth information of the scene (robot's working environment), integrate it with the data obtained by the laser radar, and perform three-dimensional reconstruction. The RGB camera is used to capture images and videos of the working environment and the current movement of people and robots, and identify and classify targets (people, obstacles). Ultrasonic sensors are used for short-range obstacle detection. The inertial measurement unit (IMU) is used to measure the acceleration and angular velocity of the current movement of people and robots to achieve human motion tracking and posture estimation. The infrared sensor is used to detect temperature changes and the location of heat sources to accurately identify the location of people and predict position changes.
[0044] The real-time perception system can detect and fuse data in real time, and then use the fused data for dynamic 3D reconstruction. Specifically, when the robot starts working, the lidar and depth camera detect the robot's environment (including obstacles in the environment), fuse the data, and reconstruct the environment in 3D. Subsequently, during the entire process of the robot working according to the operating instructions (such as commanding the robot to clean, transport materials, conduct certain experimental operations, etc.), the RGB camera, inertial measurement unit and infrared sensor remain in the started state, and the data detected by the camera, inertial measurement unit and infrared sensor are integrated to detect in real time whether someone is about to approach the robot, and predict the actions that humans who are about to approach the robot may take, adjust the robot's actions based on the actions, and correct the 3D reconstructed environment (for example, delete objects that are actually people but are identified and generated as obstacles from the environment). The robot instructions can be set and supplemented / updated at any time as needed so that the robot can adjust its actions according to the predicted human actions. For example: if it is predicted that a person is just about to pass by the robot, the robot will be controlled to stop approaching the side where the person is about to pass to avoid a collision; if it is predicted that the person will raise his hand to operate the robot's operating console (such as the operating screen, operating buttons, etc.), the robot will be controlled to pause and wait for human operation.
[0045] 1-1 The methods for data fusion of each sensor include:
[0046] 1-1-1 Data synchronization and preprocessing: The data from the depth camera, ultrasonic sensor, and IMU are synchronized in time and aligned in space to ensure that different data sources at each moment can be aligned with each other. Among them, the depth map generated by the depth camera is converted into point cloud data, which is continuously updated to reflect the dynamic changes in the scene. The close-range detection data provided by the ultrasonic sensor can be used to fill or correct the data gaps in the point cloud. IMU data is used to estimate the motion trajectory and posture changes of the sensor device. This helps to accurately map the data from the depth camera and ultrasonic sensor into a global coordinate system, thereby generating a consistent dynamic 3D model. Filtering algorithms (such as Kalman filters and particle filters) are used in the process of data fusion to optimize the accuracy and robustness of the data.
[0047] Time synchronization refers to the precise time alignment of data from multiple sensors to ensure that all data are comparable at the same time point. Optionally, time synchronization can be performed using timestamps. The data collected by each sensor is accurately timestamped using the Network Time Protocol (NTP), and then the data is time-aligned using interpolation. Spatial alignment can ensure that the data between sensors can be mapped to a common spatial reference system. Optionally, calibration can be performed using a calibration plate, that is, the camera's intrinsic and extrinsic parameters are calibrated using a calibration plate (such as a checkerboard pattern), and the position transformation matrix between cameras is calculated by observing the calibration plate multiple times. Sensor data is preprocessed by noise filtering to eliminate or reduce measurement noise. The average value of multiple consecutive data points is taken by the existing mean filtering method to reduce high-frequency noise, which is used for smoothing image or distance measurement data. Finally, the image data collected by the sensor (such as a depth map) is converted into numerical data (for example, pixel values are converted into distance values or depth values) to ensure that all data have the same representation format; and then the data collected by different sensors are converted to the same numerical range or scale through normalization processing to ensure data consistency for subsequent dynamic 3D reconstruction.
[0048] 1-1-2 Feature extraction and fusion: Extract features from data from different sensors and use the deep learning model CNN for feature extraction. Among them, geometric features that describe the geometric structure and shape of objects in the scene are extracted from the information collected by the depth camera, such as depth value, point cloud data, surface normal vector, curvature information, etc. Distance features (robot and human, robot and object) are extracted from the information collected by the ultrasonic sensor, especially the distance features in close-range perception or in the presence of occlusion. Acceleration features (such as speed, acceleration change rate, etc.) and angular velocity features are extracted from the information collected by the inertial measurement unit (rotational motion features of objects, such as angular velocity, angular acceleration, etc.)
[0049] Use existing time series models (such as LSTM) to extract time series features from inertial measurement unit data or ultrasonic sensor data. The LSTM model can capture the dynamic characteristics of data that change over time, such as speed changes, trajectory patterns, and other features.
[0050] Feature fusion techniques (such as multimodal fusion and attention mechanism) are used to combine the features of different sensors to form a comprehensive feature vector.
[0051] The preferred solution is that after obtaining the comprehensive feature vector, the following can be further performed: 1-1-3 Behavior prediction and decision-making: Input the comprehensive feature vector into the behavior prediction model (such as LSTM) to predict the person's movements and behaviors. Combine the behavior prediction results with the environmental perception information to make real-time decisions and realize human-computer interaction.
[0052] The preferred solution is to combine deep learning and behavior prediction algorithms when predicting human movements and behaviors. The algorithms specifically include: Convolutional Neural Networks (CNN), which use images captured by cameras for object recognition, environmental understanding, and human posture estimation; Long Short-Term Memory Networks (LSTM), which are used to analyze and predict behaviors based on time series, such as predictions of human movement sequences; Recurrent Neural Networks (RNN), which combine sensor data for motion pattern recognition and behavior prediction.
[0053] The preferred solution is to add a real-time obstacle avoidance module to sense obstacles (people or objects) in the environment, calculate the robot's reasonable path based on the position and motion information of the obstacles, and update the path in real time based on the collected information to save the motion path while avoiding collisions. The real-time obstacle avoidance module is a mature existing technology, such as Intel's RealSense camera system combined with depth perception technology, which can be used for real-time obstacle avoidance of the robot.
[0054] 1-2 Dynamic 3D reconstruction using fused data
[0055] The steps include: inputting the fused data (comprehensive feature vector) of all sensors into 3D reconstruction software (such as COLMAP) to generate a real-time updated 3D model for monitoring the human-computer interaction process. The frequency and accuracy of model updates depend on the efficiency of data acquisition and fusion.
[0056] 2. Multimodal interaction module
[0057] In order to improve the operability of the human-computer interaction system and make it easier for researchers to control and direct robots to perform tasks, the human-computer interaction system also includes a multimodal interaction module, which uses eye tracking, hand gestures, and even facial gestures to achieve multimodal interaction, thereby helping robots to perform instructions and intention recognition through eye tracking and hand gestures.
[0058] 2-1 Eye Tracking
[0059] Eye tracking is performed using an infrared eye tracker. Specifically, an infrared light source is used to illuminate the eyes, capture the position of the reflected light spot and the pupil position of the eyes, and calculate the gaze point. Calculating the gaze point is a mature existing technology, such as using EyeLink. Eye tracking software libraries (such as openCV) can provide eye feature point detection and pupil tracking algorithms.
[0060] 2-2 Hand pose tracking
[0061] For hand gesture recognition and prediction, existing technologies often rely on identifying the positions of finger joints, which are easily affected by occlusion, posture deformation, lighting changes, etc., thus affecting the recognition effect. At the same time, existing methods are prone to confuse the transitions between different gestures when recognizing continuous hand movements, resulting in reduced recognition accuracy.
[0062] The present invention provides a new hand gesture recognition and prediction method, which no longer focuses on the finger joints but instead focuses on the changes in hand focus, thereby reducing dependence on specific finger positions and improving robustness. Through time series analysis of focus transfer, the dynamic changes between gestures can be better captured, thereby improving the efficiency and accuracy of human-computer interaction.
[0063] A human hand has five fingers. The change in the relative positions of the five fingers will shift the focus of the entire hand between different fingers (mainly the thumb, index finger and middle finger, and the ring finger and little finger are rarely used as the main functional fingers). Collect as many hand postures as possible and model and calculate the corresponding hand focus. The hand posture recognition method includes the following steps:
[0064] Assume that the hand is a system composed of several joints (the joints of the fingers and the wrist). Each finger has its joint (from the root to the fingertip), and the focal position of the hand can be regarded as the model center of these joints.
[0065] 2-2-1 Hand joint coordinate system
[0066] Set the following joints: Each joint of the finger is represented by three coordinates (x, y, z). The hand has 20 joints (including 1 joint in the wrist, 3 joints in the thumb, 4 joints in the index finger, 4 joints in the middle finger, 4 joints in the ring finger, and 4 joints in the little finger). The schematic diagram of the hand joints is as follows Figure 2 As shown. In three-dimensional space, the set of joint points defining the hand is:
[0067] P = {p i |i=1,2,…,n}, where p i =(x i ,y i ,z i ), represents the coordinates of the i-th joint point, and n is a natural number.
[0068] 2-2-2 Define the hand focus
[0069] The hand focus is a dynamically calculated point that represents the "center of mass" of the hand pose. This point is defined as the weighted average position of all joints. The formula is:
[0070] Among them, FP refers to the hand focus in the current posture.i is related to each joint point p i The relevant weight. Weight w i It can be set according to the importance of the joints, the frequency of use of the fingers, and the specific requirements of the gesture. For example, the thumb, index finger, and middle finger have greater weights in most gestures, which can be selected as 0.15, 0.3, and 0.3, while the weights of the ring finger and pinky finger are relatively small, which can be selected as 0.05 and 0.05. The weight of the wrist can be selected as 0.15.
[0071] 2-2-3 Dynamic Calculation of Focus Position
[0072] Most existing gesture recognition methods extract and classify the overall shape and posture of the hand, rather than explicitly calculating the position of the hand's "focus" and its time-varying characteristics. The present invention regards the change of hand focus as a dynamic system and understands actions and gestures through the relative position changes of fingers. The hand focus calculation method includes:
[0073] When the hand posture changes, the position p of each joint point i =(x i ,y i ,z i ) will change accordingly, and the focus FP will also change dynamically. This dynamic change can be captured and analyzed through time series data. At time t, the hand focus is:
[0074] Among them, w i (t) and p i (t) represent the weight and coordinate of the joint point at time t respectively.
[0075] 2-2-4 Action and Future State Prediction
[0076] By analyzing the changing trend of the hand focus FP(t) through the time series model, the future movement state and posture of the hand can be predicted, and preparations can be made for human-computer interaction in advance.
[0077] The LSTM model is used to model the time series data {FP(t)} to capture the temporal dependency of the hand focus change. The specific process is: collect continuous hand focus position sequence data. For example, a sequence can represent the time from t1 to t nThe hand focus changes. Standardize the position data to be in the range of [0,1] or other suitable ranges to prevent data of different dimensions from affecting the training of the model. Divide the dataset into training set, validation set, and test set. 80% of the data is used for training, 10% of the data is used for validation, and 10% of the data is used for testing. Use a deep learning framework such as TensorFlow to build an LSTM model. Based on the trained model, it is possible to predict the future focus position given the hand focus data in the current and past period of time, thereby inferring the next action for human-computer interaction.
[0078] The following is an example of calculating the hand focus when pinching a needle with the thumb and middle finger. The coordinate data of the thumb (t), index finger (m), middle finger (i), ring finger (r), little finger (p) and wrist (w) are obtained and calculated by the sensor as follows (unit: mm):
[0079] Pt=(20,50,10); Pm=(22,52,12); Pi=(40,55,15); Pr=(45,60,12);
[0080] Pp=(50,65,11); Pw=(62,44,18).
[0081] Set weights: wt=wm=0.3, wi=0.15, wr=wp=0.05, ww=0.15.
[0082] Substitute the formula to calculate the hand focus:
[0083] FPx=(20×0.3+22×0.3+40·0.15+45×0.05+50×0.05+62×0.15) / (0.3+0.3+0.15+0.05+0.05+0.15)=32.65;
[0084] FPy=(50×0.3+52×0.3+55×0.15+60×0.05+65×0.05+44×0.15) / (0.3+0.3+0.15+0.05+0.05+0.15)=48.7;
[0085] FPz=(10×0.3+12×0.3+15×0.15+12×0.05+11×0.05+18×0.15) / (0.3+0.3+0.15+0.05+0.05+0.15)=12.70;
[0086] FP = (32.65, 48.7, 12.70).
[0087] Through the above calculations, it is determined that the hand focus when the thumb and middle finger pinch the needle is mainly concentrated between the thumb and middle finger, and is slightly affected by other fingers and wrist. This calculation process can dynamically update the position of the hand focus and adjust it in real time as the relative positions of the fingers change, which helps to capture small and fast gesture changes.
[0088] By obtaining the eye focus (gaze point) and hand focus through the above method, the next action of the person can be predicted, and the action instructions of the robot after receiving the focus information can be set according to the actual situation. For example, if the purpose is only to avoid a collision between the person and the robot, the robot can be commanded to bypass the current and upcoming positions of the eye focus and / or hand focus to avoid a collision. If the purpose is to let the robot assist the person in doing part of the work, the robot can be commanded to reach the focus position to implement special instructions. Specific hand instructions can also be set according to the different hand focuses. After the robot collects specific hand instructions, it takes corresponding actions.
[0089] 2-3 Analysis of focus convergence or separation
[0090] More preferably, if the purpose is just to prevent human-machine collision, or just to instruct the robot to implement simple human-machine interaction instructions such as handing an item to a human, then the data stream can be further used to capture and analyze whether the human eye focus and hand focus are converging or separating. Then, different operations can be performed according to the situation of focus converging or separating.
[0091] When the focus of the eyes and hands is separated, it is judged that the human is familiar with the current behavior, and further judged that there is no need to specially warn and guide the human (visual guidance, sound warning, etc.), so that the human can notice the robot approaching. Therefore, at this time, the robot can reduce the calculation rate and save energy. It only needs to slow down the speed of movement or the speed of command implementation (such as the speed of passing items, etc.), and there is no need to set special warnings (sound warnings, special text warnings on the display, etc.). If the focus overlaps, it is judged that the human is concentrating on the current action. At this time, the robot needs to pause the action, or reduce the speed of action to a lower level, and set certain warnings to inform the human that the robot is approaching, or is about to perform interactive behaviors (passing items, etc.).
[0092] A preferred solution is that the focus convergence / separation state analysis method includes:
[0093] 2-3-1 Data synchronization and integration
[0094] Time synchronization: Use the existing technology Network Time Protocol (NTP) to ensure that the eye focus data stream and the hand focus data stream are aligned on the same time axis based on the timestamp.
[0095] Unified spatial coordinates: The gaze point provided by the eye tracker is in the screen coordinate system, while the hand focus is in 3D space. They need to be mapped to the same coordinate system using a projection transformation matrix. This can be done using existing technologies such as the camera calibration function provided by OpenCV.
[0096] 2-3-2 Distance calculation and dynamic threshold setting
[0097] Calculate the Euclidean distance d(t) between the eye focus Pe(t) = (xe(t), ye(t), ze(t)) and the hand focus FP(t) = (xh(t), yh(t), zh(t)) at each time point t:
[0098]
[0099] Among them, d(t) is the Euclidean distance, x e ,y e and z e They are the three-dimensional coordinate values of the eye focus at time t, x h ,y h and z h They are the three-dimensional coordinate values of the hand focus at time t.
[0100] Set a threshold ∈ to determine whether the focus is converging or separating. If d(t)≤∈, the focus is considered to be "converging" (i.e., the eye focus and the hand focus are relatively close). If d(t)>∈, the focus is considered to be "separated" (i.e., the eye focus and the hand focus are relatively far). In order to improve safety, the threshold can be appropriately increased.
[0101] 2-3-3 Time Series Analysis
[0102] Use time series analysis methods (such as moving average method) to smooth the noise in the data stream and determine the changing trend of eye focus and hand focus. For example, use ruptures library for change point detection to detect the location of significant changes in the time series, such as changes in trend, mean or variance, to analyze the time point when the focus position suddenly changes.
[0103] A more preferred solution includes: combining augmented reality (AR) and virtual reality (VR) technologies. For example, the operator can be equipped with AR glasses, and a camera that captures facial and / or eye image information can be set on the AR glasses, and the images captured by the camera can be analyzed in real time to improve the quality and speed of data capture and recognition, so as to better cooperate with the robot.
[0104] 3. Workspace management module
[0105] In order to effectively manage the coexistence of humans and robots in a limited workspace and ensure that humans and robots can work together efficiently and safely, a more preferred solution is that the system also includes a workspace management module. The workspace management module uses artificial intelligence to pre-learn the work areas of robots and humans, identify which spaces are where humans and robots coexist, and implement special instructions in the human-machine coexistence area. The area identification and instruction issuance method includes:
[0106] 3-1 Data Collection and Labeling
[0107] Collect environmental data, including images captured by high-definition cameras, laser radar scan data, and depth images from depth sensors. The data collection frequency can be set to 10-30 frames per second, the laser radar can be set to 5,000 scans per second, and the depth sensor can be set to 30 frames per second. The amount of labeled data is preferably 1,000 hours. The environmental space is specifically divided into: areas where only humans exist, areas where only robots exist, and areas where humans and machines coexist. Data collection and area division can help the system effectively distinguish and identify different work areas.
[0108] 3-2 Feature Extraction
[0109] Extract key information from the image and depth data collected in step 3-1. For image data, use the pre-trained convolutional neural network ResNet50 to extract the shape, texture and position features of the object; for depth data, obtain the distance and spatial layout information of the object through a deep convolutional network; for behavioral data, analyze the motion trajectory through a long short-term memory network and extract time series features. These extracted features are fused to form a comprehensive feature vector to accurately describe the working area. The extraction and fusion method of the comprehensive feature vector is carried out with reference to step 1-1.
[0110] 3-3 Model Training
[0111] A multi-level fusion model is used, in which convolutional neural networks are used for image processing, long short-term memory networks are used for behavioral data processing, and features are integrated through fully connected layers. The training data includes images, depth maps, and motion data. The training cycle can be set to 100 cycles, the batch size can be set to 64, and the learning rate can be set to 0.001.
[0112] 3-4 Dynamically adjust the command issuance
[0113] In order to be able to predict the human-machine interaction process based on the model and dynamic 3D modeling results, the present invention also provides an adaptive attention mechanism to determine the possibility of human-machine interaction. The adaptive attention mechanism is preferably started when the robot enters the human-machine coexistence area to save computing power. According to the current distance and movement trend of the human-machine, the attention of the system is divided into three items: close distance, long distance and ultra-long distance. Close distance means that the relative distance between the human and the machine is within a certain range, and the possibility of interaction between the human and the machine is high. Long distance means that the relative distance between the human and the machine is within a certain range, but the possibility of human-machine interaction is low; or the relative distance between the human and the machine exceeds a certain range, but the mutual movement trend of the human and the machine is close, and there is a possibility of interaction within a certain time. Ultra-long distance means that the relative distance between the human and the machine exceeds a certain range, and there is no possibility of interaction between the human and the machine within a certain time. The attention allocation calculation formula in the adaptive attention mechanism is shown in Table 1. In Table 1, d represents the distance between the human and the robot, in mm. According to the different distances, the calculation formulas of each attention are adjusted accordingly, as shown in Table 1.
[0114] Table 1 Core formulas in the attention allocation mechanism
[0115]
[0116] A=10 / (1+np.var(historical_data)); np.var(historical_data) refers to a function call that uses the NumPy library (abbreviated np) to calculate the variance of the historical_data array or data set.
[0117] α=0.023*(1+np.mean(historical_data) / 100); np.mean(historical_data) refers to a function call that uses the NumPy library to calculate the arithmetic mean (mean) of the historical_data array or data set.
[0118] B=0.001*(1+np.std(historical_data)); np.std(historical_data) refers to a function call that uses the NumPy library to calculate the sample standard deviation of the historical_data array or data set.
[0119] k=2*(1+np.median(historical_data) / 100); np.median(historical_data) refers to a function call that uses the NumPy library to calculate the median (median) of the historical_data array or data set.
[0120] C=C0(1+σ / μ); β=β0(1+μ / d max ); D = D0 (1 + N collisions / N total ).
[0121] β represents a natural coefficient, which is usually selected in the range of 0.3-1.0, preferably 0.6. C0, β0 and D0 are initial coefficients, which are 1, 30 and 5.5 respectively. μ represents the mean, σ represents the standard deviation, dmax represents the maximum distance, and N collisions Represents the number of historical collisions, N total Indicates the total amount of data.
[0122] Combined with the human-machine motion trend and the attention values calculated based on the distance, it is determined whether it is necessary to issue instructions at the moment, which attention should be given priority, and what instructions should be issued. When the calculated attention value is > 0.5, it is determined that the item is the core item for decision-making human-machine interaction, and the item is used as the main attention in human-machine interaction to issue instructions. The specific instruction issuance method is shown in Table 2.
[0123] Table 2 Core formulas in the attention allocation mechanism
[0124]
[0125]
[0126] It should be noted that the specific values such as the spacing values and frequency setting values in Tables 1 and 2 are examples, and the actual data can be set specifically according to the location, human-computer interaction status, etc.
[0127] The embodiments described above are only descriptions of the preferred modes of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations, modifications, and substitutions made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.
Claims
1. A human-computer interaction system, characterized in that: The system includes a real-time safety perception module; the real-time safety perception module is used to perceive, collect information and predict human behavior in real time, collect information through sensors, fuse the collected information, and use the fused data to perform dynamic 3D reconstruction; Data fusion methods include: 1-1 Data synchronization and preprocessing: synchronize the data in time and align it in space. 1-2 Feature extraction and fusion: Use deep learning models to extract features from data, and use feature fusion technology to combine the features to form a comprehensive feature vector.
2. The system according to claim 1, characterized in that: The system includes a multimodal interaction module; the multimodal interaction module performs instruction and / or intention recognition through human action and posture tracking; the action and posture tracking includes hand posture tracking, and the hand focus is obtained in real time by weighting the joints of the human hand.
3. The system according to claim 2, characterized in that: The hand focus obtaining method comprises: 2-1 Define the coordinate system of the hand joints: Each joint of the finger is represented by three coordinates (x, y, z). In three-dimensional space, the set of hand joints is: P = {p i |i=1,2,…,n}, where p i =(x i ,y i ,z i ), represents the coordinates of the i-th joint point, and n is a natural number; 2-2 Define hand focus FP: w i Each joint point p i The weight of 2-3 Calculate the hand focus position: At time t, the hand focus is: w i (t) and p i (t) represent the weight and coordinate of the joint point at time t respectively.
4. The system according to claim 2, characterized in that: The motion and posture tracking includes eye focus tracking.
5. The system according to claim 4, characterized in that: According to the obtained coordinates of the hand focus and the eye focus, the current positional relationship between the hand focus and the eye focus is analyzed to determine whether the hand focus and the eye focus are currently in a convergence state or a separation state; If in the separated state, the system enters an energy-saving state without warning that the human robot is approaching; If the robot is approaching, the system will enter the warning state and warn humans that the robot is approaching.
6. The system according to claim 1, characterized in that: The system includes a workspace management module, which controls the robot according to the human-machine distance and the relative movement trend; when it is detected that the human-machine distance is less than a first set value and the relative movement trend of the human and the machine is approaching each other, the system real-time data acquisition frequency is increased; when it is detected that the human-machine distance is less than a second set value, the robot is controlled to shut down in whole or in part; the first set value>the second set value.
7. A hand gesture recognition method, characterized in that: The method comprises the following steps: 2-1 Define the coordinate system of the hand joints: Each joint of the finger is represented by three coordinates (x, y, z). In three-dimensional space, the set of hand joints is: P = {p i |i=1,2,…,n}, where p i =(x i ,y i ,z i ), represents the coordinates of the i-th joint point, and n is a natural number; 2-2 Define hand focus FP: w i Each joint point p i The weight of 2-3 Calculate the hand focus position: At time t, the hand focus is: w i (t) and p i (t) represent the weight and coordinate of the joint point at time t respectively.
8. A computer device / equipment / system comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 7.
9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the system described in any one of claims 1 to 6 or the method described in claim 7 are implemented.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the system described in any one of claims 1 to 6 or the method described in claim 7 are implemented.
Citation Information
Cited By
Digital craftsman operation real-time interaction method and system
CN120447744A
