Vision-based machine human body motion intention recognition interaction method and system

Abstract

The application discloses a visual-based human motion intention recognition interaction method and system, and relates to the technical field of computer vision. The method comprises the following steps: acquiring three-dimensional scene modeling data, detecting object spatial distribution to obtain target spatial granularity, and constructing an adaptive three-dimensional spatial anchor frame accordingly; collecting facial and binocular feature points, and predicting a visual gaze track through a binocular parallax algorithm; calibrating and optimizing the track by using the three-dimensional anchor frame, outputting a target, generating an instruction after analyzing the motion semantics of the target, and sending the instruction to a robot interaction terminal. The technical problem of low computational efficiency and ambiguous intention association caused by anchor frame redundancy in a complex physical scene is solved for the traditional visual-based intention recognition method, the technical effects of self-adaptive matching of a three-dimensional spatial anchor frame with a scene physical scale, introduction of a calibration and optimization mechanism for a visual predicted track, accurate association of human motion intention and a target object, and improvement of the accuracy, real-time performance and reliability of intention recognition are achieved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a vision-based robot human motion intention recognition and interaction method and system. Background Technology

[0002] With the rapid development of intelligent manufacturing, service robots, and human-robot collaboration technologies, robots are increasingly widely used in industrial production, public services, and complex work environments. Robots need to accurately understand human actions and interaction intentions in open, dynamic real-world scenarios to achieve safe, efficient, and natural human-robot collaboration. However, most existing vision-based human action intention recognition technologies rely on two-dimensional image features or fixed-scale target detection models. In large-scale industrial scenarios such as open factories and steel mills, this easily generates a large number of invalid detection areas that do not match the actual physical environment, leading to high computational resource consumption and unstable target association, thus affecting the real-time performance and accuracy of intention recognition. Simultaneously, the spatial relationship between the human body and multiple target objects changes significantly in complex environments. Relying solely on a single perspective or simple geometric assumptions makes it difficult to accurately infer the human's focus and action intentions. Especially when robots perform collaborative, guidance, or response tasks, the inability to reliably determine the true interaction target of the human can easily lead to misoperations or interaction failures. Summary of the Invention

[0003] This application provides a vision-based robot human motion intention recognition interaction method and system, which solves the technical problems of low computational efficiency and ambiguous intention association caused by anchor frame redundancy in traditional vision-based intention recognition methods in complex physical scenes.

[0004] The first aspect of this application provides a vision-based robot human motion intention recognition interaction method, the method comprising: The process involves acquiring spatial modeling data of a 3D scene, performing spatial distribution detection on the objects in the spatial modeling data to obtain target spatial granularity, and constructing a 3D spatial anchor frame for visually assisted object recognition using the target spatial granularity. It also involves collecting facial feature point vectors and binocular center feature point vectors, performing trajectory prediction on these vectors based on a binocular parallax algorithm to obtain a visually predicted trajectory. Furthermore, it involves calibration-assisted optimization of the visually predicted trajectory based on the 3D spatial anchor frame, outputting a visually calibrated target, performing action semantic parsing on the visually calibrated target, and sending the action semantic parsing label to the robot's interactive terminal to trigger an interactive optimization command.

[0005] A second aspect of this application provides a vision-based robot human motion intention recognition and interaction system, the system comprising: Spatial Distribution Detection Module: Acquires spatial modeling data of a 3D scene, performs spatial distribution detection on the objects in the spatial modeling data to obtain the target spatial granularity, and constructs a 3D spatial anchor box for visual-assisted object recognition using the target spatial granularity; Trajectory Prediction Module: Collects facial feature point vectors and binocular center feature point vectors of a human body, performs trajectory prediction on the facial feature point vectors and the binocular center feature point vectors based on a binocular parallax algorithm to obtain a visually predicted trajectory; Action Semantic Parsing Module: Based on the 3D spatial anchor box, performs calibration-assisted optimization on the visually predicted trajectory, outputs a visual calibration target, performs action semantic parsing on the visually calibration target, and sends the action semantic parsing label to the robot interaction terminal to trigger an interaction optimization command.

[0006] One or more technical solutions provided in this application have at least the following technical effects or advantages: First, a 3D scene is spatially modeled. By analyzing the spatial distribution characteristics of objects in the scene, a target spatial granularity matching the actual environmental scale is determined, and a 3D spatial anchor frame is constructed accordingly to assist in visual object recognition. Next, facial feature point information and binocular visual center feature point information are collected. Using the principle of binocular parallax, the trajectory of the human's gaze or direction of attention is predicted, resulting in the predicted visual trajectory. Based on this, the constructed 3D spatial anchor frame is used to calibrate and optimize the predicted visual trajectory, determining the visually targeted object currently being focused on by the human. Semantic parsing of the corresponding human action is then performed, and finally, the parsed action intent label is sent to the robot's interaction terminal to drive the robot to execute corresponding optimized interaction commands, achieving precise interaction between the robot and the human. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a schematic diagram of the interaction method for vision-based robot human motion intention recognition provided in the embodiments of this application.

[0009] Figure 2 A schematic diagram of the structure of a vision-based robot human motion intention recognition interaction system provided in an embodiment of this application.

[0010] Figure labeling: Spatial distribution detection module 11, trajectory prediction module 12, action semantic parsing module 13. Detailed Implementation

[0011] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0012] Example 1, as Figure 1 As shown, this application provides a vision-based robot human motion intention recognition interaction method, the method including: Acquire spatial modeling data of a 3D scene, perform spatial distribution detection on the object targets in the spatial modeling data to obtain target spatial granularity, and use the target spatial granularity to construct a 3D spatial anchor box for visual-assisted recognition of object targets.

[0013] In this embodiment, data is first collected from the target working environment using a binocular stereo vision device or a 3D sensing device to obtain 3D scene spatial modeling data that reflects the real spatial structure of the scene. This spatial modeling data consists of 3D point cloud data or voxelized data containing object contours, positions, and depth information. Subsequently, the spatial modeling data is preprocessed, including denoising and coordinate unification, to ensure the accuracy of subsequent detection. Denoising can be performed through statistical filtering, radius filtering, or outlier removal based on neighborhood density. Coordinate unification can be achieved through coordinate system calibration and spatial coordinate transformation, converting data from different acquisition perspectives to a unified world coordinate system or robot base coordinate system. Afterward, the spatial distribution of each object target is detected and analyzed. Specifically, this includes calculating the object target size, the density of its center point coordinates in 3D space, and the spatial proximity relationships between objects. By statistically analyzing and comprehensively evaluating these multi-dimensional spatial features, a target spatial granularity that characterizes the scale of object distribution in the current scene is obtained. Then, based on this target spatial granularity, adaptively partitioned 3D spatial anchor frames are constructed within the boundary range of the 3D scene, ensuring that the constructed anchor frames match the physical scale and spatial density of the actual objects in terms of size and distribution. This provides a structured and executable 3D spatial prior for subsequent visual-assisted recognition and spatial association of objects, solving the problems of a large number of invalid anchor frames, low target matching accuracy, and wasted computational resources in large-scale or open and complex scenes due to fixed anchor frame scales, thus improving the accuracy and stability of target localization and visual recognition.

[0014] Furthermore, the spatial distribution detection of objects in the spatial modeling data is performed to obtain the spatial granularity of the targets. The method includes: The size features, center point coordinate density distribution features, and average nearest neighbor distance features of each object target are calculated based on the spatial modeling data, which is a spatial point cloud dataset acquired based on binocular stereo vision. The size features, center point coordinate density distribution features, and average nearest neighbor distance features are processed by feature vectorization to obtain a feature vector dispersion index. If the feature vector dispersion index is less than a preset dispersion threshold, the target spatial granularity is obtained by comprehensively averaging the feature vectorization results.

[0015] Preferably, the working environment is first synchronously acquired using a binocular stereo vision device, and a scene point cloud dataset containing spatial coordinate information is obtained through disparity calculation and 3D reconstruction. Subsequently, target segmentation processing is performed on this scene point cloud dataset. In this process, the scene point cloud dataset is preprocessed, including removing outlier noise points using statistical filtering or radius filtering, and the filtered point cloud is downsampled to reduce data redundancy and improve subsequent computational efficiency.

[0016] Next, based on the extrinsic parameters of the binocular stereo vision device, the point cloud is uniformly transformed to the same world coordinate system to ensure spatial consistency. Then, a point cloud clustering method based on spatial neighborhood is used for preliminary segmentation of the point cloud. For example, region growing or density clustering algorithms based on Euclidean distance are used. By setting a distance threshold between points and a minimum number of cluster points, spatially continuous and closely spaced points are clustered into the same candidate object region. For cases with planar backgrounds or structured scenes, a plane fitting algorithm can be used to remove large-scale background point clouds such as ground and walls. That is, normal vector estimation is performed on the point cloud data. By calculating the local neighborhood normal vector of each point, the local geometric characteristics of the point cloud are described. Then, a plane fitting algorithm based on random sampling consistency is used to randomly select several point sets in the point cloud data to construct candidate planar models. The distance from the remaining points to the plane is calculated according to the plane equation. Points with a distance less than a preset plane distance threshold are considered as points within the plane. Through multiple iterations, the planar model with the most inliers and satisfying the plane size constraints is selected as the main detected planar structure. For detected planar models, semantic discrimination is performed based on their normal vector direction and spatial location. For example, a plane whose normal vector is approximately perpendicular to the direction of gravity is identified as a ground plane, while a plane whose normal vector is approximately horizontal and has a large distribution range is identified as a wall or facade structure. After determining the plane type, interior points belonging to that plane are removed from the original point cloud data, resulting in the remaining point cloud after removing large-scale background structures. For scenes with multiple background planes, the above plane fitting and removal process can be repeated until the main background planes are removed, thereby effectively reducing the interference of ground, wall, and other background point clouds on subsequent object segmentation and spatial feature calculation. After removal, the remaining point cloud is subjected to the above clustering process to reduce the interference of the background on target segmentation.

[0017] After clustering is completed, the validity of each clustering result is judged. For example, abnormal clusters are filtered out based on preset constraints on the number of point clouds and volume size. Each valid cluster that meets the constraints is then regarded as an independent object target, and its corresponding point cloud set is extracted to form a subset of the point cloud of that object target, providing basic data support for subsequent size calculation and spatial feature analysis. Then, for each object target, its size features are calculated. These size features include the bounding box size or equivalent volume size of the object in the three-dimensional coordinate axis directions. That is, the maximum and minimum coordinate values ​​of all points in the point cloud subset of each object target in the three-dimensional coordinate system in the X, Y, and Z directions are counted respectively. The difference between the two values ​​is used to obtain the length of the object in each coordinate axis direction, thus forming the three-dimensional axis-aligned bounding box size of the object target. Then, based on the bounding box size, the equivalent volume size is calculated by multiplying the lengths in the X, Y, and Z directions, which is used to represent the spatial scale of the object target as a whole. Simultaneously, based on the center point coordinates of the object targets, the spatial distribution of each center point is statistically analyzed, and the density distribution characteristics of the center point coordinates are calculated. Specifically, using a point cloud centroid calculation method, the mean value of all point coordinates in the point cloud subset of each object target is calculated to obtain the corresponding object target's center point coordinates. Then, the center point coordinates of all object targets are mapped to a unified three-dimensional spatial coordinate system. By setting a fixed voxel size or spatial grid resolution, the space containing the center points is discretized, and the number of center points contained in each voxel or grid cell is counted. The variance index of the number of center points per unit volume or the center point distribution is then calculated as the density distribution characteristics of the center point coordinates, reflecting the degree of spatial concentration or dispersion of object targets in the scene. Further, the Euclidean distance between the center point of each object target and its neighboring object center points is calculated, and these are sorted in ascending order of distance. A preset number of nearest neighbor center points are selected, or neighbor center points with a distance smaller than a preset neighborhood radius are selected. Based on this, the Euclidean distances corresponding to the selected neighbor center points are summed and averaged to obtain the average nearest neighbor distance characteristic of the object target, reflecting the spatial interval relationship between object targets.

[0018] Then, the obtained size features, center point coordinate density distribution features, and average nearest neighbor distance features are normalized to their maximum and minimum values, and then concatenated in a preset order to form a feature vector of uniform dimension. Statistical analysis is then performed on this feature vector to calculate the statistical parameters of each feature dimension, including the mean and variance. Based on this, a weighted sum or mean of the variances of each feature dimension is calculated to obtain an overall variance index that comprehensively reflects the degree of fluctuation of each feature dimension. Next, the overall variance index is normalized to the corresponding feature mean. For example, using the coefficient of variation calculation method, the square root of the overall variance is divided by the feature mean to obtain a dimensionless dispersion value, which serves as the feature vector dispersion index. The smaller the feature vector dispersion index value, the more consistent the spatial scale and distribution characteristics of objects in the scene; conversely, the larger the dispersion index value, the greater the differences in spatial scale or distribution of objects, thus providing a quantitative basis for subsequent calculation and judgment of target spatial granularity. When the feature vector dispersion index is less than the preset dispersion threshold, it indicates that the spatial distribution scale of objects in the current scene is consistent. At this time, a comprehensive mean calculation is performed on the feature vectorization results. That is, the size features, center point coordinate density distribution features, and average nearest neighbor distance features of all objects are summed and the arithmetic mean is taken to obtain the mean result of each dimension feature. In order to make the contribution of different dimension features to spatial granularity controllable, preset weights can be set for each dimension feature according to actual application needs, and the mean values ​​of each dimension feature are weighted and fused to obtain a comprehensive scale parameter. This comprehensive scale parameter is then mapped to a single spatial scale value through normalization transformation, thereby converting the fused result into a spatial length scale under a unified physical meaning, and using it as the target spatial granularity to characterize the overall distribution scale of objects in the 3D scene, as the basic size unit when constructing the 3D spatial anchor frame.

[0019] Furthermore, if the feature vector dispersion index is greater than or equal to the preset dispersion threshold, the method includes: Adaptive clustering is performed on the feature vectorization processing results to obtain multiple clustering results; the mean of the feature vectorization processing results corresponding to each of the multiple clustering results is calculated to obtain multiple target space granularities.

[0020] Preferably, when the feature vector dispersion index is greater than or equal to a preset dispersion threshold, it indicates that there are significant differences in size and spatial distribution among different objects in the scene, and a single spatial granularity is insufficient to accurately represent the overall scene characteristics. In this case, adaptive clustering analysis is performed on the aforementioned feature vectorization results. Specifically, a distance- or density-based clustering algorithm is used to automatically determine the cluster centers and number of clusters based on the similarity between feature vectors. This classifies objects with similar size characteristics, center point distribution characteristics, and average nearest neighbor distance characteristics into the same cluster, resulting in multiple distinct clusters. Subsequently, for each cluster, the feature vectorization results corresponding to all objects within it are extracted, and the same statistical analysis is performed on each feature vector within the cluster to obtain target spatial granularities that represent the overall spatial distribution scale characteristics of the objects in that cluster. These target spatial granularities correspond one-to-one with each cluster result and are used to characterize the spatial distribution characteristics of objects at different scales in the scene, thus providing a basis for subsequently constructing multi-scale 3D spatial anchor frames.

[0021] Furthermore, the method for constructing a three-dimensional spatial anchor box for visual-assisted object recognition using the target spatial granularity includes: Based on the spatial modeling data of the three-dimensional scene, the scene boundary coordinates are set; based on the scene boundary coordinates, an initial three-dimensional spatial anchor frame is obtained in units of the target spatial granularity; based on the center point coordinate density distribution characteristics and the average nearest neighbor distance characteristics, the initial three-dimensional spatial anchor frame is overlap detected, and the grid step size of the initial three-dimensional spatial anchor frame is optimized according to the overlap rate, and the optimized three-dimensional spatial anchor frame is output.

[0022] Optionally, the spatial extent of the overall scene is first determined based on the spatial modeling data of the 3D scene. By statistically analyzing the maximum and minimum values ​​of point cloud data in the 3D coordinate system, the boundary coordinates of the scene are determined in the X, Y, and Z directions, thus forming a complete 3D scene enclosing region. Subsequently, using the scene boundary coordinates as constraints and the target spatial granularity as the basic unit of division, the space within the 3D scene enclosing region is divided according to a regular grid, generating an initial set of 3D spatial anchor frames covering the entire scene. The size of each initial 3D spatial anchor frame matches the target spatial granularity, used to initially characterize the potential object distribution area.

[0023] Next, combining the previously calculated density distribution characteristics of object target center point coordinates and the average nearest neighbor distance characteristics, spatial overlap detection is performed on the initial 3D spatial anchor frames. During this process, the coverage between the anchor frames and the object target center points is calculated. For each initial 3D spatial anchor frame, it is determined whether the center point coordinates of each object target in the scene fall within the spatial range of the anchor frame. If the center point is within the 3D boundary coordinates of the anchor frame, it is counted as a valid coverage. The number of object target center points covered by each anchor frame is counted, and combined with the theoretically required number or density of targets to be covered by the anchor frame, the target coverage of the anchor frame is obtained. Coverage can be expressed as the ratio of the number of target center points within the anchor frame to the average target density of the scene. Then, the spatial overlap ratio between adjacent anchor frames is calculated. For any two spatially adjacent or intersecting 3D anchor frames, the length of their overlap interval in the X, Y, and Z directions is calculated, and the overlap volume is obtained accordingly. This overlap volume is then normalized with the volume of one or both smaller anchor frames to obtain the overlap ratio between the anchor frames. This overlap ratio reflects the degree of spatial redundancy between the two anchor frames.

[0024] Then, considering both target coverage and the overlap ratio between anchor frames, the anchor frame overlap rate is calculated. This overlap rate is a weighted result of the overlap ratio between anchor frames and the target coverage. Generally, when the overlap ratio between anchor frames is high and the target coverage is low, the anchor frame is considered to have high redundancy; when the overlap ratio between anchor frames is low and the target coverage is high, the anchor frame is considered to have good space utilization efficiency. When excessive overlap or coverage redundancy is detected in some initial 3D space anchor frames, the grid step size in one or more coordinate directions is increased according to the density distribution characteristics of the target center points in the spatial region where the anchor frame is located. That is, the overlap rate is subtracted from the maximum value of the target overlap rate range, multiplied by a preset step size adjustment coefficient, and then the product and the sum of 1 are multiplied by the current grid step size to obtain the adjusted grid step size. Conversely, when insufficient anchor frame density is detected, the minimum value of the target overlap rate interval is subtracted from the overlap rate, and this is multiplied by a preset step size contraction coefficient. The difference between 1 and the calculated product is then multiplied by the current grid step size to obtain an adjusted grid step size. This appropriately reduces the grid step size in the corresponding coordinate direction, making the anchor frame division more refined and improving the coverage of the target object. Through this optimization process, the number of invalid or highly overlapping anchor frames can be reduced, ultimately outputting optimized 3D spatial anchor frames that match the scene's physical scale and object distribution characteristics. This solves the problem that existing 3D scenes using fixed-scale or uniform partitioning to construct spatial anchor frames easily generate a large number of invalid or highly overlapping anchor frames, leading to low spatial coverage efficiency, large computational redundancy, and difficulty in adapting to changes in the actual distribution scale of objects. This provides efficient and reliable spatial support for subsequent visual trajectory calibration and target recognition.

[0025] The facial feature point vectors and binocular center feature point vectors of the human body are collected. Based on the binocular disparity algorithm, the trajectory of the facial feature point vectors and the binocular center feature point vectors is predicted to obtain the visual predicted trajectory.

[0026] In one embodiment, a binocular vision device installed in a robot or interactive environment is used to acquire real-time images of the human body, simultaneously obtaining a sequence of facial images. Multiple facial feature points are extracted from these images based on a facial landmark detection algorithm, forming corresponding facial feature point vectors. This facial landmark detection algorithm can be a dlib-based model, capable of outputting 68 or 5 standard facial landmarks, including the positions of the eyes, nose tip, and corners of the mouth; or it can be a MediaPipe Face Mesh-based model, capable of outputting 468 high-precision facial landmarks. Simultaneously, the center feature point vectors of the left and right cameras of the binocular vision device at the current moment are acquired to represent the reference positions of the binocular imaging. Subsequently, under a unified timestamp condition, the facial feature point vectors and the binocular center feature point vectors are matched and corrected. Combining the disparity information between the left and right cameras, the coordinate positions of the facial landmarks in three-dimensional space are calculated based on a binocular disparity algorithm, thereby deducing the center position of the human eyeballs and their corresponding three-dimensional gaze direction vector. By performing time-series analysis and fusion processing on the three-dimensional gaze direction vectors obtained from consecutive frames, a spatial trajectory reflecting the changing trend of human gaze is formed. Based on the spatial trajectory, the direction of human attention is predicted, and finally, the human visual prediction trajectory is obtained. This trajectory is used to characterize the spatial area or target object that the human's intention to move may point to, providing a directly callable input for subsequent calibration and optimization docking with the three-dimensional spatial anchor frame.

[0027] Furthermore, based on a binocular parallax algorithm, trajectory prediction is performed on the facial feature point vector and the binocular center feature point vector to obtain a visual prediction trajectory. The method includes: The facial feature point vector and the binocular center feature point vector are matched synchronously in time to obtain a matching feature point vector group for each frame; the 3D eye center position of the matching feature point vector group is calculated; the 3D gaze direction vector is calculated according to the 3D eye center position, the 3D gaze direction vector is fused, and trajectory prediction is performed through the 3D gaze direction vectors of consecutive frames to obtain the visual prediction trajectory.

[0028] Preferably, the face image sequences acquired by the left and right cameras of the binocular vision device are first timestamped and synchronized to ensure that the face image data acquired by the left and right cameras at the same acquisition time are in a unified temporal sequence. Subsequently, based on the face key point detection algorithm, the corresponding face feature points are extracted from the synchronized images of the left and right cameras respectively to form face feature point vectors for the left and right views. Then, the camera center feature point vectors corresponding to the left and right cameras in the current frame are associated and marked with the face feature point vectors. According to the geometric constraint relationship and disparity consistency principle of the face feature points in the left and right views, the face feature points of the left and right views are matched one-to-one. In this process, the binocular vision device is calibrated offline or online to obtain the intrinsic parameter matrix, extrinsic parameter matrix and epipolar constraint relationship of the left and right cameras. Based on the calibration results, epipolar correction is performed on the left and right view images so that the projection points of the same spatial point in the left and right images are located on the same or approximately the same horizontal epipolar line, thereby simplifying the two-dimensional matching problem into a one-dimensional search problem. Next, key facial feature points of the same type and semantic meaning are extracted from the left and right view images, such as the corners of the eyes, the center of the eyelids, the tip of the nose, and the corners of the mouth. Each feature point is assigned a unique semantic label to ensure that matching is performed only between feature points of the same semantic type, avoiding mismatches across different body parts. For any facial feature point in the left view image, a set of candidate feature points is searched along its corresponding epipolar direction in the right view image. By calculating the horizontal disparity value of the left and right feature points in pixel coordinates, candidate feature points whose disparity falls within a preset reasonable range are selected. This reasonable range is determined by the baseline length of the binocular camera and the expected working distance between the human body and the camera. After obtaining the candidate feature points, matching verification is performed using the local descriptive information of the feature points. This local descriptive information includes the grayscale distribution, gradient direction, or texture pattern of the feature point's neighborhood. By calculating the similarity between the descriptors of the left and right feature points, the feature point with the highest similarity and satisfying the disparity consistency constraint is selected as the final matching result. Next, the matching results are checked for consistency. If the left-right matching relationship of the same facial feature point is stable in consecutive frames and the disparity change is continuous and smooth, the match is considered valid. Matching results that do not meet the consistency condition are discarded. By combining the above geometric constraints, semantic constraints and disparity consistency, accurate one-to-one matching of facial feature points from the left and right perspectives is achieved, forming a set of matching feature point vectors for each frame, providing a reliable foundation for subsequent 3D reconstruction and gaze direction calculation. Then, using the principle of binocular disparity, based on the difference in pixel coordinates of the matching feature points in the imaging planes of the left and right cameras, and combined with the camera's intrinsic and extrinsic parameters, 3D reconstruction is performed on the set of matching feature point vectors to calculate the coordinate position of the human eyeball center in 3D space in each frame, i.e., the 3D eyeball center position.Next, starting from the center position of the 3D eyeball, the head orientation parameters are obtained by combining the face pose estimation results. Based on the spatial relationship between the center position of the eyeball and the eyelid center or gaze reference point among the facial key points, the initial 3D gaze direction vector corresponding to the current frame is determined. Furthermore, the 3D gaze direction vectors calculated separately for the left and right eyes are fused. This fusion process can use weighted averaging or vector normalization to reduce the impact of monocular occlusion or noise on the gaze estimation results, resulting in a stable fused 3D gaze direction vector for the current frame.

[0029] After obtaining the fused 3D gaze direction vectors, the 3D gaze direction vectors obtained from consecutive frames are cached in chronological order, and this vector sequence is smoothed in the temporal domain to eliminate instantaneous jitter and measurement errors. Based on this, trend analysis is performed on the smoothed consecutive frame 3D gaze direction vectors; that is, the 3D gaze direction vectors calculated for each frame are converted into angular representations in spherical coordinates, including pitch and yaw angles, thus obtaining a gaze direction sequence. Before using this gaze direction sequence for analysis, a pre-built temporal prediction model based on an LSTM (Long Short-Term Memory) network is obtained. For this temporal prediction model, a fixed-length L (e.g., 20 frames) sliding window is first used to divide the historical visual trajectory to construct training samples. Each sample contains an angle sequence of L consecutive frames as input, followed by M frames (e.g., 5 frames) as the prediction target. Next, a temporal prediction model structure is designed, comprising an input layer, an LSTM layer, a fully connected layer, and an output layer. The input layer accepts a sequence of shape (L,2), where 2 represents the number of elements, including pitch and yaw angles. The LSTM layer consists of two stacked LSTM units, each with 128 hidden units, and Dropout=0.2 to prevent overfitting. The fully connected layer maps the LSTM output to a (M,2) predicted sequence. The output layer outputs the predicted sequence mapped by the fully connected layer. Subsequently, training samples are input into this model structure for forward propagation. During this process, the first LSTM layer sequentially reads the input from L time steps. At each time step t, it receives the current input and the hidden state and cell state from the previous time step t-1. It learns the weights of the input gate, forget gate, and output gate to determine which long-term historical information is no longer important and can be discarded from the cell state, which new information in the current input is important and needs to be stored in the cell state, and how to generate a hidden state based on the current input and cell state to pass to the next layer. After L time-step iterations, the first LSTM layer outputs a corresponding hidden state sequence rich in temporal information for each time step of the entire input sequence. Each hidden state is a 128-dimensional vector that encodes the gaze-move context information up to time t. The second LSTM layer receives the hidden state sequence output by the first layer as its input and undergoes similar iterations, but this time it processes more abstract, higher-order features. That is, this layer learns to construct more complex motion patterns on top of primary temporal features. For example, it may learn to recognize a pattern of rapid saccades followed by smooth tracking, or a pattern of wandering and searching in a certain area. The Dropout mechanism is activated in this layer, randomly shutting down a portion of neurons, forcing the network to learn more robust features that do not overly rely on individual neurons, thereby enhancing generalization ability.At the final time step L, the second LSTM layer outputs the final hidden state, a 128-dimensional vector that highly condenses and summarizes all the information from the entire input data, containing all the context needed to predict future gaze directions. This final hidden state is then fed into a fully connected layer. The fully connected layer learns a complex non-linear mapping function; its weights learn how to decode the 128-dimensional context vector representing past patterns into specific angle predictions for the next M time points, outputting a tensor of shape (M,2). These M two-dimensional vectors are the model's direct predictions of gaze directions for the next M frames. The model's predicted output is then compared point-by-point with the corresponding real gaze angle labels for the next M frames in the training samples. The average of the squared differences between the predicted and real angles is calculated to obtain the mean squared error loss value for the model's prediction on the current sample. Next, the loss value is calculated using the backpropagation algorithm, starting from the loss function and sequentially passing through the fully connected layer, the second LSTM layer, and the first LSTM layer. This process calculates each trainable parameter in the network, such as the gate weights of all LSTM units and the weights of the fully connected layers. The gradient of the current loss indicates how to make small adjustments to each parameter to reduce the loss. The Adam optimizer uses a learning rate (0.001) combined with estimates of the first and second moments (mean and uncentered variance) of the gradient to intelligently and adaptively update each parameter, enabling the model to make more accurate predictions when encountering similar inputs in the future. This training process is repeated until the maximum number of iterations is reached or the loss converges. After training, the model is validated using data not used for training to determine if the accuracy meets the preset accuracy. If it does, the current model is stored; otherwise, hyperparameters such as the learning rate and the number of training batches are adjusted.

[0030] After activating the temporal prediction model, it is used to propagate the gaze direction sequence forward, predicting its spatial extension direction and generating a spatial trajectory representing the change of human gaze over time. Finally, the obtained spatial trajectory is output as the visual prediction trajectory, used to characterize the spatial region or target object that the human may be focusing on now and in the short term, providing reliable input for subsequent target calibration and action intent recognition.

[0031] Furthermore, the method for calculating the 3D eye center position of the matching feature point vector group includes: A binocular vision device is used to acquire facial feature point vectors and binocular center feature point vectors of the human body; the intrinsic parameter matrix of the left camera, the intrinsic parameter matrix of the right camera, the distortion coefficient of the left camera, and the distortion coefficient of the right camera are entered into the binocular vision device; the 3D gaze direction vector is calculated according to the intrinsic parameter matrix of the left camera, the intrinsic parameter matrix of the right camera, the distortion coefficient of the left camera, the distortion coefficient of the right camera, and the center position of the 3D eyeball.

[0032] Optionally, a binocular vision device is first acquired to collect facial feature point vectors and binocular center feature point vectors. This device consists of two cameras, left and right, each with independent intrinsic parameters and distortion coefficients. These parameters are essential for accurate 3D reconstruction and calibration. To accurately process the acquired image data, the intrinsic parameter matrices of the left and right cameras, the distortion coefficients of the left and right cameras, and the relative position parameters (i.e., extrinsic parameters) between the cameras need to be pre-entered. This information typically includes camera focal length, principal point coordinates, and lens distortion coefficients. These parameters allow for precise image distortion correction and coordinate calculation in 3D space. Subsequently, based on the intrinsic parameter matrices and distortion coefficients of the left and right cameras, image distortion correction is performed to ensure that the acquired images maintain consistency with the positions of objects in the actual physical world. Then, based on a binocular disparity algorithm, the pixel position difference of the same feature points in the left and right images in the horizontal direction is calculated by comparing the feature points, obtaining the disparity value for each feature point. The larger the disparity value, the closer the point is to the camera. Next, using the known camera intrinsic parameters and disparity values, and employing the principle of triangulation, the depth information of each feature point is calculated by multiplying the camera's focal length by the baseline distance between the left and right cameras and then dividing by the disparity value. This yields the actual position of the feature point in 3D space. Then, based on the calculated 3D eye center position and the actual positions of the feature points in 3D space, a 3D gaze direction vector is calculated. This 3D gaze direction vector represents the orientation of the eyeball or the direction of the gaze, providing basic data for subsequent human action intention recognition.

[0033] Specifically, the visual prediction trajectory is calibrated and optimized based on the three-dimensional spatial anchor frame, a visual calibration target is output, the visual calibration target is subjected to action semantic analysis, the action semantic analysis label is sent to the robot interaction terminal, and an interaction optimization command is triggered.

[0034] In one embodiment, a series of candidate visual calibration targets are first generated based on the visually predicted trajectory. These candidate targets are then spatially calibrated using 3D spatial anchor frames to ensure the correct spatial relationship between the predicted trajectory and the object target. Through calibration optimization, the deviation between the trajectory and the object target can be corrected to obtain the visual calibration target that best matches the scene and physical environment. Subsequently, a score is calculated based on the matching degree between the calibration target and the trajectory. By calculating the matching degree between the candidate visual calibration targets and the trajectory, the visual calibration target with the highest score is selected as the final output visual calibration target. Next, the attribute information of the object is extracted from the visual calibration target, and combined with historical human action data, feature recognition is performed on the human action to infer the current action intention. By analyzing these action features, corresponding action semantic labels are generated, indicating the intention and target of the human action. This action semantic label is then transmitted to the robot interaction terminal. Based on the received action semantic label, the robot interaction terminal analyzes and triggers corresponding interaction optimization commands. For example, if a human gesture is recognized as indicating "approaching" an object, the robot may initiate corresponding actions, such as moving to the target location, preparing to grasp the object, or adjusting its posture to assist the human operation. Through this optimization mechanism, robots can accurately understand human interaction intentions in dynamic environments and respond instantly, thereby improving the efficiency and naturalness of human-computer interaction.

[0035] Furthermore, the method for calibrating and optimizing the visually predicted trajectory based on the three-dimensional spatial anchor frame and outputting a visual calibration target includes: Candidate visual calibration targets for the predicted visual trajectory are obtained, and the candidate visual calibration targets are calibrated using the three-dimensional spatial anchor frame; a matching degree score is performed on the candidate visual calibration targets, and the visual calibration targets are output based on the matching degree score.

[0036] Preferably, firstly, based on the previously calculated visual prediction trajectory, each frame in the trajectory is spatially mapped and projected onto a potential object region in the 3D scene, thereby generating a series of candidate targets. These candidate targets are then compared with the boundaries of 3D spatial anchor frames to determine the 3D spatial anchor frames to which the candidate targets belong. These 3D spatial anchor frames are then used to calibrate the candidate targets, forming candidate visual calibration targets. These candidate visual calibration targets may include object targets related to the human visual trajectory, such as objects in the trajectory path or objects close to the human's line of sight. Subsequently, based on the spatial relationship between the visual prediction trajectory and each candidate target, their matching degree is calculated. During this process, the Euclidean distance between the candidate visual calibration target and the center point of the corresponding 3D spatial anchor frame is calculated and converted into a spatial distance score using an inverse proportional function; the angle between the trajectory movement direction and the principal axis direction of the target object corresponding to the 3D spatial anchor frame is calculated and converted into a trajectory direction consistency score using an inverse proportional function; and the number of frames the trajectory remains within the 3D spatial anchor frame is counted and converted into a temporal stability score by dividing by a standard value. Next, the three scores are weighted and aggregated to generate a matching degree for each candidate visual calibration target. Then, the target with the highest matching degree is selected from all candidate visual calibration targets as the final visual calibration target. This visual calibration target is considered to be the target that best matches the human's current visual intent, representing the object or spatial area that the human is most likely to interact with, and is used for subsequent action recognition, semantic parsing, and interaction command generation.

[0037] Furthermore, the method for performing action semantic parsing on the visually labeled target includes: Extract the object attribute information of the visually calibrated target; identify historical human action features according to the object attribute information, perform action semantic parsing on the historical human action features, and obtain the action semantic parsing label of the visually calibrated target.

[0038] Preferably, the first step is to extract object attribute information from the acquired visually calibrated target. This object attribute information includes, but is not limited to, the target object's category (e.g., valve), operation type (e.g., rotation), geometric features, material information, and position. This information can be obtained from an object attribute database. Then, based on the extracted object attribute information, all historical human action features related to this object attribute information are extracted from historical action logs. These historical human action features refer to past human behavior patterns in similar situations, including gestures, direction of movement, and speed. Next, cosine similarity calculation is performed between the current human action features and these historical human action features to identify the most matching historical human action feature. The historical action semantics corresponding to this historical human action feature are then extracted and used as the parsing result of the current human action feature. This result is then structured into action semantic parsing labels to mark the visually calibrated target. In this way, the action semantics of the visually calibrated target can be accurately parsed based on historical human action features and current object attribute information, which can then be used to trigger subsequent interactive commands or reaction actions.

[0039] Furthermore, after obtaining multiple target space granularities, the method also includes: Multiple three-dimensional spatial anchor frames are constructed using the target spatial granularity for visual-assisted object recognition; after obtaining the visual prediction trajectory, the visual prediction trajectory is calibrated and optimized by switching the multiple three-dimensional spatial anchor frames.

[0040] Optionally, after obtaining multiple target spatial granularities, the same method described above is used to construct 3D spatial anchor frames for visual aided object recognition for each target spatial granularity. These 3D spatial anchor frames have different scales, sizes, and positions to adapt to the spatial layout and distribution characteristics of different objects in the scene. When the visually predicted trajectory is obtained, due to differences in the spatial distribution of object targets, the trajectory is calibrated and optimized by switching multiple 3D spatial anchor frames. That is, based on the position of the predicted trajectory, the 3D spatial anchor frame closest to the trajectory is selected, and the same calibration and optimization described above is performed based on this 3D spatial anchor frame. When the size or dimensions of the closest 3D spatial anchor frame change, the corresponding 3D spatial anchor frame is switched to ensure that each step in the calibration process adapts to changes in object distribution. Through the above process, it is ensured that the visually predicted trajectory is highly consistent with the actual object target area, achieving accurate association between human actions and target objects. This calibration-aided optimization not only improves the accuracy and efficiency of visual target recognition but also enables robots to better understand human intentions and make corresponding interactive responses in complex dynamic environments.

[0041] In summary, the embodiments of this application have at least the following technical effects: First, spatial modeling data of the 3D scene is acquired. Spatial distribution detection is performed on the objects in the spatial modeling data to obtain the target spatial granularity. A 3D spatial anchor frame for visually assisted object recognition is constructed using this target spatial granularity. Next, facial feature point vectors and binocular center feature point vectors are collected. Trajectory prediction is performed on these vectors based on a binocular parallax algorithm to obtain a visually predicted trajectory. Finally, the visually predicted trajectory is calibrated and optimized based on the 3D spatial anchor frame, outputting a visually calibrated target. Action semantic parsing is performed on the visually calibrated target, and the action semantic parsing label is sent to the robot's interactive terminal to trigger an interaction optimization command. This method solves the technical problems of low computational efficiency and ambiguous intent association caused by redundant anchor frames in complex physical scenes in traditional vision-based intent recognition methods. It achieves the technical effect of accurately associating human action intent with target objects by constructing a 3D spatial anchor frame that adaptively matches the physical scale of the scene and introducing a calibration and optimization mechanism for the visually predicted trajectory, thereby improving the accuracy, real-time performance, and reliability of intent recognition.

[0042] Example 2 is based on the same inventive concept as the vision-based robot human motion intention recognition interaction method in the foregoing examples, such as... Figure 2 As shown, this application provides a vision-based robot human motion intention recognition and interaction system, the system comprising: Spatial distribution detection module 11: acquires spatial modeling data of a 3D scene, performs spatial distribution detection on the objects in the spatial modeling data to obtain target spatial granularity, and constructs a 3D spatial anchor frame for visual aided recognition of the objects using the target spatial granularity; Trajectory prediction module 12: collects facial feature point vectors and binocular center feature point vectors of a human body, performs trajectory prediction on the facial feature point vectors and the binocular center feature point vectors based on a binocular parallax algorithm to obtain a visually predicted trajectory; Action semantic parsing module 13: performs calibration-assisted optimization on the visually predicted trajectory based on the 3D spatial anchor frame, outputs a visual calibration target, performs action semantic parsing on the visually calibration target, sends the action semantic parsing label to the robot interaction terminal, and triggers an interaction optimization command.

[0043] Furthermore, the spatial distribution detection module 11 is used to perform the following method: The size features, center point coordinate density distribution features, and average nearest neighbor distance features of each object target are calculated based on the spatial modeling data, which is a spatial point cloud dataset acquired based on binocular stereo vision. The size features, center point coordinate density distribution features, and average nearest neighbor distance features are processed by feature vectorization to obtain a feature vector dispersion index. If the feature vector dispersion index is less than a preset dispersion threshold, the target spatial granularity is obtained by comprehensively averaging the feature vectorization results.

[0044] Furthermore, the spatial distribution detection module 11 is used to perform the following method: Adaptive clustering is performed on the feature vectorization processing results to obtain multiple clustering results; the mean of the feature vectorization processing results corresponding to each of the multiple clustering results is calculated to obtain multiple target space granularities.

[0045] Furthermore, the spatial distribution detection module 11 is used to perform the following method: Multiple three-dimensional spatial anchor frames are constructed using the target spatial granularity for visual-assisted object recognition; after obtaining the visual prediction trajectory, the visual prediction trajectory is calibrated and optimized by switching the multiple three-dimensional spatial anchor frames.

[0046] Furthermore, the spatial distribution detection module 11 is used to perform the following method: Based on the spatial modeling data of the three-dimensional scene, the scene boundary coordinates are set; based on the scene boundary coordinates, an initial three-dimensional spatial anchor frame is obtained in units of the target spatial granularity; based on the center point coordinate density distribution characteristics and the average nearest neighbor distance characteristics, the initial three-dimensional spatial anchor frame is overlap detected, and the grid step size of the initial three-dimensional spatial anchor frame is optimized according to the overlap rate, and the optimized three-dimensional spatial anchor frame is output.

[0047] Furthermore, the trajectory prediction module 12 is used to perform the following method: The facial feature point vector and the binocular center feature point vector are matched synchronously in time to obtain a matching feature point vector group for each frame; the 3D eye center position of the matching feature point vector group is calculated; the 3D gaze direction vector is calculated according to the 3D eye center position, the 3D gaze direction vector is fused, and trajectory prediction is performed through the 3D gaze direction vectors of consecutive frames to obtain the visual prediction trajectory.

[0048] Furthermore, the trajectory prediction module 12 is used to perform the following method: A binocular vision device is used to acquire facial feature point vectors and binocular center feature point vectors of the human body; the intrinsic parameter matrix of the left camera, the intrinsic parameter matrix of the right camera, the distortion coefficient of the left camera, and the distortion coefficient of the right camera are entered into the binocular vision device; the 3D gaze direction vector is calculated according to the intrinsic parameter matrix of the left camera, the intrinsic parameter matrix of the right camera, the distortion coefficient of the left camera, the distortion coefficient of the right camera, and the center position of the 3D eyeball.

[0049] Furthermore, the action semantic parsing module 13 is used to perform the following methods: Candidate visual calibration targets for the predicted visual trajectory are obtained, and the candidate visual calibration targets are calibrated using the three-dimensional spatial anchor frame; a matching degree score is performed on the candidate visual calibration targets, and the visual calibration targets are output based on the matching degree score.

[0050] Furthermore, the action semantic parsing module 13 is used to perform the following methods: Extract the object attribute information of the visually calibrated target; identify historical human action features according to the object attribute information, perform action semantic parsing on the historical human action features, and obtain the action semantic parsing label of the visually calibrated target.

[0051] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A vision-based robot human motion intention recognition and interaction method, characterized in that, The method includes: Acquire spatial modeling data of a 3D scene, perform spatial distribution detection on the object targets in the spatial modeling data to obtain the target spatial granularity, and use the target spatial granularity to construct a 3D spatial anchor box for visual-assisted recognition of object targets; Collect human facial feature point vectors and binocular center feature point vectors, and perform trajectory prediction on the facial feature point vectors and binocular center feature point vectors based on a binocular disparity algorithm to obtain a visual prediction trajectory; Specifically, the visual prediction trajectory is calibrated and optimized based on the three-dimensional spatial anchor frame, a visual calibration target is output, the visual calibration target is subjected to action semantic analysis, the action semantic analysis label is sent to the robot interaction terminal, and an interaction optimization command is triggered.

2. The method as described in claim 1, characterized in that, The spatial distribution of objects in the spatial modeling data is detected to obtain the spatial granularity of the targets. The method includes: The size characteristics, center point coordinate density distribution characteristics, and average nearest neighbor distance characteristics of each object target are calculated based on the spatial modeling data. The spatial modeling data is a spatial point cloud dataset obtained based on binocular stereo vision. The size feature, center point coordinate density distribution feature, and average nearest neighbor distance feature are processed into feature vectors to obtain the feature vector dispersion index. If the feature vector dispersion index is less than the preset dispersion threshold, the target space granularity is obtained by calculating the comprehensive mean of the feature vectorization processing results.

3. The method as described in claim 2, characterized in that, If the feature vector dispersion index is greater than or equal to the preset dispersion threshold, the method includes: Adaptive clustering is performed on the feature vectorization results to obtain multiple clustering results; The mean of the feature vectorization processing result corresponding to each of the multiple clustering results is calculated to obtain multiple target space granularities.

4. The method as described in claim 3, characterized in that, After obtaining multiple target space granularities, the method also includes: Multiple three-dimensional spatial anchor boxes are constructed using the target spatial granularity for visual-assisted object recognition; After obtaining the visual prediction trajectory, the visual prediction trajectory is calibrated and optimized by switching the multiple three-dimensional spatial anchor frames.

5. The method as described in claim 2, characterized in that, The method for constructing a 3D spatial anchor box for visual-assisted object recognition using the target spatial granularity includes: Based on the spatial modeling data of the three-dimensional scene, set the scene boundary coordinates; Based on the scene boundary coordinates, an initial three-dimensional spatial anchor frame is obtained in units of the target spatial granularity. The initial 3D spatial anchor frame is overlap-detected based on the center point coordinate density distribution characteristics and the average nearest neighbor distance characteristics. The grid step size of the initial 3D spatial anchor frame is optimized according to the overlap rate, and the optimized 3D spatial anchor frame is output.

6. The method as described in claim 1, characterized in that, Based on a binocular parallax algorithm, trajectory prediction is performed on the facial feature point vector and the binocular center feature point vector to obtain a visually predicted trajectory. The method includes: The facial feature point vector and the binocular center feature point vector are matched synchronously in time to obtain the matching feature point vector group for each frame. Calculate the 3D eye center position of the matching feature point vector group; The 3D gaze direction vector is calculated based on the 3D eyeball center position, the 3D gaze direction vector is fused, and the trajectory is predicted by using the 3D gaze direction vectors of consecutive frames to obtain the visual prediction trajectory.

7. The method as described in claim 6, characterized in that, The method for calculating the 3D eye center position of the matching feature point vector group includes: A binocular vision device for acquiring facial feature point vectors and binocular center feature point vectors of the human body; The intrinsic parameter matrix of the left camera, the intrinsic parameter matrix of the right camera, the distortion coefficient of the left camera, and the distortion coefficient of the right camera of the binocular vision device are entered. The 3D gaze direction vector is calculated based on the intrinsic parameter matrix of the left camera, the intrinsic parameter matrix of the right camera, the distortion coefficient of the left camera, the distortion coefficient of the right camera, and the center position of the 3D eyeball.

8. The method as described in claim 1, characterized in that, The method includes: performing calibration-assisted optimization on the visually predicted trajectory based on the three-dimensional spatial anchor frame, and outputting a visual calibration target; Candidate visual calibration targets for the predicted visual trajectory are obtained, and the candidate visual calibration targets are calibrated using the three-dimensional spatial anchor frame; The matching degree is scored from the candidate visual calibration targets, and the visual calibration targets are output based on the matching degree score.

9. The method as described in claim 1, characterized in that, The method for performing action semantic parsing on the visually labeled target includes: Extract the object attribute information of the visually calibrated target; Based on the object attribute information, historical human action features are identified, and action semantic analysis is performed on the historical human action features to obtain the action semantic analysis label of the visually labeled target.

10. A vision-based robot human motion intention recognition and interaction system, characterized in that, The system is used to implement the vision-based robot human motion intention recognition and interaction method according to any one of claims 1-9, the system comprising: Spatial distribution detection module: acquires spatial modeling data of a 3D scene, performs spatial distribution detection on the objects in the spatial modeling data to obtain the spatial granularity of the objects, and uses the spatial granularity of the objects to construct a 3D spatial anchor box for visual-assisted recognition of objects. Trajectory prediction module: Collects facial feature point vectors and binocular center feature point vectors of the human body, and performs trajectory prediction on the facial feature point vectors and binocular center feature point vectors based on the binocular parallax algorithm to obtain the visual predicted trajectory; Action semantic parsing module: Based on the three-dimensional spatial anchor frame, the visual predicted trajectory is calibrated and optimized, a visual calibration target is output, action semantic parsing is performed on the visual calibration target, and the action semantic parsing label is sent to the robot interaction terminal to trigger an interaction optimization command.

Images