Man-machine interaction type target identification and positioning system based on multi-modal visual information and carrier thereof

Abstract

The invention relates to the field of robot perception and intelligent man-machine interaction, an equipment platform and a perception module, a system is provided with an Intel RealSense D435i depth camera which is used for acquiring an RGB-D image sequence containing depth information in real time, and meanwhile, an advanced YOLOv8-Pose deep learning model is embedded, so that the depth information of the RGB-D image sequence is acquired in real time. High-precision detection, natural interaction, a gesture guidance mechanism, direction inference and target candidate aggregation, interactive target screening and judgment, a robustness enhancement strategy, deep deletion and noise robustness and an incomplete target completion mechanism of human upper body skeleton key points in a front scene are realized; the man-machine collaborative target object identification and positioning method based on depth vision, attitude estimation and semantic understanding is mainly realized. The technology can be suitable for service robots, intelligent assistant robots and other practical application scenes with high requirements for autonomous perception and natural interaction, and man-machine co-fusion and intelligent environment construction are further promoted.

Description

Technical Field

[0001] The present invention relates to the field of robot perception and intelligent human-computer interaction, and in particular to a human-computer interactive target recognition and positioning system based on multimodal visual information and a carrier thereof. Background Art

[0002] Traditional object recognition and localization primarily utilizes features such as color, shape, and texture, relying on static image information from a single camera. This approach suffers from limitations such as weak differentiation and insufficient spatial positioning accuracy in situations with numerous similar objects, complex environments, and severe occlusion. Some methods have attempted to utilize depth cameras for three-dimensional positioning, but these methods still lack effective links with natural human interaction in terms of user intent identification. With breakthroughs in deep learning, significant progress has been made in human skeleton keypoint recognition technology, which can be used in a variety of scenarios, including motion analysis and gesture recognition. However, integrating this technology with the spatial recognition and localization of target objects to achieve higher-level semantic understanding and intent inference remains a pressing challenge.

[0003] Previous research has focused on object detection and single-user action segmentation, lacking a systematic approach that integrates depth, semantics, temporal, and spatial information. Especially when there are multiple, morphologically similar target objects, efficiently and accurately determining the pointed object by integrating the user's natural pointing gestures and outputting the object's 3D spatial position in real time to provide a basis for subsequent robot grasping path planning and autonomous operation remains a cutting-edge technical challenge in the field of vision-robotic interaction.

[0004] With the maturity of real-time target detection algorithms such as YOLO and the widespread application of RGB-D depth cameras (such as Intel RealSense D435i) in three-dimensional perception, combining multimodal perception information to achieve complex scene target perception for natural human-computer interaction has become a research hotspot and development trend for improving the key capabilities of intelligent robots.

[0005] Purpose of the Invention

[0006] The purpose of this invention is to propose a human-machine interactive target recognition and positioning system based on multimodal visual information and its carrier, so as to solve the technical problem of how a robot can achieve accurate and real-time three-dimensional positioning of a specific target object based on the natural indication of user gestures when facing objects of the same type and similar appearance. The adoption of this invention not only allows the robot to accurately identify similar objects in the scene that are completely or partially obscured, but also can efficiently map human intentions to object pose inference through the user's natural finger pointing movements, effectively improving the robot's autonomous perception and interactive performance in complex scenarios such as actual homes, offices, and public services, and providing a solid technical foundation for further promoting the construction of an intelligent human-machine integration system and the creation of a "people-oriented" intelligent environment. Summary of the Invention

[0007] The specific solution is as follows: A human-computer interactive target recognition and positioning system based on multimodal visual information, characterized by:

[0008] 1) Depth and color stream initialization and alignment;

[0009] This system uses a 640×480 resolution and a 30Hz frame rate, opening both the color and depth data channels to ensure data quality and timing synchronization during subsequent processing. Due to the actual hardware structure, there is a certain baseline offset between the depth camera and the color camera. That is, the images captured by the two sensors are not strictly aligned in space. To ensure that the depth value of each pixel can be accurately mapped to the color image coordinate system, the original depth map needs to be precisely aligned at the pixel level.

[0010] Using the intrinsic parameters (intrinsics) of the depth camera and color camera and the spatial extrinsics of the two, each pixel point (u, v) in the depth map is mapped to (u′, v′) in the color camera coordinate system through the transformation relationship of projection and back projection. The alignment process can be abstracted as shown in formula (1);

[0011] (u′,v′)=A(u,v,intrinsics,extrinsics)(1)

[0012] Here, A represents the spatial coordinate conversion function implemented based on specific camera calibration parameters. Through this conversion, each depth pixel can find the corresponding color image area, and achieve accurate data fusion for subsequent target detection and spatial restoration;

[0013] 2) YOLOv8 Pose key point detection;

[0014] The present invention uses the YOLOv8 Pose model to detect key points of the human body in the input image. The system can automatically output the position of each 25 key points of the human body and their corresponding confidence (conf), which are recorded as (x, y, conf). In order to ensure the accuracy of the detection results, the confidence threshold T needs to be pre-set. c =0.25, use formula (2) to make judgments, retain key points with confidence higher than the threshold, and eliminate the error caused by low confidence;

[0015]

[0016] For each key point that passes the confidence screening, more robust spatial depth information is obtained. With the key point (u, v) as the center, multiple points of the surrounding neighborhood are sampled in combination with the depth map. Median filtering is used to eliminate mutations or noise, and the truncated mean is used: only the mean of the middle M points after sorting is taken as the final depth, which minimizes the interference of abnormal areas such as occlusion and reflection on the results. For the key point center (x0, y0), N neighboring points D are sampled. i =depth(x0+δx i ,y0+δy i ), take the median value D of the sampling set f =Median({D i}), and then average the middle segment after sorting to get the robust depth D * , then, using the given camera intrinsic parameters (focal length f x , f y and main point c x , c y ), use the camera back-projection formula (3) to restore all key points to the three-dimensional space coordinates (X, Y, Z) under the camera perspective;

[0017]

[0018] The three-dimensional key points are further processed using spatial vectors and geometric transformations. Assuming that the left shoulder point P1 = (X1, Y1, Z1) and the left wrist point P2 = (X2, Y2, Z2), the spatial vector of the left arm V = P2-P1. The unit vector is obtained by normalization using formula (4), and then the angle with each axis is solved separately. The x-axis angle θ is calculated as formula (5). x , similarly, we can find θ y ,θ z , used to analyze changes in limb posture;

[0019]

[0020] Object 3D position detection and edge defect completion system;

[0021] The data collection part of this module is similar to the human key point detection module. At the same time, the YOLOv5 target detection model is used to identify objects in the self-built data and detect their three-dimensional positions.

[0022] The harmonic mean score F1 of the model's precision and recall shows that when the confidence level is 0.9, the model performs best when deployed for detection.

[0023] However, in real-world scenarios, objects are often found at the image boundary. In such cases, it is difficult to determine the object's accurate three-dimensional position solely by relying on the above methods.

[0024] For these objects with incomplete edges, the present invention adds an edge incomplete judgment module. If the detected object detection frame area deviates significantly from the actual scene expectation or the detection frame boundary is close to the image edge, the edge completion logic is automatically activated. Specifically, it includes two parts:

[0025] 3) Determine whether an object is complete or missing

[0026] The system prioritizes whether the detection frame is close to the original image boundary to quickly determine whether the target is complete. The judgment rules are as follows: Let the detection frame coordinates be (x1, y1, x2, y2), the image width and height be W and H, and the boundary tolerance threshold be ε. The judgment condition is shown in formula (6):

[0027]

[0028] When the upper left corner and the lower right corner of the detection frame do not exceed the set boundary threshold ε, the target is considered complete; otherwise, it is considered an edge-incomplete object and needs to be corrected.

[0029] 4) 3D position correction of objects with incomplete edges;

[0030] When the upper left corner of the target detection frame is detected to be out of bounds (i.e., x1 < ε, y1 < ε), it indicates that part of the object is located at the edge of the image. At this time, the true three-dimensional position cannot be directly obtained. Therefore, the system uses the known three-dimensional information of the diagonal corner of the detection frame and formula (7) to infer the missing position through spatial reverse calculation.

[0031] (X0,Y0)=(X right_down -Δ x ,Y right_down -Δ y )(7)

[0032] Among them, (X right_down ,Y right_down ) is the three-dimensional coordinate obtained by reverse projection of the lower right corner of the detection frame, (Δ x ,Δ y ) is a disturbance term set based on experience such as object category and detection frame size, which is used to correct the deviation. When the four corner points of the target cross the boundary, their diagonal space points and disturbance values ​​are different. Each combination uses a similar completion method, as shown in formula (8);

[0033] (X,Y) suppl =f(border,opposite corner 3d,Δ x ,Δ y )(8)

[0034] The completion mechanism will spatially map the effective depth within a known area, and based on information such as object category and typical shape, infer the approximate outer corners and sizes of the occluded or missing parts, thereby reconstructing and completing the complete three-dimensional volume information and posture of the object, greatly improving the system's applicability and robustness for target recognition and spatial reconstruction in complex real-world environments.

[0035] A human-computer interactive carrier based on multimodal visual information, characterized by:

[0036] The experimental hardware structure mainly consists of Jetson ORIN NANO combined with S300Mini car, Dorbot CR5 robotic arm and RealSense D435i depth camera. Among them, Jetson ORIN NANO is used as the development board of the present invention to complete the upper-level control of the detection task. The S300Mini car and Dorbot CR5 robotic arm serve as the chassis structure and grasping structure of the present invention respectively to complete the actual test task of the invention performance. The experimental dataset is a self-built sample dataset, which contains 3,000 pictures of actual application scenarios. Labeling tools are used for annotation. 80% of the pictures are used for model training and 20% for model testing.

[0037] The overall framework of this system is mainly divided into two core modules that work together:

[0038] The first is the human body 3D key point detection and 3D pose estimation module based on the YOLOv8 Pose algorithm. This module is primarily responsible for accurately locating key points of the human body in the input video image stream, restoring the coordinates of each point in 3D space based on depth information, and further analyzing and inferring the pose information of various parts of the human body.

[0039] The second is the three-dimensional position detection and boundary completion module for multiple categories of objects based on the YOLOv5 model. This part can not only identify and locate the spatial positions of various common target objects in the scene, but also designs a boundary completion algorithm to achieve complete restoration of the object's spatial information when spatial information is lost due to occlusion or proximity to the image boundary. The two core functional modules share many basic common functions in the system, including the synchronous acquisition and alignment of depth maps and color images, coordinate transformation from two-dimensional pixels to three-dimensional space, cyclic parallel processing using multi-threading mechanism, and post-processing processes customized for special scene requirements.

[0040] Compared with the prior art, the technical effects and advantages of the present invention are:

[0041] (1) Natural and efficient human-computer interaction

[0042] Users only need to point directly at the target object with their arms, without the need for tedious input, multi-step selection or voice guidance, achieving intuitive and cognitively habitual target interactive designation, greatly improving the efficiency and ease of use of human-computer collaboration.

[0043] (2) Zero-ambiguity designation of similar targets

[0044] The present invention uses the relative spatial relationship between three-dimensional skeleton lines and target objects to distinguish and screen designated targets within completely similar objects. It can eliminate user pointing ambiguity with high accuracy and improve the robustness of target screening in multi-target scenarios.

[0045] (3) Accurate and robust three-dimensional positioning capability

[0046] The present invention adopts a method of random sampling combined with median filtering aggregation in the depth information collection process, which can effectively suppress the influence of depth abnormal data; at the same time, the designed corner-based three-dimensional completion strategy can further ensure a more accurate estimation of the target three-dimensional center point under conditions such as edges and partial occlusions.

[0047] (4) Adapt to dynamic and complex environments

[0048] The present invention is highly generalizable. Through multimodal information fusion and spatial geometric reasoning, the system can adapt to complex real-world environments such as various targets, lighting, obstacles, etc., significantly expanding the usability of robots in complex real-world scenarios such as homes and offices. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is the distribution of the number of targets in each image in the training dataset;

[0050] Figure 2 is the co-occurrence relationship between different target categories;

[0051] Figure 3 It is the overall design process of the system;

[0052] Figure 4 It is the algorithm process of human body 3D key point detection and angle estimation;

[0053] Figure 5 It is the object three-dimensional position detection and edge completion algorithm process;

[0054] Figure 6 It is a graph showing the F1 score of the model as the confidence threshold changes;

[0055] Figure 7 is the precision under different confidence thresholds;

[0056] Figure 8 It is the recall curve under different confidence levels;

[0057] Figure 9 It is a performance result diagram of each stage of model training. DETAILED DESCRIPTION

[0058] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0059] The present invention relates to the following process:

[0060] (1) Device platform and perception module

[0061] The system is equipped with an Intel RealSense D435i depth camera for real-time acquisition of RGB-D image sequences containing depth information. It also embeds an advanced YOLOv8-Pose deep learning model to achieve high-precision detection of key points of the human upper body skeleton in the scene ahead.

[0062] (2) Natural interaction: gesture guidance mechanism

[0063] When a user stands in front of the robot's field of view and gestures toward a target object (such as a water cup), the system automatically extracts the 2D positions of key skeletal points, such as the wrist and forearm, in image coordinates. Combined with the D435i depth image, the system simultaneously maps and reconstructs the 3D spatial coordinates of these key points.

[0064] (3) Direction inference and target candidate aggregation

[0065] By connecting the three-dimensional coordinates of key points such as the wrist and elbow, the spatial vector of the user's arm guidance direction is obtained. Using YOLOv5 and depth information, the three-dimensional positions of all water cups in the scene are detected and clustered in real time, generating a list of the spatial positions of all similar targets.

[0066] (4) Interactive target screening and judgment

[0067] The spatial distance between the 3D center point of each candidate cup and the line in the finger's direction is calculated, and the cup closest to it is determined to be the target cup currently designated by the user. The 3D coordinate output is used to guide subsequent robotic arm grasping or related operations.

[0068] (5) Robustness enhancement strategy

[0069] 1) Depth loss and noise robustness: To address the situation where depth cameras may experience depth holes or errors in complex lighting, reflective, or edge areas, a random sampling method based on candidate object data points combined with a median filter is proposed to effectively filter out outliers and improve the stability and accuracy of the 3D position estimation of the object center point.

[0070] 2) Incomplete object completion mechanism: When YOLOv5 is unable to recognize the complete object outline and center point due to factors such as occlusion or edges, a geometric inference completion strategy based on corner points (such as detected boundary points of the object) is introduced. Through multi-angle interpolation and model fitting, robust restoration of the 3D center of missed or partially visible objects is achieved.

[0071] This paper proposes a method for detecting 3D key points of the human body and identifying the spatial position of objects based on a depth camera and the YOLOv8-pose object detection algorithm. This method can detect the 3D coordinates of key points of the human body in real time, estimate body posture, and perform 3D coordinate recovery and boundary completion of objects. To achieve high-precision and robust object detection and 3D information acquisition, the system design incorporates multi-module parallelism, data preprocessing, regional deep sampling and filtering, 3D spatial transformation, multi-source anomaly detection, and spatial completion techniques. This method is suitable for human-computer interaction, behavior analysis, and on-site item management in complex scenarios.

[0072] The experimental hardware structure mainly consists of Jetson ORIN NANO combined with S300Mini car, Dorbot CR5 robot arm and RealSense D435i depth camera. Among them, Jetson ORIN NANO is used as the development board of the present invention to complete the upper control of the detection task, and the S300Mini car and Dorbot CR5 robot arm are used as the chassis structure and grasping structure of the present invention respectively to complete the actual test task of the invention performance. The experimental data set is a self-built sample data set, which contains 3,000 pictures of actual application scenarios. The number and situation of targets in the data set are as follows: Figure 1 and Figure 2 As shown in the figure, the Labeling tool is used for annotation, 80% is used for model training and 20% is used for model testing.

[0073] The overall framework of this system is as follows Figure 3The entire framework consists of two core modules working together: the first is a 3D human keypoint detection and 3D pose estimation module based on the YOLOv8 Pose algorithm. This module is responsible for accurately locating human keypoints in the input video image stream and, combined with depth information, restoring the coordinates of each point in 3D space. It also further analyzes and infers the pose information of various human body parts. The second is a 3D position detection and boundary completion module for multiple categories of objects based on the YOLOv5 model. This module not only identifies and locates the spatial positions of various common target objects in the scene, but also implements a boundary completion algorithm to fully restore the spatial information of objects when spatial information is lost due to occlusion or proximity to image boundaries. These two core modules share many fundamental common functions within the system, including the simultaneous acquisition and alignment of depth maps and color images, 2D pixel-to-3D coordinate transformation, multi-threaded parallel processing, and customized post-processing for specific scenarios. Overall, the system's components, supported by tight process collaboration, achieve efficient and accurate extraction and real-time completion of spatial information about humans and objects, meeting the multi-dimensional visual perception requirements of complex application scenarios.

[0074] The specific implementation process of the present invention is:

[0075] (1) Human body key point detection and 3D posture estimation system

[0076] 1) Depth and color stream initialization and alignment

[0077] This system uses a 640×480 resolution and a 30Hz frame rate, enabling both the color and depth data channels to ensure data quality and timing synchronization during subsequent processing. Due to the actual hardware structure, there is a certain baseline offset between the depth camera and the color camera. This means that the images captured by the two sensors are not strictly aligned spatially. To ensure that the depth value of each pixel is accurately mapped to the color image coordinate system, precise pixel-level alignment of the original depth map is required.

[0078] Using the intrinsic parameters of the depth camera and color camera and their spatial extrinsics, each pixel (u, v) in the depth map is mapped to the color camera coordinate system (u′, v′) through projection and back-projection transformations. The alignment process can be abstracted as shown in formula (1).

[0079] (u′,v′)=A(u,v,intrinsics,extrinsics)(1)

[0080] Here, A represents the spatial coordinate conversion function implemented based on specific camera calibration parameters. Through this conversion, each depth pixel can be found in the corresponding color image region, enabling accurate data fusion for subsequent object detection and spatial restoration.

[0081] 2) YOLOv8 Pose key point detection

[0082] This paper uses the YOLOv8 Pose model to detect key points of the human body in the input image. The module implementation block diagram is as follows: Figure 4 As shown. This system can automatically output the position of each of the 25 key points of the human body and its corresponding confidence (conf), and record it as (x, y, conf). In order to ensure the accuracy of the detection results, the confidence threshold T needs to be set in advance. c The confidence level is 0.25, and the formula (2) is used for judgment. The key points with confidence levels higher than the threshold are retained, and the error caused by low confidence levels is eliminated.

[0083]

[0084] For each key point that passes the confidence filter, more robust spatial depth information is obtained. With the key point (u, v) as the center, multiple points of the surrounding neighborhood are sampled in combination with the depth map. Median filtering is used to eliminate mutations or noise, and a truncated mean is used: only the mean of the middle M points after sorting is taken as the final depth, which minimizes the interference of abnormal areas such as occlusion and reflection on the results. For the key point center (x0, y0), N neighboring points D are sampled. i =depth(x0+δx i ,y0+δy i ), take the median value D of the sampling set f =Median({D i}), and then average the middle segment after sorting to get the robust depth D * Then, using the given camera internal parameters (focal length f x , f y and main point c x , c y ), the filtered two-dimensional pixel coordinates (u, v) and depth d are restored to the three-dimensional space coordinates (X, Y, Z) under the camera perspective using the camera back projection formula (3).

[0085]

[0086] The three-dimensional key points are further processed using spatial vectors and geometric transformations. Assuming that the left shoulder point P1 = (X1, Y1, Z1) and the left wrist point P2 = (X2, Y2, Z2), the spatial vector of the left arm V = P2-P1, and the unit vector is obtained by normalization using formula (4). The angles with each axis are then solved, as shown in formula (5) to calculate the x-axis angle θ x , similarly, we can find θ y ,θ z , used to analyze changes in limb posture.

[0087]

[0088] (2) Object 3D position detection and edge defect completion system

[0089] The data collection part of this module is similar to the human key point detection module. At the same time, the YOLOv5 target detection model is used to identify objects in the self-built data and detect their three-dimensional positions. The harmonic mean score F1 of the model's precision and recall rate changes with different confidence thresholds as shown in the following curve: Figure 5 As shown in , it can be seen that when the confidence level is 0.9, the model is best deployed for detection. In addition, other related curves obtained after model training can be obtained by Figure 6 、 Figure 7 and Figure 8 As shown in the figure, they collectively represent the performance of each stage of model training. It can be seen that the model detection accuracy of this system is extremely high. When the object is completely within the detection frame, the system can obtain a relatively accurate three-dimensional position using the above method.

[0090] However, in actual scenes, we often encounter situations where objects are at the edge of the image. At this time, it is difficult to further determine the accurate three-dimensional spatial position of the object by relying solely on the above method. For these objects with incomplete edges, the present invention adds an edge incomplete judgment module. If the area of ​​the object detection frame is detected to have a large deviation from the actual scene expectation or the boundary of the detection frame is close to the edge of the image, the edge completion logic is automatically started, and the algorithm implementation framework is as follows: Figure 9 As shown, it specifically includes two parts:

[0091] 5) Determine whether an object is complete or missing

[0092] The system prioritizes whether the detection frame is close to the original image boundary to quickly determine whether the target is complete. The judgment rule is as follows: let the detection frame coordinates be (x1, y1, x2, y2), the image width and height be W and H, and the boundary tolerance threshold be ε, then the judgment condition is shown in formula (6):

[0093]

[0094] When the upper left corner and the lower right corner of the detection frame do not exceed the set boundary threshold ε, the target is considered complete; otherwise, it is considered an object with incomplete edges and needs to be completed and corrected.

[0095] 6) 3D position correction of objects with incomplete edges

[0096] When the upper left corner of the target detection frame is detected as out of bounds (i.e., x1 < ε, y1 < ε), it indicates that part of the object is located at the edge of the image; in this case, the true 3D position cannot be directly obtained. Therefore, the system uses the known 3D information of the opposite corner of the detection frame and formula (7) to infer the missing position through spatial reverse calculation.

[0097] (X0,Y0)=(X right_down -Δ x ,Y right_down -Δ y ) (7)

[0098] Among them, (X right_down ,Y right_down ) is the three-dimensional coordinate obtained by reverse projection of the lower right corner of the detection frame, (Δ x ,Δ y ) is a perturbation term set empirically based on object category and detection box size, used to correct deviations. When the four corner points of the target cross the boundary, their diagonal spatial points and perturbation values ​​are different. Each combination uses a similar completion method, as shown in formula (8).

[0099] (X,Y) suppl =f(border,opposite corner 3d,Δ x ,Δ y )(8)

[0100] The completion mechanism spatially maps the effective depth within a known area and, based on information such as object category and typical morphology, infers the approximate outer edges and dimensions of the occluded or missing portion, thereby reconstructing and completing the object's complete 3D volume and pose. This significantly improves the system's applicability and robustness for target recognition and spatial reconstruction in complex real-world environments.

[0101] The above are only preferred embodiments of the present invention, which are used to illustrate the design and technical principles of the present invention and do not constitute a limitation on the scope of the present invention. Although the present invention has been described in detail in conjunction with the above embodiments, it is clear to those skilled in the art that any modification of the technical solution or any equivalent replacement of the technical features without departing from the concept and spirit of the present invention shall fall within the scope of protection of the present invention.

Claims

1. A human-computer interactive target recognition and positioning system based on multimodal visual information, characterized by: (1) Depth and color stream initialization and alignment; This system uses a 640×480 resolution and a 30Hz frame rate, opening both the color and depth data channels to ensure data quality and timing synchronization during subsequent processing. Due to the actual hardware structure, there is a certain baseline offset between the depth camera and the color camera. That is, the images captured by the two sensors are not strictly aligned in space. To ensure that the depth value of each pixel can be accurately mapped to the color image coordinate system, the original depth map needs to be precisely aligned at the pixel level. Using the intrinsic parameters (intrinsics) of the depth camera and color camera and the spatial extrinsics of the two, each pixel point (u, v) in the depth map is mapped to (u′, v′) in the color camera coordinate system through the transformation relationship of projection and back projection. The alignment process can be abstracted as shown in formula (1); (u′,v′)=A(u,v,intrinsics,extrinsics)(1) Here, A represents the spatial coordinate conversion function implemented based on specific camera calibration parameters. Through this conversion, each depth pixel can find the corresponding color image area, and achieve accurate data fusion for subsequent target detection and spatial restoration; (2) YOLOv8 Pose key point detection; The present invention uses the YOLOv8 Pose model to detect key points of the human body in the input image. The system can automatically output the position of each 25 key points of the human body and their corresponding confidence (conf), which are recorded as (x, y, conf). In order to ensure the accuracy of the detection results, the confidence threshold T needs to be pre-set. c =0.25, use formula (2) to make judgments, retain key points with confidence higher than the threshold, and eliminate the error caused by low confidence; For each key point that passes the confidence screening, more robust spatial depth information is obtained. With the key point (u, v) as the center, multiple points of the surrounding neighborhood are sampled in combination with the depth map. Median filtering is used to eliminate mutations or noise, and the truncated mean is used: only the mean of the middle M points after sorting is taken as the final depth, which minimizes the interference of abnormal areas such as occlusion and reflection on the results. For the key point center (x0, y0), N neighboring points D are sampled. i =depth(x0+δx i ,y0+δy i ), take the median value D of the sampling set f =Median({D i }), and then average the middle segment after sorting to get the robust depth D * , then, using the given camera intrinsic parameters (focal length f x , f y and main point c x , c y ), use the camera back-projection formula (3) to restore all key points to the three-dimensional space coordinates (X, Y, Z) under the camera perspective; The three-dimensional key points are further processed using spatial vectors and geometric transformations. Assuming that the left shoulder point P1 = (X1, Y1, Z1) and the left wrist point P2 = (X2, Y2, Z2), the spatial vector of the left arm V = P2-P1. The unit vector is obtained by normalization using formula (4), and then the angle with each axis is solved separately. The x-axis angle θ is calculated as formula (5). x , similarly, we can find θ y ,θ z , used to analyze changes in limb posture; (3) Object three-dimensional position detection and edge defect completion system; The data collection part of this module is similar to the human key point detection module. At the same time, the YOLOv5 target detection model is used to identify objects in the self-built data and detect their three-dimensional positions. The harmonic mean score F1 of the model's precision and recall shows that when the confidence level is 0.9, the model performs best when deployed for detection. However, in real-world scenarios, objects are often found at the image boundary. In such cases, it is difficult to determine the object's accurate three-dimensional position solely by relying on the above methods. For these objects with incomplete edges, the present invention adds an edge incomplete judgment module. If the detected object detection frame area deviates significantly from the actual scene expectation or the detection frame boundary is close to the image edge, the edge completion logic is automatically activated. Specifically, it includes two parts: 1) Determine whether an object is complete or missing The system prioritizes whether the detection frame is close to the original image boundary to quickly determine whether the target is complete. The judgment rules are as follows: Let the detection frame coordinates be (x1, y1, x2, y2), the image width and height be W and H, and the boundary tolerance threshold be ε. The judgment condition is shown in formula (6): When the upper left corner and the lower right corner of the detection frame do not exceed the set boundary threshold ε, the target is considered complete; otherwise, it is considered an edge-incomplete object and needs to be corrected. 2) 3D position correction of objects with incomplete edges; When the upper left corner of the target detection frame is detected to be out of bounds (i.e., x1 < ε, y1 < ε), it indicates that part of the object is located at the edge of the image. At this time, the true three-dimensional position cannot be directly obtained. Therefore, the system uses the known three-dimensional information of the diagonal corner of the detection frame and formula (7) to infer the missing position through spatial reverse calculation. (X0,Y0)=(X right-down -D x ,Y right-down -D y ) (7) Among them, (X right-down ,Y right-down ) is the three-dimensional coordinate obtained by reverse projection of the lower right corner of the detection frame, (Δ x ,Δ y ) is a disturbance term set based on experience such as object category and detection frame size, which is used to correct the deviation. When the four corner points of the target cross the boundary, their diagonal space points and disturbance values ​​are different. Each combination uses a similar completion method, as shown in formula (8); (X,Y) suppl =f(border,opposite corner 3d,Δ x ,Δ y ) (8) The completion mechanism will spatially map the effective depth within a known area, and based on information such as object category and typical shape, infer the approximate outer corners and sizes of the occluded or missing parts, thereby reconstructing and completing the complete three-dimensional volume information and posture of the object, greatly improving the system's applicability and robustness for target recognition and spatial reconstruction in complex real-world environments.

2. The human-computer interactive carrier based on multimodal visual information according to claim 1, characterized in that: The experimental hardware structure mainly consists of Jetson ORIN NANO combined with S300Mini car, Dorbot CR5 robotic arm and RealSense D435i depth camera. Among them, Jetson ORIN NANO is used as the development board of the present invention to complete the upper-level control of the detection task. The S300Mini car and Dorbot CR5 robotic arm serve as the chassis structure and grasping structure of the present invention respectively to complete the actual test task of the invention performance. The experimental dataset is a self-built sample dataset, which contains 3,000 pictures of actual application scenarios. Labeling tools are used for annotation. 80% of the pictures are used for model training and 20% for model testing. The overall framework of this system is mainly divided into two core modules that work together: The first is the human body 3D key point detection and 3D pose estimation module based on the YOLOv8 Pose algorithm. This module is primarily responsible for accurately locating key points of the human body in the input video image stream, restoring the coordinates of each point in 3D space based on depth information, and further analyzing and inferring the pose information of various parts of the human body. The second is the three-dimensional position detection and boundary completion module for multiple categories of objects based on the YOLOv5 model. This part can not only identify and locate the spatial positions of various common target objects in the scene, but also designs a boundary completion algorithm to achieve complete restoration of the object's spatial information when spatial information is lost due to occlusion or proximity to the image boundary. The two core functional modules share many basic common functions in the system, including the synchronous acquisition and alignment of depth maps and color images, coordinate transformation from two-dimensional pixels to three-dimensional space, cyclic parallel processing using multi-threading mechanism, and post-processing processes customized for special scene requirements.

Images