Hand-held article detection method for space station cargo hold goods

Abstract

The invention relates to a handheld article detection method for space station cargo hold goods, which comprises the following steps of: identifying a grabbing gesture in a shot image to obtain a grabbing gesture identification result; generating a corresponding ROI (Region of Interest) according to a mapping relation between the grasp gesture and the ROI based on the grasp gesture recognition result; and taking the RO I as the input of handheld article detection so as to detect a target article in the shot image. In order to improve the article detection accuracy of article detection under a space task, an efficient gesture prior information extraction and classification module is fused on the basis of a YOLOv8 architecture, background interference can be effectively eliminated in a complex space cabin internal environment by dynamically adjusting RO I, handheld articles can be accurately recognized, and the object detection accuracy is improved. Therefore, the real-time performance and the accuracy of article management are improved, and a test result shows that the target article detection speed is improved by 19% compared with that of YOLOv8 reference model detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of space technology, and relates to a handheld object detection method for space station cargo bay cargo, and particularly relates to a handheld object detection method MH-YOLOv8 for space station cargo bay cargo by fusing MediaPipe Hands and YOLOv8. BACKGROUND

[0002] With the rapid development of space technology, the uplink cargo capacity of cargo spacecraft is continuously improved, and the scale and complexity of space missions are also increasing. From the successful implementation of about 5.6 tons of supplies uplink of Tianzhou 7 to the increase of more than 200 liters and more than 100 kilograms of cargo capacity on the basis of Tianzhou 8. This progress shows that in the future, cargo spacecraft will have more heavy and complex cargo management and resource scheduling. The traditional manual cargo management mode based on manual recording gradually exposes a series of disadvantages, such as high work intensity, risk of manual operation error, low resource scheduling efficiency and the like. In recent years, with the rapid development of computer vision and artificial intelligence (AI) technology in the field of space, a new idea is provided for object detection, tracking and management in space missions. In space missions, astronauts often need to operate complex equipment and manage various experimental objects. Precise and real-time cargo tracking and positioning have important applications in cargo management, resource scheduling and space utilization optimization. This puts high requirements on the accuracy and real-time performance of hand-held object detection, especially in the limited space and light conditions in the cabin. Traditional target handheld object detection methods often cannot balance accuracy and efficiency.

[0003] Gesture recognition refers to the technology of expressing certain instructions or emotions through the movements of hands, arms, heads or bodies. Current gesture recognition mainly relies on computer vision and sensor technology. Vision-based gesture recognition methods capture hand images through RGB cameras or depth cameras and use deep learning models or traditional image processing techniques for gesture analysis. Zhu Yinlong et al. (Zhu Yinlong, Shen Hongjun, Wu Jie, et al. Gesture recognition research of data glove based on flexible strain sensor [J]. Transactions of the Chinese Agricultural Machinery Society, 2024, 55(6): 451-458) proposed a flexible strain sensor data glove, combined with BP neural network and template matching algorithm, to improve the complex gesture recognition rate to 98.5%. However, wearable devices are not suitable for daily use, and some devices are expensive. Computer vision gesture recognition technology does not require wearing devices and can efficiently recognize gestures through camera acquisition, greatly reducing the burden on astronauts and improving the naturalness and flexibility of gesture operation. Although computer vision is a popular field, few have specifically targeted handheld object detection, and even fewer have applied object detection combined with gesture recognition to the field of space. SUMMARY

[0004] The application aims to provide a handheld object detection method for space station cargo bay cargo, in particular to provide a handheld object detection method MH-YOLOv8 for space station cargo bay cargo by fusing MediaPipe Hands and YOLOv8, which can avoid detecting background objects other than hands, accurately detect the objects held by astronauts, improve the interaction efficiency between astronauts and cargo bay cargo management system, effectively eliminate background interference, accurately identify handheld objects, and thus improve the real-time performance and accuracy of object management.

[0005] The application provides a handheld object detection method for space station cargo bay cargo, comprising the following steps:

[0006] Identifying a gripping gesture in the photographed image to obtain a gripping gesture recognition result;

[0007] Generating a corresponding ROI according to the mapping relationship between the gripping gesture and the ROI of interest based on the gripping gesture recognition result; and

[0008] Taking the ROI as the input of handheld object detection to detect the target object in the photographed image.

[0009] Further, identifying a gripping gesture in the photographed image comprises:

[0010] Collecting a gripping gesture to obtain a gripping gesture dataset and complete the definition of the gripping gesture;

[0011] Detecting the hand region in the input image based on the MediaPipe Hands module, and extracting hand key point features according to the gripping gesture features;

[0012] The hand key points are subjected to high-dimensional feature coding by the feature extraction module to form a feature vector containing gesture spatial structure information; and

[0013] Performing gripping gesture category determination on the feature vector.

[0014] MediaPipe is an open-source cross-platform framework launched by Google Research, which provides solutions in the fields of face detection, gesture recognition, pose estimation, etc., for building high-performance, cross-platform computer vision applications, and the MediaPipe Hands module provides a high-fidelity hand and finger tracking solution.

[0015] Further, in the extraction of hand key point features according to the gripping gesture features, the gripping gesture features include angle features and distance features, the angle features include the bending angle θ of the fingers and the opening and closing angle between the fingers The distance feature includes the Euclidean distance ratio d of the fingers; and / or

[0016] The key hand features include finger bending state features and finger opening / closing state features. The finger bending angle θ and the Euclidean distance ratio d of the fingers reflect the finger bending state, and the finger opening / closing angle... It reflects the characteristics of the opening and closing state between the fingers.

[0017] Furthermore, the grasping gestures include circular grasping gestures, elongated grasping gestures, pinching gestures, and lifting gestures; the grasping gesture dataset is captured in video format, and the captured videos are exported as image datasets. Each image in the image dataset is processed by feature extraction and fusion to form a 12-dimensional feature vector.

[0018] Furthermore, the hand key point features include 21 hand key points, including:

[0019] 0. Wrist, 1. Thumb metacarpophalangeal joint, 2. Base of thumb, 3. Interphalangeal joint of thumb, 4. Tip of thumb, 5. Base of index finger, 6. Proximal interphalangeal joint of index finger, 7. Distal interphalangeal joint of index finger, 8. Tip of index finger, 9. Base of middle finger, 10. Proximal interphalangeal joint of middle finger, 11. Distal interphalangeal joint of middle finger, 12. Tip of middle finger, 13. Base of ring finger, 14. Proximal interphalangeal joint of ring finger, 15. Distal interphalangeal joint of ring finger, 16. Tip of ring finger, 17. Base of little finger, 18. Proximal interphalangeal joint of little finger, 19. Distal interphalangeal joint of little finger, 20. Tip of little finger.

[0020] Furthermore, the finger bending angle θ includes the thumb bending angle θ. thumb ,according to

[0021] Formulas 1 and 2 are used to calculate the bending angle θ of the thumb. thumb :

[0022]

[0023] Where l1 represents the vector from the base of the thumb to the tip of the thumb, l2 represents the vector from the base of the thumb to the wrist, and (x0, y0), (x2, y2), and (x4, y4) represent the coordinates of the wrist, the base of the thumb, and the tip of the thumb in two-dimensional space, respectively; the calculation of the bending angle of the other fingers is the same as the calculation of the bending angle of the thumb. That is, the calculation of the bending angle of the other fingers is the same as Equation 2), the only difference being that the two-dimensional coordinates of the corresponding joints are replaced according to the key hand features (Table 1).

[0024] Furthermore, the finger bending angle θ also includes the index finger bending angle θ. index finger The bending angle θ of the middle fingermiddle finger The bending angle θ of the ring finger ring finger The bending angle θ of the little finger pinky .

[0025] Furthermore, the opening and closing angles of the index and ring fingers are calculated according to Equation 3).

[0026]

[0027] Where l5 represents the vector from the base of the index finger to the midpoint of that finger, l6 represents the vector from the base of the ring finger to the midpoint of that finger, and (x5, y5), (x6, y6), (x 13 y 13 ) and (x 14 y 14 ) represent the coordinates of the base of the index finger, the proximal interphalangeal joint of the index finger, the base of the ring finger, and the proximal interphalangeal joint of the ring finger in two-dimensional space, respectively; the calculation of the opening and closing angles of the other fingers is the same as the calculation of the opening and closing angles of the index finger and the ring finger.

[0028] Furthermore, the calculation of the opening and closing angles of the middle and little fingers is the same as in Equation 3), except that the coordinates of the base of the index finger, the proximal interphalangeal joint of the index finger, the base of the ring finger, and the proximal interphalangeal joint of the ring finger are replaced with the coordinates of the base of the middle finger, the proximal interphalangeal joint of the middle finger, the base of the little finger, and the proximal interphalangeal joint of the little finger.

[0029] Furthermore, in the distance feature, the distance can be Euclidean distance or Manhattan distance. Preferably, the distance is calculated using Euclidean distance to determine the magnitude of the vector.

[0030] Furthermore, the Euclidean distance of the fingers is d, which includes the Euclidean distance of the thumb. thumb The d thumb Let d be the ratio of the magnitude of vector l3 from the tip of the thumb to the base of the little finger to the magnitude of vector l4 from the base of the thumb to the base of the little finger. Calculate d according to Equation 4). thumb :

[0031]

[0032] Where l3 represents the vector from the tip of the thumb to the base of the little finger, and l4 represents the vector from the base of the thumb to the base of the little finger, (x 17 y 17 ) represents the coordinates of the base of the little finger in two-dimensional space.

[0033] Furthermore, the Euclidean distance ratio d of the fingers also includes the Euclidean distance ratio d of the index finger. index finger The Euclidean distance of the middle finger is greater than d. middle fingerThe Euclidean distance of the ring finger is greater than d. ring finger The Euclidean distance of the little finger is θ pinky The Euclidean distance ratio 'd' of the fingers other than the thumb is the ratio of the distance from the fingertip to the wrist, used to distinguish the bending state of the fingers other than the thumb. The calculation of the Euclidean distance ratio 'd' of the fingers other than the thumb is the same as that of the thumb. thumb Calculations, for example, the Euclidean distance of the index finger to d index finger It is the ratio of the magnitude of the vector from the fingertip to the wrist to the magnitude of the vector from the base of the index finger to the wrist.

[0034] Furthermore, extracting key hand features based on grasping gestures also includes:

[0035] A threshold is set for the grasping gesture feature value to distinguish state boundaries: when θ < 40°, the fingers are in a straight state; when θ ≥ 40°, the fingers are in a bent state; when d < 1, the fingers are in a bent state; when d ≥ 1, the fingers are in a straight state; when α < 15, the fingers are in a closed state; when α ≥ 15, the fingers are in an open state.

[0036] Furthermore, in the process of determining the hand gesture category using the feature vector, the feature vector is processed by the SVM classifier to improve classification performance.

[0037] Choose a radial basis kernel function, as shown in Equation 5):

[0038]

[0039] Here, γ controls the influence range of the high-dimensional mapping. For any value of γ, the radial basis function is a non-negative monotonic function. The radial basis kernel function improves classification performance by mapping low-dimensional data to a high-dimensional space and finding a hyperplane in the high-dimensional feature space. At this time, the SVM classification function is Equation 6):

[0040]

[0041] Among them, a i Lagrange multipliers, representing the weights of each support vector in the decision function, x i Let b represent the support vectors in the training set, and let b represent the bias term.

[0042] Furthermore, the extraction of key hand features based on grasping gestures also includes:

[0043] The angle and distance features are concatenated using feature stitching;

[0044] The grasping gesture features are binarized: finger bending is 1, straightening is 0; fingers together is 0, opening is 1; and

[0045] Output characteristics.

[0046] Furthermore, based on the grasping gesture recognition result, according to the mapping relationship between the grasping gesture and the Region of Interest (ROI), a corresponding ROI is generated, including:

[0047] The MediaPipe Hands module is used to detect the hand region in the input image and find the key point of the hand in the upper left corner of the image (x). min ,y min ) and the key point of the hand in the lower right corner (x) max ,y max (), to locate the area of ​​the hand in the image;

[0048] Define a scaling factor for the grasping gesture:

[0049]

[0050] Among them, W bas H bas These are the scaling factors for the region's width and height, respectively; and

[0051] The target area is expanded in the hand region by the scaling factor to obtain the mapping relationship from each grasping gesture to the ROI, and the corresponding ROI is generated to determine the target detection area.

[0052] Furthermore, it also includes:

[0053] A handheld object detection dataset (HIS-13 dataset) was collected for training and validation of MH-YOLOv8. This dataset was captured in a simulated real spacecraft cargo bay environment. MH-YOLOv8 includes MediaPipe and YOLOv8.

[0054] Furthermore, the collected and produced HIS-13 dataset was divided into a 7:2:1 ratio for training, validation, and testing, respectively. Training, validation, and testing were all completed by calling the MH-YOLOv8 script.

[0055] Furthermore, the BackBone of MH-YOLOv8 is responsible for extracting multi-scale semantic features and achieving multi-scale feature fusion through the Neck structure, and finally outputting the target detection results at the Head end.

[0056] This invention has at least the following beneficial effects: 1) This invention provides an innovative solution for the intelligent detection of items held by astronauts, combining gesture recognition and target detection technologies. This significantly improves the interaction efficiency between astronauts and the cargo management system. Compared with traditional manual recording methods, this method not only reduces the workload of astronauts but also avoids errors and high-intensity repetitive labor in manual operations; 2) To improve the accuracy of item detection in aviation, MH-YOLOv8, based on the YOLOv8 architecture, integrates an efficient gesture prior information extraction and classification module. By dynamically adjusting the ROI, this invention… It can effectively eliminate background interference in the complex spacecraft cabin environment and accurately identify handheld items, thereby improving the real-time performance and accuracy of item management. Experimental results show that the present invention improves the target item detection speed by 19% compared with the YOLOv8 benchmark model, and can effectively avoid false detection, missed detection and multiple detection. In addition, the method does not rely on wearable devices and is particularly suitable for the space environment, and has good engineering application value; 3) The present invention not only optimizes the interaction between astronauts and cargo management system and improves the efficiency of mission execution, but also provides more efficient technical support for item management and resource scheduling in future space missions. Attached Figure Description

[0057] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the embodiments of the invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.

[0058] Figure 1 A flowchart of a handheld item detection method for cargo in a space station cargo hold, as shown in some embodiments of the present invention, is illustrated.

[0059] Figure 2 A schematic diagram of 21 key hand points in this invention is shown;

[0060] Figure 3 This diagram illustrates the extraction of thumb bending angle features in some embodiments of the present invention.

[0061] Figure 4 A schematic diagram showing the opening and closing of the index and ring fingers in some embodiments of the present invention is shown;

[0062] Figure 5 The diagram illustrates the extraction of thumb bending distance features in some embodiments of the present invention.

[0063] Figure 6 The flowchart of the hand key point feature extraction module in some embodiments of the present invention is shown;

[0064] Figure 7 A schematic diagram illustrating the mapping relationship between grasping gestures and ROI in some embodiments of the present invention is shown;

[0065] Figure 8 A schematic diagram of a handheld item detection dataset is shown in some embodiments of the present invention;

[0066] Figure 9 The network structure diagram of the handheld item detection method for cargo in a space station cargo hold according to the present invention is shown. Detailed Implementation

[0067] It should be noted that the components in the accompanying drawings may be shown exaggerated for illustrative purposes and may not be to scale.

[0068] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0069] In this invention, unless otherwise specified, the quantifiers “a” and “one” do not exclude scenarios involving multiple elements.

[0070] It should also be noted that, in the embodiments of the present invention, only a portion of the parts or components may be shown for clarity and simplicity. However, those skilled in the art will understand that, under the teachings of the present invention, the required parts or components can be added as needed for specific scenarios.

[0071] It should also be noted that within the scope of this invention, the terms "same", "equal", and "equal to" do not mean that the two values ​​are absolutely equal, but allow for a certain reasonable error. In other words, the terms also cover "substantially the same", "substantially equal", and "substantially equal to".

[0072] It should also be noted that in the description of this invention, the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not explicitly or implicitly suggest that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0073] Furthermore, the embodiments of the present invention describe the process steps in a specific order. However, this is only for the convenience of distinguishing each step, and is not a limitation on the order of each step. In different embodiments of the present invention, the order of each step can be adjusted according to the process.

[0074] In the following embodiments, MediaPipe is an open-source cross-platform framework launched by Google Research, which provides solutions in the fields of face detection, gesture recognition, pose estimation, etc., for building high-performance, cross-platform computer vision applications. The MediaPipe Hands module provides a high-fidelity hand and finger tracking solution. Figure 9 The network structure diagram of the handheld item detection method for cargo in the space station cargo hold is shown. The BackBone of the MH-YOLOv8 module is responsible for extracting multi-scale semantic features and realizing multi-scale feature fusion through the Neck structure. Finally, the target detection result is output at the Head end.

[0075] The handheld item detection dataset in this embodiment includes 13 categories of items, such as fruits, vegetables, and tools. Considering that the system is used in a cargo spacecraft, based on this application scenario, the handheld item detection dataset should meet the following requirements:

[0076] 1) Simulated cabin scenario: Data was collected in a simulated cargo spacecraft cabin environment to ensure applicability;

[0077] 2) Complex background: Considering the large number of items inside the cabin, it is necessary to shoot from different angles and under different lighting conditions to enhance the complexity of the background;

[0078] 3) Diversity of subjects: Images of different experimental personnel and different clothing were collected to simulate changes in a real space environment;

[0079] 4) Gesture diversity: It covers four grasping gestures to take into account the situation where objects are obscured under different gestures;

[0080] Experimenters performed different hand gestures in a simulated cargo hold, which were captured using a Hikvision MINIPTZ camera (2560×1440 resolution, 25fps). The camera was placed 1.4-1.6 meters away from the participants. Taking three participants as an example, the data collection for this handheld object detection dataset included the following scenarios:

[0081] 1) Different clothing: The experimenters wore blue and white clothing respectively. These two colors were chosen because they are colors commonly used by astronauts.

[0082] 2) Left and right hands: Use the left hand and right hand respectively to grasp;

[0083] 3) Rotate your hand: Slowly rotate your hand to obtain images from different angles;

[0084] 4) Clear and blurry images: The experimenter first made the gesture slowly, and then made the same gesture at high speed.

[0085] Finally, static images of the handheld objects are extracted from the acquired video at 15-frame intervals to obtain the final handheld object detection dataset, such as... Figure 8 As shown.

[0086] The following embodiments provide a method for detecting handheld items in cargo compartments of space stations. Figure 1 A flowchart illustrating a method for detecting handheld items in cargo compartments of a space station is shown, including the following steps:

[0087] A handheld object detection dataset was collected for training and validation of the MH-YOLOv8 model. The handheld object detection dataset was captured in a simulated real spacecraft cargo hold environment.

[0088] The grasping gesture is identified in the captured image to obtain the grasping gesture recognition result;

[0089] Based on the hand gesture recognition results, and according to the mapping relationship between the hand gesture and the Region of Interest (ROI) in the image, the corresponding ROI is generated; and

[0090] The ROI is used as input for handheld object detection to detect target objects in the captured image.

[0091] Among them, identifying grasping gestures in the captured images includes:

[0092] Collect grasping gestures to obtain a grasping gesture dataset and complete the grasping gesture definition;

[0093] The MediaPipe Hands module is used to detect the hand region in the input image and extract key hand features based on the grasping gesture characteristics.

[0094] The key hand points are processed by a feature extraction module for high-dimensional feature encoding, forming a feature vector containing information about the spatial structure of the gesture; and

[0095] Grasp the feature vector to determine the type of hand gesture.

[0096] In extracting key hand features based on grasping gesture characteristics, grasping gesture characteristics include angle features and distance features. Angle features include the bending angle θ of the fingers and the opening angle between the fingers. Distance features include the Euclidean distance ratio d of the fingers; key hand features include finger flexion state features and finger opening / closing state features. The finger flexion angle θ and the Euclidean distance ratio d reflect the finger flexion state, while the finger opening / closing angle reflects the finger opening / closing state. This reflects the opening and closing characteristics of the fingers. The bending angle of the fingers and the distance from the fingertip to the wrist are used to distinguish the bending state of each finger. Grasping gestures include circular grasping gestures, elongated grasping gestures, pinching gestures, and lifting gestures; the grasping gesture dataset is captured in video format, and the captured videos are exported as an image dataset. Each image in the image dataset is processed by feature extraction and fusion to form a 12-dimensional feature vector. The MediaPipe Hands module can accurately detect the coordinates of 21 2.5D hand keypoints from a single frame. The hand keypoint features include 21 hand keypoints, as shown in Table 1 and... Figure 2 As shown, it includes: 0. wrist, 1. thumb metacarpophalangeal joint, 2. base of thumb, 3. interphalangeal joint of thumb, 4. fingertip of thumb, 5. base of index finger, 6. proximal interphalangeal joint of index finger, 7. distal interphalangeal joint of index finger, 8. fingertip of index finger, 9. base of middle finger, 10. proximal interphalangeal joint of middle finger, 11. distal interphalangeal joint of middle finger, 12. fingertip of middle finger, 13. base of ring finger, 14. proximal interphalangeal joint of ring finger, 15. distal interphalangeal joint of ring finger, 16. fingertip of ring finger, 17. base of little finger, 18. proximal interphalangeal joint of little finger, 19. distal interphalangeal joint of little finger, 20. fingertip of little finger. Figure 6 The flowchart of the hand key point feature extraction module is shown. After extracting the hand key point features, the bending angle θ of the fingers and the opening and closing angle between the fingers are calculated. And the calculation of the Euclidean distance ratio d of the fingers; the concatenation of angle features and distance features is performed by feature splicing, and the features are directly spliced ​​to form a larger feature vector; the grasping gesture features are binarized: finger bending is 1, straightening is 0; fingers together is 0, opening is 1; and the output features are also performed.

[0097] Table 1 Key Features of the Hand

[0098]

[0099]

[0100] Figure 3 This diagram illustrates the extraction of thumb bending angle features. The thumb bending angle θ is calculated according to equations 1 and 2. thumb :

[0101]

[0102] Where l1 represents the vector from the base of the thumb to the tip of the thumb, l2 represents the vector from the base of the thumb to the wrist, and (x0, y0), (x2, y2) and (x4, y4) represent the coordinates of the wrist, the base of the thumb and the tip of the thumb in two-dimensional space, respectively; the calculation method for the bending angle features of other fingers is the same as above, that is, the calculation of the bending angle of the remaining fingers is the same as Equation 2), the only difference is that the two-dimensional coordinates of the corresponding joint points are replaced according to the key hand features (Table 1).

[0103] Figure 4 A schematic diagram of the opening and closing of the index and ring fingers is shown. The opening and closing angles of the index and ring fingers are calculated according to Equation 3).

[0104]

[0105] Where l5 represents the vector from the base of the index finger to the midpoint of that finger, l6 represents the vector from the base of the ring finger to the midpoint of that finger, and (x5, y5), (x6, y6), (x 13 y 13 ) and (x 14 y 14 ) represent the coordinates of the base of the index finger, the proximal interphalangeal joint of the index finger, the base of the ring finger, and the proximal interphalangeal joint of the ring finger in two-dimensional space, respectively; the calculation of the opening and closing angles of the middle finger and little finger is the same as in equation 3), except that the coordinates of the base of the index finger, the proximal interphalangeal joint of the index finger, the base of the ring finger, and the proximal interphalangeal joint of the ring finger are replaced with the coordinates of the base of the middle finger, the proximal interphalangeal joint of the middle finger, the base of the little finger, and the proximal interphalangeal joint of the little finger.

[0106] The Euclidean distance of the fingers is d, and the Euclidean distance of the thumb is d. thumb , Figure 5 The diagram illustrates the extraction of the thumb's bending distance feature. The Euclidean distance ratio d of the thumb is calculated according to Equation 4). thumb :

[0107]

[0108] Where l3 represents the vector from the tip of the thumb to the base of the little finger, and l4 represents the vector from the base of the thumb to the base of the little finger, (x 17 y 17 This represents the coordinates of the base of the little finger in two-dimensional space. Extracting key hand features based on grasping gestures also includes:

[0109] A threshold is set for the value of the grasping gesture characteristics to distinguish state boundaries: when θ < 40°

[0110] When θ < 40°, the fingers are in a straight state; when θ ≥ 40°, the fingers are in a bent state; when d < 1, the fingers are in a bent state; when d ≥ 1, the fingers are in a straight state; when α < 15, the fingers are in a closed state; when α ≥ 15, the fingers are in an open state.

[0111] In the process of determining the hand gesture category by capturing the feature vector, the feature vector is used to determine the hand gesture category to improve classification performance:

[0112] Choose a radial basis function, as shown in Equation 5):

[0113]

[0114] Here, γ controls the influence range of the high-dimensional mapping. For any value of γ, the radial basis function is a non-negative monotonic function. The radial basis kernel function improves classification performance by mapping low-dimensional data to a high-dimensional space and finding a hyperplane in the high-dimensional feature space. At this time, the SVM classification function is Equation 6):

[0115]

[0116] Among them, a i Lagrange multipliers, representing the weights of each support vector in the decision function, x i Let b represent the support vectors in the training set, and let b represent the bias term.

[0117] Based on the grasping gesture recognition result and the mapping relationship between the grasping gesture and the ROI, such as... Figure 7 As shown, the corresponding ROI is generated, including:

[0118] The MediaPipe Hands module is used to detect the hand region in the input image and find the key point of the hand in the upper left corner of the image (x). min ,y min ) and the key point of the hand in the lower right corner (x) max ,y max (), to locate the area of ​​the hand in the image;

[0119] Define a scaling factor for the grasping gesture:

[0120]

[0121] Among them, W bas H bas These are the scaling factors for the region's width and height, respectively; and

[0122] The target area is expanded by scaling factor in the hand region to obtain the mapping relationship from each grasping gesture to the ROI, and the corresponding ROI is generated to determine the target detection area.

[0123] To evaluate the performance of the above methods, this embodiment uses precision (P), recall (R), F1 score, mean average precision (mAP), and frame rate (FPS) as evaluation metrics. The experimental environment was configured as follows: Ubuntu 22.04 operating system, 14 vCPU Intel(R) Xeon(R) Platinum 8362 CPU @ 2.80GHz, NVIDIA GeForce RTX3090 (24GB) GPU, 30GB dedicated memory, 50GB data memory, and CUDA version 12.1. PyCharm was used as the integrated development environment, Python version 3.12, and the deep learning development framework was PyTorch 2.3.0; some network training parameters are shown in Table 2.

[0124] Table 2 Network Training Parameters

[0125]

[0126] The F1 score is mathematically the harmonic mean of precision and recall, and mAP represents the mean of the area enclosed by the precision-recall curves for all categories and the coordinate axes. Average precision (AP) is the integral of the precision-recall (PR) curve, with recall represented on the x-axis and precision on the y-axis, as shown in Equation 11).

[0127]

[0128] Where TP represents true positives, TN represents true negatives, FP represents false positives, and FN represents false negatives;

[0129]

[0130] Therefore, when there are N categories, the mAP calculation formula is as shown in 12):

[0131]

[0132] The value of F1 is shown in 13):

[0133]

[0134] To verify the superiority of the SVM classifier, experiments were conducted on different machine learning models using the collected hand gesture dataset for comparison. The specific metrics of the experimental results are shown in Table 3. The results show that the hand gesture dataset performs better in all aspects of training on the SVM model, achieving an F1 score of 0.9349.

[0135] Ablation experiments were conducted to compare MH-YOLOv8 and the baseline model YOLOv8n on a simulated cargo hold test set. The YOLOv8n detection model, trained on the simulated cargo hold dataset, was directly tested on the original test set; MH-YOLOv8, however, added an ROI extraction operation. The test results are shown in Table 4. Analysis of Table 4 shows that the YOLOv8n model achieved precision, recall, and mAP all above 0.92, indicating that the original YOLOv8n has good performance and is suitable as a baseline model. The proposed method improves detection speed by 19% compared to the direct YOLOv8n end-to-end detection method while maintaining performance.

[0136] Table 3 Evaluation results of multiple gesture recognition models

[0137]

[0138] Table 4. Model Comparison Results

[0139]

[0140]

[0141] While some embodiments of the present invention have been described in this application, those skilled in the art will understand that these embodiments are merely illustrative. Numerous variations, alternatives, and improvements will arise in those skilled in the art under the teachings of this invention without departing from its scope. The appended claims are intended to define the scope of the invention and thereby cover methods and structures within the scope of the claims themselves and their equivalents.

Claims

1. A method for detecting handheld items in cargo compartments of a space station, characterized in that, Includes the following steps: The grasping gesture is identified in the captured image to obtain the grasping gesture recognition result; Based on the hand gesture recognition results, the corresponding ROI is generated according to the mapping relationship between the hand gesture and the ROI of the image of interest; as well as The ROI is used as input for handheld object detection to detect target objects in the captured image.

2. The method for detecting handheld items for cargo in a space station cargo hold according to claim 1, characterized in that, Identifying grasping gestures in captured images includes: Collect grasping gestures to obtain a grasping gesture dataset and complete the grasping gesture definition; The MediaPipe Hands module is used to detect the hand region in the input image and extract key hand features based on the grasping gesture characteristics. The key hand points are subjected to high-dimensional feature encoding by a feature extraction module to form a feature vector containing gesture spatial structure information; and The feature vector is used to determine the type of hand gesture.

3. The method for detecting handheld items for cargo in a space station cargo hold according to claim 2, characterized in that, In extracting key hand features based on grasping gesture characteristics, the grasping gesture characteristics include angle features and distance features. The angle features include the bending angle θ of the fingers and the opening angle between the fingers. The distance feature includes the Euclidean distance ratio d of the fingers; and / or The key hand features include finger bending state features and finger opening / closing state features. The finger bending angle θ and the Euclidean distance ratio d of the fingers reflect the finger bending state, and the finger opening / closing angle... It reflects the characteristics of the opening and closing state between the fingers.

4. The method for detecting handheld items for cargo in a space station cargo hold according to claim 3, characterized in that, The hand key point features include 21 hand key points, including:

0. Wrist, 1. Thumb metacarpophalangeal joint, 2. Base of thumb, 3. Interphalangeal joint of thumb, 4. Tip of thumb, 5. Base of index finger, 6. Proximal interphalangeal joint of index finger, 7. Distal interphalangeal joint of index finger, 8. Tip of index finger, 9. Base of middle finger, 10. Proximal interphalangeal joint of middle finger, 11. Distal interphalangeal joint of middle finger, 12. Tip of middle finger, 13. Base of ring finger, 14. Proximal interphalangeal joint of ring finger, 15. Distal interphalangeal joint of ring finger, 16. Tip of ring finger, 17. Base of little finger, 18. Proximal interphalangeal joint of little finger, 19. Distal interphalangeal joint of little finger, 20. Tip of little finger.

5. The method for detecting handheld items for cargo in a space station cargo hold according to claim 4, characterized in that, The finger bending angle θ includes the thumb bending angle θ. thumb The bending angle θ of the thumb is calculated according to equations 1 and 2. thumb : Where l1 represents the vector from the base of the thumb to the tip of the thumb, l2 represents the vector from the base of the thumb to the wrist, and (x0, y0), (x2, y2) and (x4, y4) represent the coordinates of the wrist, the base of the thumb and the tip of the thumb in two-dimensional space, respectively; the calculation of the bending angle of the other fingers is the same as the calculation of the bending angle of the thumb.

6. The method for detecting handheld items for cargo in a space station cargo hold according to claim 4, characterized in that, Calculate the opening and closing angles of the index and ring fingers according to formula 3). Where l5 represents the vector from the base of the index finger to the midpoint of that finger, l6 represents the vector from the base of the ring finger to the midpoint of that finger, and (x5, y5), (x6, y6), (x 13 y 13 ) and (x 14 y 14 ) represent the coordinates of the base of the index finger, the proximal interphalangeal joint of the index finger, the base of the ring finger, and the proximal interphalangeal joint of the ring finger in two-dimensional space, respectively; the calculation of the opening and closing angles of the other fingers is the same as the calculation of the opening and closing angles of the index finger and the ring finger.

7. The method for detecting handheld items for cargo in a space station cargo hold according to claim 4, characterized in that, The Euclidean distance of the fingers is d, and the Euclidean distance of the thumb is d. thumb The d thumb Let d be the ratio of the magnitude of vector l3 from the tip of the thumb to the base of the little finger to the magnitude of vector l4 from the base of the thumb to the base of the little finger. Calculate d according to Equation 4). thumb : Where l3 represents the vector from the tip of the thumb to the base of the little finger, and l4 represents the vector from the base of the thumb to the base of the little finger, (x 17 y 17 ) represents the coordinates of the base of the little finger in two-dimensional space.

8. The method for detecting handheld items for cargo in a space station cargo hold according to claim 4, characterized in that, Extracting key hand features based on grasping gestures also includes: A threshold is set for the grasping gesture feature value to distinguish state boundaries: when θ < 40°, the fingers are in a straight state; when θ ≥ 40°, the fingers are in a bent state; when d < 1, the fingers are in a bent state; when d ≥ 1, the fingers are in a straight state; when α < 15, the fingers are in a closed state; when α ≥ 15, the fingers are in an open state.

9. The method for detecting handheld items for cargo in a space station cargo hold according to claim 4, characterized in that, In determining the hand gesture category of the feature vector, the feature vector is used by the SVM classifier to determine the hand gesture category: Choose a radial basis kernel function, as shown in Equation 5): Here, γ controls the influence range of the high-dimensional mapping. For any value of γ, the radial basis function is a non-negative monotonic function, and the SVM classification function is Equation 6): Among them, a i Lagrange multipliers, representing the weights of each support vector in the decision function, x i Let b represent the support vectors in the training set, and let b represent the bias term.

10. The method for detecting handheld items for cargo in a space station cargo hold according to claim 1, characterized in that, Based on the hand gesture recognition result, and according to the mapping relationship between the hand gesture and the Region of Interest (ROI) in the image, a corresponding ROI is generated, including: The MediaPipe Hands module is used to detect the hand region in the input image and find the key point of the hand in the upper left corner of the image (x). min ,y min ) and the key point of the hand in the lower right corner (x) max ,y max (), to locate the area of ​​the hand in the image; Define a scaling factor for the grasping gesture: Among them, W bas H bas These are the scaling factors for the region's width and height, respectively; and The target area is expanded in the hand region by the scaling factor to obtain the mapping relationship from each grasping gesture to the ROI, and the corresponding ROI is generated.

Images