Hand binding method and device, electronic equipment, storage medium and product

Abstract

The invention provides a hand binding method and device, electronic equipment, a storage medium and a product. The method comprises the following steps: detecting a hand frame, a first human-shaped frame, skeleton points and an external human-shaped frame corresponding to the skeleton points from a cabin image; matching a first human-shaped frame in the cabin image with a preset seat area, matching an external human-shaped frame corresponding to the skeleton points with the first human-shaped frame, and determining seat information corresponding to the skeleton points; and matching the skeleton points with the hand frame, and determining the seat information corresponding to the skeleton points matched with the hand frame as the seat information corresponding to the hand frame. According to the scheme, through matching between the human-shaped frame and the seat area, matching between the skeleton points and the human-shaped frame, and matching between the skeleton points and the hand frame, the seat of the human-shaped frame is transmitted to the skeleton points and then transmitted to the hand, so that the seat corresponding to the hand can be determined, binding between the hand and the seat is realized, the seat of the user executing the hand action is effectively positioned, and the user experience is improved. And the recognition accuracy of the gesture control instruction is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a hand-binding method, device, electronic device, storage medium, and product. Background Technology

[0002] With the rapid development of artificial intelligence technology, in the field of intelligent driving cockpits, obtaining commands through AI gesture recognition technology has become the main upper-level input for current intelligent vehicle user interaction functions, greatly improving the actual driving experience. Users in different positions within the cockpit have different gesture control permissions. In scenarios with multiple users and multiple hands, traditional hand detection methods cannot effectively locate the position of the user performing the hand action within the cockpit, thus affecting the accuracy of gesture control command recognition. Summary of the Invention

[0003] Based on the above requirements, this application proposes a hand binding method, device, electronic device, storage medium, and product that can bind various cockpit positions to hand positions, thereby effectively locating the position of the user performing hand actions in the cockpit and improving the recognition accuracy of gesture control commands.

[0004] To achieve the above objectives, this application proposes the following technical solution: According to a first aspect of the embodiments of this application, a hand binding method is provided, including: Detect the hand bounding box, the first humanoid bounding box, the skeletal points, and the bounding boxes corresponding to the skeletal points from the cockpit image; The first humanoid frame in the cockpit image is matched with a pre-set seating area, and the outer humanoid frame corresponding to the skeletal point is matched with the first humanoid frame to determine the seating information corresponding to the skeletal point. The skeletal points are matched with the hand frame, and the seat information corresponding to the skeletal points that match the hand frame is determined as the seat information corresponding to the hand frame.

[0005] Optionally, matching the first humanoid bounding box in the cockpit image with a pre-defined seating area, and matching the outer humanoid bounding box corresponding to the skeletal point with the first humanoid bounding box to determine the seating information corresponding to the skeletal point, includes: The first human-shaped frame in the cockpit image is matched with a pre-set seating area to determine the seating information corresponding to the first human-shaped frame; The bounding box corresponding to the skeletal point is matched with the first human figure frame, and the seat information corresponding to the first human figure frame that matches the bounding box corresponding to the skeletal point is determined as the seat information corresponding to the skeletal point.

[0006] Optionally, the skeletal points include wrist skeletal points and hand skeletal points; Matching the skeletal points with the hand frame, and determining the seat information corresponding to the skeletal points that match the hand frame as the seat information corresponding to the hand frame, includes: Based on the wrist bone points and hand bone points, the outer hand frame corresponding to the bone points is determined, and the seat information corresponding to the bone points is determined as the seat information of the outer hand frame corresponding to the bone points. Match the external hand frame corresponding to the skeletal point with the hand frame, and determine the seat information of the external hand frame that matches the hand frame as the seat information corresponding to the hand frame.

[0007] Optionally, before matching the first humanoid bounding box in the cockpit image with a pre-set seating area, and matching the outer humanoid bounding box corresponding to the skeletal point with the first humanoid bounding box to determine the seating information corresponding to the skeletal point, the method further includes: Detect the cockpit image to determine the head outline in the cockpit image; Match the head frame in the cockpit image with the first human figure frame in the cockpit image, and determine whether the first human figure frame that matches the head frame is abnormal; If the first human-shaped frame that matches the head frame is abnormal, the first human-shaped frame is adjusted based on the head frame to obtain the target human-shaped frame; Correspondingly, the first humanoid bounding box in the cockpit image is matched with a pre-set seating area, and the outer humanoid bounding box corresponding to the skeletal point is matched with the first humanoid bounding box to determine the seating information corresponding to the skeletal point, including: The target human-shaped frame is matched with a pre-set seating area, and the circumscribed human-shaped frame corresponding to the skeletal point is matched with the target human-shaped frame to determine the seating information corresponding to the skeletal point.

[0008] Optionally, determining whether the first human-shaped frame matching the head frame is abnormal includes: Calculate the distance on the horizontal axis between the center point of the first human-shaped frame that matches the human head frame and the center point of the human head frame; If the distance is within the preset distance range, then the first human-shaped frame that matches the human head frame is determined to be normal; or, If the distance is not within the preset distance range, the first human-shaped frame that matches the human head frame is determined to be abnormal.

[0009] Optionally, based on the head frame, the first human figure frame is adjusted to obtain the target human figure frame, including: According to the pre-set ratio parameters between the head frame and the human shape frame, the head frame is expanded outward to obtain the second human shape frame corresponding to the head frame; According to the pre-set human figure frame adjustment rules, the first human figure frame that matches the human head frame and the second human figure frame corresponding to the human head frame are merged to obtain the target human figure frame.

[0010] Optionally, before matching the skeletal points with the hand frame and determining the seat information corresponding to the skeletal points matching the hand frame as the seat information corresponding to the hand frame, the method further includes: Based on the pre-set contestant rules, a target hand frame is selected from the hand frame; Correspondingly, the skeletal points are matched with the hand frame, and the seat information corresponding to the skeletal points that match the hand frame is determined as the seat information corresponding to the hand frame, including: The skeletal points are matched with the target hand frame, and the seat information corresponding to the skeletal points that match the target hand frame is determined as the seat information corresponding to the target hand frame.

[0011] Optionally, based on pre-set player rules, selecting a target hand frame from the hand frame includes: Set player anchor points within a pre-defined gesture area; Calculate the distance between the contestant's anchor point and the center point of the hand frame, as well as the intersection-union ratio between the gesture area and the hand frame; According to the pre-set contestant rules, the distance, and the intersection-union ratio, the target hand frame corresponding to the contestant anchor point is selected.

[0012] Optionally, detecting the hand bounding box, the first humanoid bounding box, the skeletal points, and the bounding boxes corresponding to the skeletal points from the cockpit image includes: Feature extraction is performed on the cockpit image to obtain hand features and human figure features in the cockpit image; Based on the hand features, predict the hand bounding box in the cockpit image; based on the human shape features, predict the first human shape bounding box in the cockpit image; based on the splicing features of the hand features and the human shape features, predict the skeletal points in the cockpit image and the circumscribed human shape bounding boxes corresponding to the skeletal points.

[0013] Optionally, feature extraction is performed on the cockpit image to obtain hand features and human-shaped features in the cockpit image, including: The cockpit image is input into a pre-constructed feature extraction network to obtain hand features and human features in the cockpit image; wherein, the feature extraction network includes: a backbone network and a PANet fusion network.

[0014] According to a second aspect of the embodiments of this application, a hand-binding device is provided, comprising: The image detection module is used to detect hand frames, first humanoid frames, skeletal points, and the outer humanoid frames corresponding to the skeletal points from cockpit images. The skeleton point seating determination module is used to match the first humanoid frame in the cockpit image with a pre-set seating area, and to match the outer humanoid frame corresponding to the skeleton point with the first humanoid frame to determine the seating information corresponding to the skeleton point. The hand seat determination module is used to match the skeletal points with the hand frame, and determine the seat information corresponding to the skeletal points that match the hand frame as the seat information corresponding to the hand frame.

[0015] According to a third aspect of the embodiments of this application, an electronic device is provided, including: a memory and a processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the above-described hand binding method by running a program in the memory.

[0016] According to a fourth aspect of the embodiments of this application, a storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described hand binding method.

[0017] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including computer program instructions, which, when executed by a processor, cause the processor to implement the above-described hand binding method.

[0018] The hand binding method proposed in this application detects a hand bounding box, a first human bounding box, skeletal points, and the corresponding circumscribed human bounding boxes from a cockpit image. It matches the first human bounding box in the cockpit image with a pre-defined seating area, and matches the circumscribed human bounding box corresponding to the skeletal point with the first human bounding box to determine the seating information corresponding to the skeletal point. It then matches the skeletal point with the hand bounding box, and determines the seating information corresponding to the skeletal point that matches the hand bounding box as the seating information corresponding to the hand bounding box. Using this technical solution, the seat corresponding to the human bounding box can be determined by matching the human bounding box with the seating area. The seat of the human bounding box is transferred to the skeletal point through the association between the skeletal point and the human bounding box. Then, based on the association between the skeletal point and the hand, the seat of the skeletal point is transferred to the hand, thereby determining the seat corresponding to the hand and achieving hand-seat binding. This effectively locates the position of the user performing the hand action in the cockpit, thus improving the recognition accuracy of gesture control commands. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a hand binding method provided in an embodiment of this application.

[0021] Figure 2 This is a schematic diagram of the feature extraction network provided in an embodiment of this application.

[0022] Figure 3 This is a flowchart illustrating another hand binding method provided in an embodiment of this application.

[0023] Figure 4 This is a flowchart illustrating another hand binding method provided in an embodiment of this application.

[0024] Figure 5 This is a schematic diagram of the cockpit gesture area and contestant anchor points provided in an embodiment of this application.

[0025] Figure 6 This is a schematic diagram of a hand binding device provided in an embodiment of this application.

[0026] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of this application are applicable to intelligent driving scenarios, specifically to hand-binding applications in gesture recognition. By employing the technical solutions of this application, various cockpit positions can be bound to hand positions, thereby effectively locating the user's position within the cockpit and improving the accuracy of gesture control command recognition.

[0028] With the rapid iteration and deep implementation of artificial intelligence technology, intelligent driving cockpits are evolving towards highly humanized and intelligent interaction. AI gesture recognition technology based on computer vision, with its contactless, intuitive, and convenient interactive characteristics, has gradually replaced traditional physical buttons and voice commands, becoming the core upper-level input method for intelligent in-vehicle user interaction functions. Drivers and passengers can precisely control in-vehicle navigation, entertainment systems, air conditioning, and window operation using simple gestures such as waving, making a heart shape, or swiping, without touching any physical devices. This greatly simplifies the driving process, effectively reduces the risk of driver distraction, and significantly improves the safety and comfort of the driving experience.

[0029] However, in real-world cabin usage scenarios, multiple people are often present, with the driver, front passenger, and rear passengers frequently making hand gestures simultaneously. Furthermore, users in different positions within the cabin are assigned differentiated gesture control permissions. For example, the driver can operate core vehicle-related functions, the front passenger can only control the entertainment system, and rear passengers have the authority to adjust the rear air conditioning and seats. This hierarchical permission design places higher demands on the accuracy of gesture recognition technology, requiring not only accurate recognition of the gesture itself but also precise location of the user performing the gesture within the cabin.

[0030] However, traditional hand detection methods mostly fail to incorporate spatial context information such as cabin seating layout and human posture, making it difficult to associate detected hand targets with corresponding user identities. This leads to situations where multiple people and multiple hands are involved, making it difficult for the system to accurately determine which hand belongs to the user with the corresponding operating permissions, resulting in misrecognition and misexecution of gesture control commands. For example, a rear passenger's adjustment gesture may be misinterpreted as the driver's navigation operation, or a front passenger's audio / video switching command may be incorrectly assigned to the rear seat adjustment system, severely impacting the reliability of smart cockpit interaction functions and user experience.

[0031] Based on this, this application proposes a hand binding method. This technical solution can bind each cockpit position to a hand position, thereby effectively locating the position of the user performing the hand action in the cockpit. This solves the problem in the prior art that the position of the user performing the hand action in the cockpit cannot be effectively located, thus affecting the recognition accuracy of gesture control commands.

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] Exemplary methods See Figure 1 As shown in the figure, this application proposes a hand binding method. The method includes: S101. Detect the hand bounding box, the first humanoid bounding box, the skeletal point, and the bounding box corresponding to the skeletal point from the cockpit image.

[0034] In this embodiment, to perform gesture recognition of user hand movements inside the car cabin, it is necessary to capture images of the car cabin in real time to obtain cabin images. The hands in the cabin images are then bound to the seats to determine the identity information of the user to whom the hands belong. Then, according to the gesture recognition rules corresponding to the user's identity information, the user's hand movements are recognized to determine the corresponding instructions, so that the controller inside the car cabin can execute the instructions corresponding to the hand movements to achieve gesture control.

[0035] After acquiring the cockpit image, this embodiment needs to perform detection on the cockpit image, including human figure detection, hand detection, skeletal point detection, and bounding rectangular boxes around the skeletal points of each human figure. This embodiment obtains the first human figure bounding box in the cockpit image by human figure detection, obtains the hand bounding box in the cockpit image by hand detection, obtains the skeletal points of each human figure in the cockpit image by skeletal point detection, and then binds the skeletal points of each human figure with rectangular boxes to obtain the bounding human figure bounding box corresponding to the skeletal points of each human figure.

[0036] In this embodiment, human and hand detection in the cockpit image can be performed by preprocessing the cockpit image (e.g., denoising, brightness / contrast normalization, etc.), and then using a deep learning object detection model (e.g., YOLOv5, YOLOv8, SSD, Faster R-CNN, etc.) to perform human and hand detection on the preprocessed cockpit image, thereby obtaining human bounding boxes and hand bounding boxes in the cockpit image. Alternatively, a target detection model capable of simultaneously performing human and hand detection can be pre-trained, or separate target detection models for human and hand detection can be pre-trained to perform human and hand detection on the preprocessed cockpit image respectively.

[0037] In this embodiment, skeletal point detection of the cockpit image can be based on a deep learning-based human pose estimation model, such as OpenPose, HRNet, and SimplePose. The model extracts features of the human body at different scales through a feature pyramid network, predicts the two-dimensional coordinates and confidence scores of preset skeletal points, and filters out skeletal points with confidence scores below a preset threshold, retaining valid joints. Based on prior rules for human skeletal connections (such as shoulder→elbow→wrist, hip→knee→ankle), the valid joints are connected to form a human pose skeleton, creating visualized skeletal lines, thus obtaining the skeletal points of each human figure in the cockpit image. In this embodiment, the preset skeletal points include not only the conventional 17 joints, such as nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle, but also six hand joints, such as the tip of the left middle finger, the base of the left middle finger, the base of the left thumb, the tip of the right middle finger, the base of the right middle finger, and the base of the right thumb.

[0038] In this embodiment, a bounding box is constructed for each humanoid skeleton point. First, the coordinate set of the skeleton points of a single humanoid is extracted, and the coordinate set is traversed to calculate the minimum (x_min) and maximum (x_max) x-coordinate and the minimum (y_min) and maximum (y_max) y-coordinate of all skeleton points of a single humanoid. The upper left corner coordinates of the bounding box of a single humanoid skeleton point are (x_min, y_min), and the lower right corner coordinates are (x_max, y_max). The width W of the bounding box is W = x_max - x_min, and the height H is H = y_max - y_min. The calculated bounding box of the skeleton points of a single humanoid is used as the bounding box of the humanoid corresponding to the skeleton points of that single humanoid, thereby obtaining the bounding boxes of the skeleton points of each humanoid included in the cockpit image. In this embodiment, after calculating the bounding rectangle of the skeletal points of a single human figure, the bounding rectangle can be expanded, for example, by expanding it outward by 5-10 pixels, to ensure complete coverage of the human body area surrounding all skeletal point sets. The expanded rectangle is then used as the bounding rectangle of the human figure corresponding to the skeletal points of a single human figure.

[0039] In one specific implementation, detecting the hand bounding box, the first human bounding box, the skeletal points, and the bounding boxes corresponding to the skeletal points from the cockpit image includes the following steps: First, feature extraction is performed on the cockpit image to obtain hand and human features in the cockpit image.

[0040] This embodiment extracts features from the cockpit image. Traditional computer vision-based feature extraction methods can be used, such as contour and edge-based feature extraction or skin color and texture-based feature extraction, to extract human and hand features from the cockpit image. Alternatively, deep learning-based feature extraction methods can be employed, such as using the backbone networks of object detection models like YOLOv5, YOLOv8, and Faster R-CNN to extract deep features of the human figure and hands.

[0041] Furthermore, this embodiment can also pre-construct a feature extraction network, inputting the cockpit image into the pre-constructed feature extraction network to obtain hand features and human-shaped features from the cockpit image. The feature extraction network can include a backbone network and a PANet fusion network. For example... Figure 2 As shown, the backbone network consists of convolutional layers P0-P6. Convolutional layers P3-P6 all output hand and human features. Each convolutional layer in P3-P6 inputs the extracted hand and human features into the PANet fusion network (i.e., Figure 2 In PANet feature fusion, the PANet fusion network includes a top-down path (i.e., ... Figure 2 Top-down path and bottom-up path (i.e.) Figure 2 Bottom-uppath in (according to) Figure 2 As shown, the hand and human features output by convolutional layers P3-P6 are input to each layer in the top-down path, respectively, and feature fusion is performed on the hand and human features from top to bottom. Each layer in the top-down path transmits the fused hand and human features to each corresponding layer in the bottom-up path. The bottom-up path then performs feature fusion on the hand and human features from bottom to top, and outputs the hand and human features, respectively. Figure 2 The hands head1-head4 and the humanoid head1-head4.

[0042] Second, based on hand features, predict the hand bounding box in the cockpit image; based on human shape features, predict the first human shape bounding box in the cockpit image; and based on the splicing features of hand and human shape features, predict the skeletal points in the cockpit image and the corresponding bounding boxes of the skeletal points.

[0043] In this embodiment, after extracting hand and human features from the cockpit image, the hand bounding box in the cockpit image is predicted based on the hand features, the first human bounding box in the cockpit image is predicted based on the human features, and the skeletal points in the cockpit image and the corresponding bounding boxes of the skeletal points are predicted based on the splicing features of the hand and human features.

[0044] This embodiment can employ a target detection model that includes the aforementioned pre-built feature extraction network. After extracting hand and human features using the feature extraction network, it uses the hand features (i.e., ...) to perform the target detection. Figure 2 The bounding boxes for the hands (head1-head4) are predicted based on human features (i.e., ... Figure 2 The first human bounding box is predicted using the humanoid features (head1-head4) in the image. Hand and humanoid features are concatenated to obtain concatenated features (i.e., hand head1 + humanoid head1, hand head2 + humanoid head2, hand head3 + humanoid head3, hand head4 + humanoid head4). Then, based on these concatenated features, skeletal points in the cockpit image and their corresponding bounding boxes are predicted. This embodiment can simultaneously predict the hand bounding box, the first human bounding box, skeletal points, and their corresponding bounding boxes using a single object detection model. This eliminates the need for separate hand detection networks, humanoid detection networks, and skeletal point detection networks, combining the previously three independent detection tasks (humanoid, hand, and skeletal points) into a single network detection task, significantly reducing the computational cost of multiple models.

[0045] Furthermore, the target detection model in this embodiment can predict not only the 17 conventional joint points, but also 6 hand joint points when predicting skeletal points, so as to achieve hand binding.

[0046] S102. Match the first humanoid bounding box in the cockpit image with the pre-set seat area, and match the outer humanoid bounding box corresponding to the skeletal point with the first humanoid bounding box to determine the seat information corresponding to the skeletal point.

[0047] To associate a hand in a cockpit image with a seat, this embodiment requires matching a first humanoid bounding box in the cockpit image with a pre-defined seating area, and matching the outer bounding box corresponding to a skeletal point with the first humanoid bounding box. The seat information of the humanoid bounding box is then transferred to the skeletal point, thereby determining the seat information corresponding to the skeletal point. Notably, all skeletal points of the same humanoid bounding box correspond to the same seat information.

[0048] In one specific implementation, the first human-shaped bounding box in the cockpit image is matched with a pre-defined seating area, and the bounding box corresponding to the skeletal point is matched with the first human-shaped bounding box to determine the seating information corresponding to the skeletal point. This process specifically includes the following steps: First, the first human-shaped frame in the cockpit image is matched with a pre-set seating area to determine the seat information corresponding to the first human-shaped frame.

[0049] This embodiment requires dividing the cockpit image into seating areas. For example, for a five-seat, two-row car cockpit, the cockpit image can be divided into the driver's seat area, the front passenger seat area, the rear left seat area, the rear right seat area, and the rear middle seat area.

[0050] Each first humanoid bounding box in the detected cockpit image is matched with a pre-defined seating area to determine the seating area matched by each first humanoid bounding box. The cockpit location information corresponding to this seating area is then used as the seating information corresponding to the first humanoid bounding box. For example, the seating information may include seat identification and coordinate data of the seating area.

[0051] In this embodiment, the first human bounding box is matched with the seat area. This can be done using Hungarian matching, for example, by calculating the distance (e.g., Euclidean distance) between the center point of each detected first human bounding box and the center point of each seat area. A cost matrix is ​​then constructed based on these calculated distances. Assuming m first human bounding boxes are detected in the cabin image and there are n seat areas in the cabin, an m×n cost matrix C can be constructed, where C... ij This represents the distance between the i-th first human-shaped bounding box and the j-th seat area. Besides distance, other factors can be considered to construct the cost matrix, such as the consistency between the orientation of the human-shaped bounding box and the orientation of the seat, and the degree of matching between the size of the human-shaped bounding box and the size of the seat area. These factors are quantified and integrated into the calculation of the cost matrix to improve matching accuracy. The matching result with the minimum cost matrix value is taken as the final matching result. This matching result records the matching relationship between the first human-shaped bounding box and the seat area. Based on the seat area matched by the first human-shaped bounding box, the seat information corresponding to the first human-shaped bounding box is determined.

[0052] Alternatively, the first human-shaped bounding box can be matched with the seat area using a direct matching method based on the intersection-over-union ratio (IoU). The IoU ratio of the first human-shaped bounding box with each seat area bounding box is calculated, and the seat area with the largest IoU ratio with the first human-shaped bounding box that exceeds a preset threshold is taken as the seat area matched by the first human-shaped bounding box. Based on the seat area matched by the first human-shaped bounding box, the seat information corresponding to the first human-shaped bounding box is determined.

[0053] Alternatively, matching the first human bounding box with the seat area can also be based on deep learning-based feature matching. First, a cabin matching dataset is constructed, including matching samples labeled with the true matching relationships between sample human bounding boxes and sample seat areas, encompassing different sitting postures, lighting conditions, and occlusion scenarios. Then, the sample human bounding boxes and sample seat areas are input into a matching model to predict the predicted matching relationship. The model is trained with the goal of minimizing the difference between the predicted and true matching relationships. Finally, all first human bounding boxes and all seat areas in the cabin image are input into the trained matching model to obtain the matching relationships between all first human bounding boxes and all seat areas. Based on the seat areas matched by the first human bounding box, the seat information corresponding to the first human bounding box is determined.

[0054] Second, the bounding box corresponding to the skeleton point is matched with the first human figure frame, and the seat information corresponding to the first human figure frame that matches the bounding box corresponding to the skeleton point is determined as the seat information corresponding to the skeleton point.

[0055] In this embodiment, after determining the seat information corresponding to each first humanoid bounding box in the cockpit image, it is also necessary to match the outer humanoid bounding box corresponding to the skeletal point with the first humanoid bounding box to determine the first humanoid bounding box matched by the outer humanoid bounding box corresponding to the skeletal point of each humanoid. Then, the seat information corresponding to the first humanoid bounding box that matches the outer humanoid bounding box corresponding to the skeletal point is determined as the seat information corresponding to the skeletal point. The matching method used to match the outer humanoid bounding box corresponding to the skeletal point with the first humanoid bounding box can be any of the matching methods proposed in the above steps for matching the first humanoid bounding box with the seat area; this embodiment will not elaborate further.

[0056] S103. Match the skeleton points with the hand frame, and determine the seat information corresponding to the skeleton points that match the hand frame as the seat information corresponding to the hand frame.

[0057] In this embodiment, after determining the seating information corresponding to the skeletal points of each human figure, the hand skeletal point region is extracted from the skeletal points of each human figure. Then, the hand skeletal point region is matched with the hand frame, and the seating information corresponding to the skeletal points in the hand skeletal point region that matches the hand frame is determined as the seating information corresponding to the hand frame. The matching method used to match the hand skeletal point region with the hand frame can be any of the matching methods proposed in the above steps for matching the first human figure frame with the seating region; this embodiment will not elaborate further.

[0058] In one specific implementation, the skeletal points are matched with the hand frame, and the seat information corresponding to the skeletal points that match the hand frame is determined as the seat information corresponding to the hand frame. This specifically includes the following steps: First, based on the wrist and hand bone points, determine the outer hand frame corresponding to the bone points, and determine the seat information corresponding to the bone points as the seat information of the outer hand frame corresponding to the bone points.

[0059] In this embodiment, the skeletal points of each human figure include wrist skeletal points and hand skeletal points. Based on the wrist and hand skeletal points, the circumscribed rectangle of the skeletal point set composed of the wrist and hand skeletal points is determined. This circumscribed rectangle is used as the circumscribed hand frame corresponding to the skeletal point, and the seating information corresponding to the skeletal point is determined as the seating information of the circumscribed hand frame corresponding to the skeletal point. The determination of the circumscribed rectangle of the skeletal point set composed of the wrist and hand skeletal points can adopt the method for determining the circumscribed human figure frame corresponding to the skeletal point in the above embodiment, which will not be repeated in this embodiment.

[0060] Second, match the outer hand frame corresponding to the skeletal point with the hand frame, and determine the seat information of the outer hand frame that matches the hand frame as the seat information corresponding to the hand frame.

[0061] The bounding boxes corresponding to the identified skeletal points are matched with the hand bounding boxes in the cockpit image to obtain the matching relationship between the bounding boxes and the hand bounding boxes. The seat information of the bounding box that matches the hand bounding box is determined as the seat information corresponding to that hand bounding box. The matching method used to match the bounding boxes corresponding to the skeletal points with the hand bounding boxes in the cockpit image can be any of the matching methods proposed in the above steps, such as matching the first human-shaped bounding box with the seat area; this embodiment will not elaborate further.

[0062] In addition, in this embodiment, when using the Hungarian matching method to match the external hand bounding box with the hand bounding box, the cost matrix is ​​set as follows: First, the Euclidean distance between the center point of the hand bounding box and the center point of the circumscribed hand bounding box is calculated, and this Euclidean distance is divided by the diagonal length of the cockpit image to obtain the normalized distance loss value. Then, the intersection-union ratio (CUI) between the hand bounding box and the circumscribed hand bounding boxes is calculated, and this CUI is used as the region overlap loss value. Finally, based on pre-set cost matrix construction rules and the distance loss value and region overlap loss value, the cost matrix is ​​determined. This embodiment determines the cost value of the cost matrix between the hand bounding box and each circumscribed hand bounding box, and the circumscribed hand bounding box with the smallest cost value is selected as the matching circumscribed hand bounding box. The cost matrix construction rules are as follows: Loss=(1-α)×dis_loss+α×(1-iou) Where Loss represents the cost matrix, dis_loss represents the normalized distance loss value, iou represents the intersection-union ratio between the hand bounding box and the outer hand bounding box, and α is preferably set to 0.7.

[0063] After determining the seat information corresponding to the hand frame in the cockpit image, this embodiment can bind the hand in the cockpit image to the seat. Based on the binding relationship between the hand and the seat, the user identity information to which each hand belongs can be determined. The user identity information includes: driver, front passenger, rear left seat passenger, rear right seat passenger, rear middle seat passenger, etc.

[0064] By recognizing hand gestures in each hand frame in the cockpit image, the hand gesture corresponding to each hand frame is determined. Based on the hand gesture corresponding to each hand frame, the user identity information to which each hand frame belongs, and the gesture recognition rules corresponding to each user identity information, the control command represented by the hand gesture corresponding to each hand frame is determined. The controller in the cockpit executes the recognized control command, thereby realizing gesture control.

[0065] As described above, the hand binding method proposed in this application detects a hand frame, a first human-shaped frame, skeletal points, and the corresponding circumscribed human-shaped frames from a cockpit image. It matches the first human-shaped frame in the cockpit image with a pre-set seating area, and matches the circumscribed human-shaped frames corresponding to the skeletal points with the first human-shaped frame to determine the seating information corresponding to the skeletal points. It then matches the skeletal points with the hand frame, and determines the seating information corresponding to the skeletal points that match the hand frame as the seating information corresponding to the hand frame. Using this embodiment, the seat corresponding to the human-shaped frame can be determined by matching the human-shaped frame with the seating area. The seat of the human-shaped frame is transferred to the skeletal point through the association between the skeletal point and the human-shaped frame. Then, based on the association between the skeletal point and the hand, the seat of the skeletal point is transferred to the hand, thereby determining the seat corresponding to the hand and achieving hand-seat binding. This effectively locates the position of the user performing the hand action in the cockpit, thereby improving the recognition accuracy of gesture control commands.

[0066] As an optional implementation, this application also proposes a hand binding method. See [link to relevant documentation]. Figure 3 As shown, the method includes: S301. Detect the hand bounding box, the first human bounding box, the skeletal point, and the bounding box corresponding to the skeletal point from the cockpit image.

[0067] S302. Detect the cockpit image and determine the head bounding box in the cockpit image.

[0068] This embodiment performs head detection on cockpit images to determine the bounding boxes of heads in the cockpit images. Specifically, head features can be extracted from the cockpit images, and the bounding boxes of heads in the cockpit images can be predicted based on these features. The detection of head bounding boxes can employ the same human detection method used in the previous embodiment, which will not be elaborated upon in this embodiment.

[0069] Furthermore, this embodiment can detect the head bounding box simultaneously from the cockpit image, including the hand bounding box, the first human bounding box, the skeletal points, and the corresponding bounding boxes. Figure 2 When the feature extraction network shown extracts hand and human features, it can also extract head features at the same time. The extraction method is the same as that for hand and human features. Then, the target detection model containing the feature extraction network is used to predict the hand bounding box, the first human bounding box, the skeletal point and the bounding box corresponding to the skeletal point. At the same time, the target detection model can also predict the head bounding box in the cockpit image based on the head features.

[0070] S303. Match the head frame in the cockpit image with the first human figure frame in the cockpit image, and determine whether the first human figure frame that matches the head frame is abnormal.

[0071] This embodiment matches the head frame in the cockpit image with the first human-shaped frame in the cockpit image to determine the matching relationship between the head frame and the first human-shaped frame. The matching method used to match the head frame with the first human-shaped frame can be any of the matching methods proposed in the previous embodiments for matching the first human-shaped frame with the seat area; this embodiment will not elaborate further.

[0072] This embodiment needs to determine whether the first human-shaped frame that matches the head frame is abnormal based on the positional relationship between the head frame and the first human-shaped frame that matches the head frame. For example, if the matching head frame and the first human-shaped frame conform to the human body rules (that is, the head frame is located within the area where the head is located in the first human-shaped frame), it means that the first human-shaped frame that matches the head frame is normal. If the matching head frame and the first human-shaped frame do not conform to the human body rules, it means that the first human-shaped frame that matches the head frame is abnormal.

[0073] In one specific implementation, determining whether the first human-shaped frame matching the human head frame is abnormal includes the following steps: First, calculate the distance on the horizontal axis between the center point of the first human-shaped frame that matches the human head frame and the center point of the human head frame.

[0074] This embodiment first calculates the coordinates of the center point of the head frame and the coordinates of the center point that matches the head frame. Then, it calculates the distance on the horizontal axis between the center point of the first humanoid frame that matches the head frame and the center point of the head frame. This distance is in the target direction, and the horizontal axis is perpendicular to the direction of the human body in the first humanoid frame, which is from head to toe. For example, if the direction of the human body in the first humanoid frame is vertical, then the horizontal axis is horizontal, and the horizontal distance between the center point of the first humanoid frame that matches the head frame and the center point of the head frame needs to be calculated.

[0075] Second, if the distance is within the preset distance range, the first human figure frame that matches the head frame is determined to be normal; or, if the distance is not within the preset distance range, the first human figure frame that matches the head frame is determined to be abnormal.

[0076] In this embodiment, if the distance between the center point of the first human-shaped frame matching the head frame and the center point of the head frame on the horizontal axis is too large, it indicates that the person in the first human-shaped frame may be reaching into the armrest box or other seating area, resulting in a larger first human-shaped frame. This could lead to other human-shaped frames being missed, affecting the accuracy of hand binding. Therefore, the first human-shaped frame in this case needs to be identified as abnormal so that it can be adjusted. This embodiment pre-sets a preset distance range. When the distance is not within the preset distance range, it indicates that the distance is too large, and the first human-shaped frame matching the head frame is identified as abnormal. When the distance is within the preset distance range, it indicates that the distance is normal, and the first human-shaped frame matching the head frame is also normal.

[0077] S304. If the first human figure frame that matches the head frame is abnormal, then adjust the first human figure frame based on the head frame to obtain the target human figure frame.

[0078] If an anomaly is identified in the first human-shaped frame that matches the head frame, then the first human-shaped frame is adjusted based on the head frame to obtain the target human-shaped frame. The specific steps are as follows: First, the head frame is expanded outward according to the pre-set proportional parameters between the head frame and the human shape frame to obtain the second human shape frame corresponding to the head frame.

[0079] This embodiment requires pre-setting proportional parameters between the head frame and the human shape frame. Specifically, it involves statistically analyzing the length and width ratios of the head frame and the human shape frame for a typical seated posture in a real vehicle scenario. The resulting height ratio, width ratio, and the horizontal coordinate shift ratio of the head frame's origin are used as the proportional parameters between the head frame and the human shape frame. Specifically, the upper left corner of the head frame in a normal-direction cockpit image is used as the origin, with the horizontal direction to the right as the positive direction of the horizontal axis and the vertical direction downwards as the ordinate axis. A normal-direction cockpit image is one where the roof is at the top and the bottom is at the bottom. Mapping a human shape frame to a head frame follows a certain pattern: the height and width of the head frame are appropriately enlarged, while the upper left corner of the head frame is proportionally enlarged to obtain the upper left corner coordinate of the human shape frame, which is the same as enlarging the width. This enlargement ratio is the aforementioned shift ratio. The enlargement ratio is obtained by statistically calculating the head frame and human shape frame based on a large amount of data collected from real vehicles.

[0080] In this embodiment, the head frame is expanded outward according to a pre-set proportional parameter between the head frame and the human shape frame to obtain a second human shape frame corresponding to the head frame. The specific expansion rules are as follows: Roi_humanE_x=Roi_head_x-Roi_head_w×hx_s; Roi_humanE_y=Roi_head_y-Roi_head_h / 5; Roi_humanE_w=Roi_head_w×w_s; Roi_humanE_h=Roi_head_h×h_s.

[0081] In the aforementioned expansion rule, the positive direction of the horizontal axis is from the top left corner to the top right corner of the human head frame, and the positive direction of the vertical axis is from the top left corner to the bottom left corner of the human head frame. Roi_humanE_x represents the horizontal coordinate of the top left corner of the second human head frame, Roi_head_x represents the horizontal coordinate of the top left corner of the human head frame, Roi_head_w represents the width of the human head frame, and hx_s represents the translation ratio of the horizontal coordinate of the origin of the human head frame; Roi_humanE_y represents the vertical coordinate of the top left corner of the second human head frame, Roi_head_y represents the vertical coordinate of the top left corner of the human head frame, and Roi_head_h represents the height of the human head frame. Since the upper boundary of the human head frame is basically just above the human head, and in a real vehicle scenario, dividing the height of the human head frame by 5 is already a small value, then subtracting this value from the vertical coordinate of the top left corner of the human head frame can simulate a relatively accurate vertical coordinate of the upper boundary of the second human head frame. In this embodiment, a small fixed value can be preset, and the ordinate of the upper left corner of the head frame minus this fixed value can be used as the ordinate of the upper left corner of the second humanoid frame. However, since the distance between each person and the shooting device in the cockpit is different, the height of each humanoid frame in the cockpit image will also be different. If the ordinate of the upper left corner of the head frame is set to be minus the fixed value to calculate the ordinate of the upper left corner of the second humanoid frame, the ratio of the second humanoid frame to the head frame will be different for different head frames. Compared with the above method of dividing the head frame height by 5, the accuracy is lower. In addition, this embodiment is not limited to dividing the head frame height by 5. It can also be divided by 4 or 6, as long as the value is a small value. Roi_humanE_w represents the width of the second humanoid frame, Roi_head_w represents the width of the head frame, and w_s represents the width ratio between the head frame and the humanoid frame; Roi_humanE_h represents the height of the second humanoid frame, Roi_head_h represents the height of the head frame, and h_s represents the height ratio between the head frame and the humanoid frame.

[0082] Second, according to the pre-set human figure frame adjustment rules, the first human figure frame that matches the human head frame and the second human figure frame corresponding to the human figure frame are merged to obtain the target human figure frame.

[0083] In this embodiment, after expanding the head frame through the above steps to obtain the second humanoid frame corresponding to the head frame, it is necessary to combine the first humanoid frame detected from the cockpit image that matches the head frame with the second humanoid frame obtained by expanding the head frame to obtain a more accurate target humanoid frame, thereby achieving the adjustment of the first humanoid frame. In this embodiment, the humanoid frame adjustment rules are preset as follows: Roi_humanR=α×Roi_humanD+(1-α)Roi_humanE Wherein, Roi_humanR represents the target human bounding box corresponding to the head bounding box, Roi_humanD represents the first human bounding box detected from the cockpit image that matches the head bounding box, Roi_humanE represents the second human bounding box obtained by expanding the head bounding box, and α represents the weighting weight, which is a dynamic value. When the head bounding box matches the corresponding first human bounding box, α is preferably set to 0.68; when the head bounding box does not match the corresponding first human bounding box, α is set to 0. That is to say, when matching the head bounding box in the cockpit image with the first human bounding box in the cockpit image in step S303, for head bounding boxes that do not match the first human bounding box, it belongs to the human bounding box missing scenario. At this time, a target human bounding box can be obtained directly by expanding the head bounding box, thereby compensating for the human bounding box missing detection, improving the accuracy of human bounding box detection, and thus improving the accuracy of hand binding.

[0084] S305. Match the target human-shaped frame with the pre-set seating area, and match the outer human-shaped frame corresponding to the skeleton point with the target human-shaped frame to determine the seating information corresponding to the skeleton point.

[0085] In this embodiment, for cases where the first human-shaped frame is abnormal or missed, after identifying a normal target human-shaped frame, the target human-shaped frame is matched with a pre-set seating area, and the circumscribed human-shaped frame corresponding to the skeletal point is matched with the target human-shaped frame to determine the seating information corresponding to the skeletal point. For cases where the first human-shaped frame is normal, the normal first human-shaped frame is still matched with the pre-set seating area, and the circumscribed human-shaped frame corresponding to the skeletal point is matched with the first human-shaped frame to determine the seating information corresponding to the skeletal point. The specific execution method of this step has been described in the above embodiments and will not be repeated in this embodiment.

[0086] S306. Match the skeleton points with the hand frame, and determine the seat information corresponding to the skeleton points that match the hand frame as the seat information corresponding to the hand frame.

[0087] This embodiment detects the human head frame in the cockpit image, determines whether the detected first human frame is abnormal, and adjusts the abnormal first human frame, thereby improving the accuracy of the human frame in the cockpit image. This improves the matching accuracy between the human frame and the seat area, as well as the matching accuracy between the human frame and the human frame bounding the skeletal point. The seat information corresponding to the human frame is then transmitted to the skeletal point, and then to the hand frame, thus improving the accuracy of the seat information corresponding to the hand frame. In other words, the binding accuracy between the hand and the seat is improved.

[0088] In this embodiment, the execution method of step S301 is the same as that of step S101 in the above embodiment, and the execution method of step S306 is the same as that of step S103 in the above embodiment. This embodiment will not repeat the details.

[0089] As an optional implementation, this application also proposes a hand binding method. See [link to relevant documentation]. Figure 4 As shown, the method includes: S401. Detect the hand bounding box, the first human bounding box, the skeletal point, and the bounding box corresponding to the skeletal point from the cockpit image.

[0090] S402. Match the first humanoid bounding box in the cockpit image with the pre-set seat area, and match the outer humanoid bounding box corresponding to the skeletal point with the first humanoid bounding box to determine the seat information corresponding to the skeletal point.

[0091] S403. Based on the pre-set player rules, select the target hand frame from the hand frame.

[0092] When there are multiple people in the cabin, there will be more hands. For hands in poor positions, the accuracy of gesture recognition will be affected. Therefore, this embodiment needs to select hand positions and hand frames with better field of vision from all hand frames for different seats as target hand frames, and bind the target hand frames to the seats to improve the accuracy of gesture recognition.

[0093] In this embodiment, a pre-set contestant rule is used to select a target hand frame from all hand frames. This contestant rule can involve detecting the size and position of the hand frames, and selecting hand frames located within a preset area of ​​good visibility and whose size reaches a preset size threshold as target hand frames. Alternatively, this embodiment can also use the following steps to select a target hand frame from the hand frames: First, set player anchor points in the pre-defined gesture area.

[0094] In this embodiment, a gesture area is pre-defined in the cockpit image based on the field of view. This gesture area is the region with the best field of view in the cockpit image, such as... Figure 5 In the diagram, the shaded rectangular area represents the pre-defined gesture area. This embodiment also requires setting player anchor points within the gesture area, such as... Figure 5 Anchor points C, B, and A are set in the driver's seat, passenger seat, and rear seat respectively to select the target hand frame corresponding to each anchor point. If the number of target hand frames to be selected is to be increased, more anchor points can be set as needed. This embodiment will not be described in detail.

[0095] Second, calculate the distance between the contestant's anchor point and the center point of the hand frame, as well as the intersection-over-union ratio between the gesture area and the hand frame.

[0096] This embodiment determines the center point of each hand frame, and then calculates the distance between the contestant's anchor point and the center point of each hand frame. This distance can be calculated using Euclidean distance. It also needs to calculate the intersection-union ratio (IUGR) between each hand frame and the gesture area.

[0097] Third, select the target hand bounding box corresponding to the contestant's anchor point according to the pre-set contestant rules, distance, and intersection-union ratio.

[0098] In this embodiment, contestant rules are pre-set. Based on these contestant rules and the calculated distance between the contestant anchor point and the center point of the hand frame, as well as the intersection-union ratio between the hand frame and the gesture area, the cost loss value between the contestant anchor point and each hand frame is calculated. The hand frame with the smallest cost loss value is taken as the target hand frame corresponding to the contestant anchor point. The target hand frame corresponding to each contestant anchor point can be determined according to this method.

[0099] The pre-defined contestant rules are the cost loss function between the contestant's anchor point and the hand bounding box, as follows: Loss=(1-α)×dis_loss+α×(1-iou) Where Loss represents the cost loss between the contestant's anchor point and the hand bounding box, dis_loss represents the distance between the contestant's anchor point and the center point of the hand bounding box divided by the diagonal length of the cockpit image, and iou represents the intersection-union ratio between the hand bounding box and the gesture region. Since smaller hand bounding boxes have smaller iou values, to avoid selecting smaller hand bounding boxes and to prioritize selecting larger foreground hand bounding boxes, the weight α corresponding to iou value is relatively large, preferably set to 0.8.

[0100] S404. Match the skeleton points with the target hand frame, and determine the seat information corresponding to the skeleton points that match the target hand frame as the seat information corresponding to the target hand frame.

[0101] After selecting the target hand bounding box through the above steps, the skeletal points are matched with the target hand bounding box, and the seating information corresponding to the skeletal points that match the target hand bounding box is determined as the seating information corresponding to the target hand bounding box. The specific execution method of this step has been described in detail in the above embodiments, and will not be repeated in this embodiment.

[0102] Matching the target hand frame with the skeleton point allows the seat information of the skeleton point to be transmitted to the target hand frame. This avoids situations where, when the driver's area is empty and the passenger's hand reaches into the driver's area, although the target hand frame corresponding to that hand is used as the target hand frame for the driver's anchor point, this target hand frame needs to be bound to the passenger's seat information. After this matching step, the target hand frame corresponding to the driver's anchor point will be correctly bound to the passenger's, and it can be correctly identified no matter where the passenger's hand is, thus improving the accuracy of hand-seat binding.

[0103] As an optional implementation, this application also proposes a hand binding method. The method includes: First, detect the hand bounding box, the first humanoid bounding box, the skeletal points, and the bounding boxes corresponding to the skeletal points from the cockpit image.

[0104] Second, the cockpit image is detected to determine the head outlines in the cockpit image.

[0105] Third, match the head frame in the cockpit image with the first human figure frame in the cockpit image to determine whether the first human figure frame that matches the head frame is abnormal.

[0106] Fourth, if the first human-shaped frame that matches the head frame is abnormal, then the first human-shaped frame is adjusted based on the head frame to obtain the target human-shaped frame.

[0107] Fifth, match the target human-shaped bounding box with the pre-set seating area, and match the bounding box of the skeleton point with the target human-shaped bounding box to determine the seating information corresponding to the skeleton point.

[0108] Sixth, based on the pre-set contestant rules, select the target hand frame from the hand frame.

[0109] Seventh, match the skeleton points with the target hand frame, and determine the seat information corresponding to the skeleton points that match the target hand frame as the seat information corresponding to the target hand frame.

[0110] The specific execution method of the above steps has been described in detail in the above embodiments, and will not be repeated in this embodiment.

[0111] Exemplary device Accordingly, this application also provides a hand restraint device, see [link to relevant documentation]. Figure 6 As shown, the device includes: The image detection module 100 is used to detect hand bounding boxes, first human bounding boxes, skeletal points, and corresponding outer human bounding boxes from cockpit images. The skeleton point seat determination module 110 is used to match the first human-shaped frame in the cockpit image with a pre-set seat area, and to match the outer human-shaped frame corresponding to the skeleton point with the first human-shaped frame to determine the seat information corresponding to the skeleton point. The hand seat determination module 120 is used to match the skeletal points with the hand frame and determine the seat information corresponding to the skeletal points that match the hand frame as the seat information corresponding to the hand frame.

[0112] As can be seen from the above description, the hand binding device proposed in this application can determine the seat corresponding to the human-shaped frame by matching the human-shaped frame with the seat area, and transfer the seat of the human-shaped frame to the bone points through the association relationship between the bone points and the human-shaped frame. Then, based on the association relationship between the bone points and the hand, the seat of the bone points is transferred to the hand, thereby determining the seat corresponding to the hand and realizing the binding between the hand and the seat. This can effectively locate the position of the user performing the hand action in the cockpit, thereby improving the recognition accuracy of gesture control commands.

[0113] As an optional implementation, another embodiment of this application discloses a skeletal point positioning determination module 110, specifically used for: Match the first human-shaped frame in the cockpit image with the pre-set seating area to determine the seat information corresponding to the first human-shaped frame; Match the bounding box of the skeleton point with the first human figure frame, and determine the seat information corresponding to the first human figure frame that matches the bounding box of the skeleton point as the seat information corresponding to the skeleton point.

[0114] As an optional implementation, another embodiment of this application discloses that the skeletal points include wrist skeletal points and hand skeletal points; the hand seat determination module 120 is specifically used for: Based on wrist and hand bone points, the bounding box of the hand corresponding to the bone point is determined, and the seat information corresponding to the bone point is determined as the seat information of the bounding box of the hand corresponding to the bone point. Match the outer hand frame corresponding to the skeletal point with the hand frame, and determine the seat information of the outer hand frame that matches the hand frame as the seat information corresponding to the hand frame.

[0115] As an optional implementation, another embodiment of this application discloses a hand binding device, which further includes an anomaly detection module and a human-shaped frame adjustment module.

[0116] The image detection module 100 is also used to detect cockpit images and determine the head bounding boxes in the cockpit images; The anomaly detection module is used to match the head frame in the cockpit image with the first human figure frame in the cockpit image, and to determine whether the first human figure frame that matches the head frame is abnormal. The human frame adjustment module is used to adjust the first human frame based on the human head frame if the first human frame that matches the human head frame is abnormal, so as to obtain the target human frame. Furthermore, the skeleton point seating determination module 110 is also used to match the target human shape frame with a pre-set seating area, and to match the outer human shape frame corresponding to the skeleton point with the target human shape frame to determine the seating information corresponding to the skeleton point.

[0117] As an optional implementation, another embodiment of this application discloses an anomaly detection module, specifically used for: Calculate the distance on the horizontal axis between the center point of the first human-shaped frame that matches the human head frame and the center point of the human head frame. If the distance is within the preset distance range, then the first human figure frame that matches the human head frame is considered normal; or, If the distance is not within the preset distance range, the first human figure frame that matches the human head frame is determined to be abnormal.

[0118] As an optional implementation, another embodiment of this application discloses a human-shaped frame adjustment module, specifically used for: According to the pre-set proportional parameters between the head frame and the human shape frame, the head frame is expanded outward to obtain the second human shape frame corresponding to the head frame. According to the pre-set human frame adjustment rules, the first human frame that matches the human head frame and the second human frame corresponding to the human frame are merged to obtain the target human frame.

[0119] As an optional implementation, another embodiment of this application discloses a hand binding device, which further includes a player module.

[0120] The contestant module is used to select a target hand frame from the hand frame based on pre-set contestant rules; Furthermore, the hand seat determination module 120 is also used to match the skeletal points with the target hand frame, and determine the seat information corresponding to the skeletal points that match the target hand frame as the seat information corresponding to the target hand frame.

[0121] As an optional implementation, another embodiment of this application discloses a player module, specifically used for: Set player anchor points within a pre-defined gesture area; Calculate the distance between the player's anchor point and the center point of the hand frame, as well as the intersection-union ratio between the gesture area and the hand frame; Based on the pre-set contestant rules, distance, and intersection-union ratio, select the target hand bounding box corresponding to the contestant's anchor point.

[0122] As an optional implementation, another embodiment of this application discloses an image detection module 100, including a feature extraction unit and a prediction unit.

[0123] The feature extraction unit is used to extract features from the cockpit image to obtain hand and human features in the cockpit image; The prediction unit is used to predict the hand bounding box in the cockpit image based on hand features, predict the first human bounding box in the cockpit image based on human features, and predict the skeletal points in the cockpit image and the corresponding bounding boxes based on the spliced ​​features of hand and human features.

[0124] As an optional implementation, another embodiment of this application discloses a feature extraction unit, specifically used for: The cockpit image is input into a pre-built feature extraction network to obtain hand features and human features in the cockpit image; the feature extraction network includes a backbone network and a PANet fusion network.

[0125] The hand binding device provided in this embodiment belongs to the same concept as the hand binding method provided in the above embodiments of this application. It can execute the hand binding method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects for executing the hand binding method. Technical details not described in detail in this embodiment can be found in the specific processing content of the hand binding method provided in the above embodiments of this application, and will not be repeated here.

[0126] Exemplary electronic devices Another embodiment of this application also provides an electronic device, see [link to relevant documentation] Figure 7 As shown, the device includes: Memory 200 and processor 210; The memory 200 is connected to the processor 210 and is used to store programs; The processor 210 is used to implement the hand binding method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0127] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0128] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them: A bus can include a pathway for transmitting information between various components of a computer system.

[0129] Processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0130] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0131] The memory 200 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0132] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0133] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0134] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0135] The processor 210 executes the program stored in the memory 200 and calls other devices, and can be used to implement the various steps of any of the hand binding methods provided in the above embodiments of this application.

[0136] Exemplary computer program products and storage media In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the hand binding methods according to various embodiments of this application as described in the "Exemplary Methods" section of this specification.

[0137] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0138] Furthermore, embodiments of this application may also be storage media storing a computer program, which is executed by a processor in the steps of the hand binding methods according to various embodiments of this application described in the "Exemplary Methods" section above.

[0139] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0140] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0141] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0142] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0143] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0144] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0145] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0146] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0147] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0148] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0149] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A hand binding method, characterized in that, include: Detect the hand bounding box, the first humanoid bounding box, the skeletal points, and the bounding boxes corresponding to the skeletal points from the cockpit image; The first humanoid frame in the cockpit image is matched with a pre-set seating area, and the outer humanoid frame corresponding to the skeletal point is matched with the first humanoid frame to determine the seating information corresponding to the skeletal point. The skeletal points are matched with the hand frame, and the seat information corresponding to the skeletal points that match the hand frame is determined as the seat information corresponding to the hand frame.

2. The hand binding method according to claim 1, characterized in that, Matching the first humanoid bounding box in the cockpit image with a pre-defined seating area, and matching the outer humanoid bounding box corresponding to the skeletal point with the first humanoid bounding box to determine the seating information corresponding to the skeletal point, includes: The first human-shaped frame in the cockpit image is matched with a pre-set seating area to determine the seating information corresponding to the first human-shaped frame; The bounding box corresponding to the skeletal point is matched with the first human figure frame, and the seat information corresponding to the first human figure frame that matches the bounding box corresponding to the skeletal point is determined as the seat information corresponding to the skeletal point.

3. The hand binding method according to claim 1, characterized in that, The skeletal points include wrist skeletal points and hand skeletal points; Matching the skeletal points with the hand frame, and determining the seat information corresponding to the skeletal points that match the hand frame as the seat information corresponding to the hand frame, includes: Based on the wrist bone points and hand bone points, the outer hand frame corresponding to the bone points is determined, and the seat information corresponding to the bone points is determined as the seat information of the outer hand frame corresponding to the bone points. Match the external hand frame corresponding to the skeletal point with the hand frame, and determine the seat information of the external hand frame that matches the hand frame as the seat information corresponding to the hand frame.

4. The hand binding method according to claim 1, characterized in that, Before matching the first humanoid bounding box in the cockpit image with a pre-defined seating area, and matching the outer humanoid bounding box corresponding to the skeletal point with the first humanoid bounding box to determine the seating information corresponding to the skeletal point, the method further includes: Detect the cockpit image to determine the head outline in the cockpit image; Match the head frame in the cockpit image with the first human figure frame in the cockpit image, and determine whether the first human figure frame that matches the head frame is abnormal; If the first human-shaped frame that matches the head frame is abnormal, the first human-shaped frame is adjusted based on the head frame to obtain the target human-shaped frame; Correspondingly, the first humanoid bounding box in the cockpit image is matched with a pre-set seating area, and the outer humanoid bounding box corresponding to the skeletal point is matched with the first humanoid bounding box to determine the seating information corresponding to the skeletal point, including: The target human-shaped frame is matched with a pre-set seating area, and the circumscribed human-shaped frame corresponding to the skeletal point is matched with the target human-shaped frame to determine the seating information corresponding to the skeletal point.

5. The hand binding method according to claim 4, characterized in that, Determining whether the first human-shaped frame matching the head frame is abnormal includes: Calculate the distance on the horizontal axis between the center point of the first human-shaped frame that matches the human head frame and the center point of the human head frame; If the distance is within the preset distance range, then the first human-shaped frame that matches the human head frame is determined to be normal; or, If the distance is not within the preset distance range, the first human-shaped frame that matches the human head frame is determined to be abnormal.

6. The hand binding method according to claim 4, characterized in that, Based on the head frame, the first human figure frame is adjusted to obtain the target human figure frame, including: According to the pre-set ratio parameters between the head frame and the human shape frame, the head frame is expanded outward to obtain the second human shape frame corresponding to the head frame; According to the pre-set human figure frame adjustment rules, the first human figure frame that matches the human head frame and the second human figure frame corresponding to the human head frame are merged to obtain the target human figure frame.

7. The hand binding method according to claim 1, characterized in that, Before matching the skeletal points with the hand frame and determining the seat information corresponding to the skeletal points that match the hand frame as the seat information corresponding to the hand frame, the method further includes: Based on the pre-set contestant rules, a target hand frame is selected from the hand frame; Correspondingly, the skeletal points are matched with the hand frame, and the seat information corresponding to the skeletal points that match the hand frame is determined as the seat information corresponding to the hand frame, including: The skeletal points are matched with the target hand frame, and the seat information corresponding to the skeletal points that match the target hand frame is determined as the seat information corresponding to the target hand frame.

8. The hand binding method according to claim 7, characterized in that, Based on pre-set player rules, selecting a target hand frame from the hand frame includes: Set player anchor points within a pre-defined gesture area; Calculate the distance between the contestant's anchor point and the center point of the hand frame, as well as the intersection-union ratio between the gesture area and the hand frame; According to the pre-set contestant rules, the distance, and the intersection-union ratio, the target hand frame corresponding to the contestant anchor point is selected.

9. The hand binding method according to claim 1, characterized in that, Detecting hand bounding boxes, first humanoid bounding boxes, skeletal points, and the corresponding bounding boxes of the skeletal points from cockpit images includes: Feature extraction is performed on the cockpit image to obtain hand features and human figure features in the cockpit image; Based on the hand features, predict the hand bounding box in the cockpit image; based on the human shape features, predict the first human shape bounding box in the cockpit image; based on the splicing features of the hand features and the human shape features, predict the skeletal points in the cockpit image and the circumscribed human shape bounding boxes corresponding to the skeletal points.

10. The hand binding method according to claim 9, characterized in that, Feature extraction is performed on the cockpit image to obtain hand features and human figure features in the cockpit image, including: The cockpit image is input into a pre-constructed feature extraction network to obtain hand features and human features in the cockpit image; wherein, the feature extraction network includes: a backbone network and a PANet fusion network.

11. A hand restraint device, characterized in that, include: The image detection module is used to detect hand frames, first humanoid frames, skeletal points, and the outer humanoid frames corresponding to the skeletal points from cockpit images. The skeleton point seating determination module is used to match the first humanoid frame in the cockpit image with a pre-set seating area, and to match the outer humanoid frame corresponding to the skeleton point with the first humanoid frame to determine the seating information corresponding to the skeleton point. The hand seat determination module is used to match the skeletal points with the hand frame, and determine the seat information corresponding to the skeletal points that match the hand frame as the seat information corresponding to the hand frame.

12. An electronic device, characterized in that, include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the hand binding method as described in any one of claims 1 to 10 by running a program in the memory.

13. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the hand binding method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, It includes computer program instructions that, when executed by a processor, cause the processor to implement the hand binding method as described in any one of claims 1 to 10.

Images