A human-computer gesture interaction method, system, electronic device, and storage medium
By extracting key points and performing instance segmentation in the vehicle-to-vehicle interaction, the target posture category is identified and combined interaction commands are constructed, which solves the problem of low recognition accuracy in complex environments and achieves higher recognition accuracy and personalized control.
Patent Information
- Application Number
- CN202311073459.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-24
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-08-24
AI Technical Summary
Existing vehicle-to-vehicle interaction methods suffer from low accuracy or errors in complex recognition environments, especially in crowded places such as squares or exhibitions, where too many image recognition targets lead to interaction errors.
By acquiring the frame image to be processed, key points are extracted and instance segmentation is performed using a preset algorithm to determine the pose category of the target object. The interaction mode is activated only after the interaction start pose is recognized, and pose recognition is performed based on key points and masks to construct combined interaction command poses to improve recognition accuracy.
It improves the accuracy of target recognition in multi-target environments, optimizes the accurate recognition of interaction modes, reduces the impact of other target actions, and enhances the fun and personalized control of the interaction.
Smart Images

Figure CN117111740B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of vehicle intelligent interaction technology, specifically to a human-machine posture interaction method, system, electronic device, and storage medium. Background Technology
[0002] With the popularization of intelligent devices, people have higher and higher requirements for the interactive experience of devices. As one of the carriers of intelligent life, the demand for intelligent interaction of in-vehicle systems is increasing. The interactive control of in-vehicle systems can be specifically divided into in-vehicle interactive control and out-of-vehicle interactive control. Among them, out-of-vehicle interactive control is mainly based on image acquisition devices to acquire and recognize images, and then control the in-vehicle system according to the recognition results.
[0003] However, existing methods for vehicle-to-vehicle interaction often suffer from low accuracy or even errors in human-computer interaction when faced with complex recognition environments, such as squares, roads, exhibitions, and other high-traffic environments. This is because images contain a large number of recognition targets, which can lead to errors in recognizing interactive objects and actions. Summary of the Invention
[0004] In view of the shortcomings of the prior art described above, the present invention provides a human-computer posture interaction method, system, electronic device and storage medium to solve the technical problem of low accuracy or even misidentification of human-computer interaction recognition in complex interaction environments.
[0005] This invention provides a human-computer posture interaction method, comprising: acquiring a frame image to be processed, wherein the frame image contains at least one object to be identified, and the frame image to be processed is acquired through an image acquisition device installed in a vehicle-mounted system; extracting key points from the frame image to be processed according to a first preset algorithm to obtain multiple key points to be processed, and performing instance segmentation on the frame image to be processed according to a second preset algorithm to obtain multiple masks to be processed; determining a target object to be identified based on each key point to be processed and each mask to be processed, and identifying the posture of the target object to be identified to determine the posture category of the target object to be identified in the frame image to be processed; if the posture category is an interaction start posture, controlling the activation of the interaction mode, and identifying the posture of the target object to be identified in the current frame image, wherein the current frame image is a frame image following the frame image to be processed; if the posture of the target object to be identified in the current frame image is an interaction command posture, determining a preset control command that has a mapping relationship with the interaction command posture according to the interaction command posture, and executing the preset control command in the vehicle-mounted system.
[0006] In one embodiment of the present invention, before controlling the activation of the interaction mode and recognizing the posture of the target object in the current frame image, the human-computer posture interaction method further includes: acquiring a preset number of start verification frame images after the frame image to be processed, and recognizing the posture of the target object in each of the start verification frame images, wherein the preset number of start verification frame images are a preset number of consecutive frame images after the frame image to be processed; if the posture of the target object in each of the start verification frame images is an interaction start posture, then controlling the activation of the interaction mode and performing voice broadcast with a first preset prompt tone.
[0007] In one embodiment of the present invention, determining the target recognition object based on each of the key points to be processed and each of the masks to be processed includes: matching the mask regions to which each key point to be processed belongs with each mask to be processed to obtain multiple initial matching combinations, and performing overlap calculations on each initial matching combination to obtain multiple matching combinations to be processed; if there are key points to be processed for multiple objects in each matching combination to be processed, then performing visibility confidence calculations on each key point to be processed, and selecting the point with the highest visibility confidence as the key point to be processed of the mask to be processed, to obtain the key point set corresponding to each mask to be processed; comparing the number of key point sets in each mask to be processed, and determining the mask to be processed with the most key points in the key point set as the target mask; performing three-dimensional coordinate transformation on the key points to be processed in the target mask to obtain the three-dimensional coordinates of each key point to be processed in the target mask, and determining the target recognition object based on the three-dimensional coordinates of each key point to be processed in the target mask.
[0008] In one embodiment of the present invention, the three-dimensional coordinate transformation of the key points to be processed within the target mask includes: acquiring image size information of the frame image to be processed; performing pixel coordinate ratio conversion on the key points to be processed within the target mask according to the image size information to determine the two-dimensional coordinate parameters to be converted for each key point to be processed in the target mask; and calculating each two-dimensional coordinate parameter to be converted based on a preset coordinate transformation algorithm to obtain the three-dimensional coordinates of each key point to be processed within the target mask.
[0009] In one embodiment of the present invention, after determining the target recognition object based on the three-dimensional coordinates of each key point to be processed within the target mask, the human-computer posture interaction method further includes: determining the maximum angle deviation value of each key point to be processed within the target mask based on the three-dimensional coordinates of each key point to be processed within the target mask and the three-dimensional coordinates of each key point of a preset posture type; if the maximum angle deviation value of each key point to be processed is less than or equal to the preset recognition deviation value, then the preset posture type is used as the posture recognition result of the target recognition object.
[0010] In one embodiment of the present invention, after recognizing that the pose of the target object in the current frame image is an interactive command pose, the human-machine pose interaction method further includes: acquiring a preset number of recognition verification frame images after the current frame image, and recognizing the pose of the target object in each of the recognition verification frame images, wherein the preset number of recognition verification frame images are a preset number of consecutive frame images after the current frame image; if the pose of the target object in each of the recognition verification frame images is an interactive command pose, then a preset control command that has a mapping relationship with the interactive command pose is determined according to the interactive command pose, and the preset control command is executed in the vehicle system, and voice broadcast is performed based on the preset control command.
[0011] In one embodiment of the present invention, before determining the target recognition object based on each of the key points to be processed and each of the masks to be processed, the human-computer posture interaction method further includes: calculating the loss weight of the key points to be processed in each target box based on a preset key point loss function, so as to reduce the reduction in recognition accuracy caused by the loss of key points to the frame image to be processed, wherein the target box is the target bounding box generated for each target object during the key point extraction process.
[0012] In one embodiment of the present invention, the loss weight calculation can be performed based on the following preset keypoint loss function:
[0013]
[0014] Where n is the number of bounding boxes, and kpt_loss i nkpt is the keypoint loss value of the i-th target box in the image to be processed. i nkpt is the total number of keypoints in the i-th bounding box. i,v=1 is the total number of visible keypoints in the i-th bounding box, and kpt_loss is the total loss value of keypoints within the frame image to be processed.
[0015] In one embodiment of the present invention, after recognizing the posture of the target object in the current frame image, the human-machine posture interaction method further includes: if the posture of the target object in the current frame image is recognized as the start posture of a combined interaction command, and the recognition results of a preset number of consecutive frames after the current frame image are all the start posture of a combined interaction command, then a new frame image is acquired and the posture of the target object in each newly acquired frame image is recognized; the posture recognition results of the target object in the newly acquired frame image are recorded sequentially until the posture of the target object in the newly acquired frame image is restored to a preset end posture, thereby obtaining the current combined interaction command posture recognition result, and ending the current recognition of the combined interaction command posture of the target object in the current frame image; the combined interaction command posture recognition result is compared with a preset combined interaction command posture, and if there is a preset combined interaction command posture that is consistent with the combined interaction command posture recognition result, then a preset control command that has a mapping relationship with the preset combined interaction command posture is determined, and the preset control command is executed in the vehicle system.
[0016] In one embodiment of the present invention, after recognizing the pose of the target object in the current frame image, the human-computer pose interaction method further includes: if the pose of the target object in the current frame image is recognized as an interaction end pose, then the interaction mode is turned off and a second preset prompt tone is used for voice broadcast.
[0017] This invention also provides a human-machine posture interaction system, comprising: an image acquisition module for acquiring a frame image to be processed, wherein the frame image to be processed contains at least one object to be identified, and the frame image to be processed is acquired through an image acquisition device installed in the vehicle's infotainment system; a posture recognition module for extracting key points from the frame image to be processed according to a first preset algorithm to obtain multiple key points to be processed, and performing instance segmentation on the frame image to be processed according to a second preset algorithm to obtain multiple masks to be processed; determining a target object to be identified based on each key point to be processed and each mask to be processed, and identifying the posture of the target object to be identified, thereby determining the posture category of the target object to be identified in the frame image to be processed; if the posture category is an interaction start posture, controlling the activation of the interaction mode, and identifying the posture of the target object to be identified in the current frame image, wherein the current frame image is a frame image following the frame image to be processed; an instruction management module for determining a preset control instruction that has a mapping relationship with the interaction instruction posture if the posture of the target object to be identified in the current frame image is an interaction instruction posture; and a controller management module for executing the preset control instruction in the vehicle's infotainment system.
[0018] The present invention also provides an electronic device, the electronic device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device enables the human-computer gesture interaction method as described in any of the above embodiments.
[0019] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the human-computer gesture interaction method as described in any of the above embodiments.
[0020] The beneficial effects of this invention are as follows: An embodiment of this invention provides a human-computer posture interaction method, system, electronic device, and storage medium. By acquiring a frame image to be processed, key point extraction and instance segmentation are performed on the frame image according to a preset algorithm. The target object is determined based on the key points and the mask to be processed, and posture recognition is performed to determine the posture category of the target object. If it is an interaction start posture, the interaction mode is activated and the posture of the target object in the current frame image is identified. If it is an interaction command posture, a preset control command is determined based on the interaction command posture and executed on the vehicle system. This method simultaneously extracts key points and masks, and uses key points and masks as common recognition factors, improving the accuracy of target recognition in multi-target environments. Furthermore, the interaction mode is activated only after the interaction start action is recognized, further optimizing the accurate recognition of the target and reducing the impact of other target actions.
[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:
[0023] Figure 1 This is a schematic diagram illustrating the implementation environment of a human-computer gesture interaction method according to an exemplary embodiment of this application;
[0024] Figure 2 This is a flowchart illustrating a human-computer gesture interaction method in an exemplary embodiment of this application;
[0025] Figure 3 This is a schematic diagram illustrating the distribution and recognition of specific human body key points, as shown in an exemplary embodiment of this application.
[0026] Figure 4 This is a schematic diagram illustrating the specific sequence of coordinate transformation of key human body points, as shown in an exemplary embodiment of this application.
[0027] Figure 5 This is a flowchart illustrating a specific human-computer gesture interaction method as an exemplary embodiment of this application;
[0028] Figure 6 This is a flowchart illustrating a specific pose recognition method as an exemplary embodiment of this application;
[0029] Figure 7 This is a block diagram illustrating a human-computer gesture interaction system as shown in an exemplary embodiment of this application;
[0030] Figure 8 This is a block diagram illustrating a specific human-computer posture interaction system as shown in an exemplary embodiment of this application;
[0031] Figure 9 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation
[0032] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.
[0033] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0034] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.
[0035] The term "and / or" used in this application describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship.
[0036] First, it's important to clarify that image recognition refers to the technology of using computers to process, analyze, and understand images in order to identify targets and objects of various patterns. It is a practical application of deep learning algorithms. Currently, image recognition technology generally involves four steps: image acquisition, image preprocessing, feature extraction, and image recognition. However, depending on the target being recognized and the specific recognition process and the objects being recognized in each step, there will be variations.
[0037] The YOLO model is an object detection model used to find specific objects in an image. Object detection requires not only identifying the object's type but also labeling its location, including its center position (x, y), length and width (h, w), and object type. YOLO's prediction is based on the entire image, and it outputs all detected object information at once, including category and location. In some implementable environments of this application, object detection and recognition based on the YOLO model are performed at the image frame acquisition frequency. Therefore, the cycle time of object detection and recognition can be adjusted by changing the image frame acquisition frequency. This adjustment can be specifically tailored to the accuracy requirements of the implementation scheme, and the embodiments of this application do not limit this.
[0038] Instance segmentation refers to distinguishing different instances and labeling them with masks. Therefore, instance segmentation requires more refined mask localization based on object detection, and it also requires distinguishing masks of different instances of the same category based on semantic segmentation.
[0039] A mask is a region or process used to control image processing (all or part) of a selected image, graphic, or object. A mask is a binary image composed of 0s and 1s. When a mask is applied in a function, regions with 1s are processed, while regions with 0s are masked out of the calculation. Image masks are defined by specifying data values, data ranges, finite or infinite values, regions of interest, and annotation files. Any combination of these options can also be used as input to create a mask.
[0040] This application also provides the following beneficial effects: Firstly, the construction of combined interactive command gestures can improve the progress requirements for command determination during the interaction process, avoiding accidental actions in some scenarios that trigger interaction and control the vehicle system to execute commands. Secondly, the construction of combined interactive command gestures can enhance the fun of the interaction and achieve personalized command interaction. After determining the key points to be processed within the target mask, the root node of the human object is calculated, and the relative coordinates of each key point to be processed are calculated sequentially based on the root node to reduce the computational load of 3D coordinate calculation. The selection of a preset number of start verification frames after the processed frame images is used to ensure the interaction action... To improve accuracy, in specific implementation, one feasible approach is to select all frames within two seconds after processing the frame image for recognition, effectively improving the certainty of each pose recognition and the accuracy of interactive actions. Human skeletal parameters are obtained through T-pose or starting pose, and the same batch of data is calibrated. The coordinates of two-dimensional key points are converted to three-dimensional coordinates, and the deviation between poses is calculated through spatial angles, achieving the effect that pose recognition accuracy is unaffected by changes in skeletal parameters or distance. The implementation scheme of this application does not require a large amount of labeled data; only a few images are needed to achieve expanded support for new poses, effectively saving computational resources.
[0041] Figure 1 This is a schematic diagram illustrating the implementation environment of a human-computer gesture interaction method according to an exemplary embodiment of this application. Figure 1 As shown, the implementation environment includes at least personnel 101 and computer equipment 102. After the vehicle 101 acquires a frame image containing personnel 101 through an image acquisition device installed in the vehicle's infotainment system, the vehicle 101 extracts key points from the frame image according to a first preset algorithm to obtain multiple key points to be processed, and performs instance segmentation on the frame image according to a second preset algorithm to obtain multiple masks to be processed. Based on each key point to be processed and each mask to be processed, a target recognition object is determined, and the pose of the target recognition object is recognized to determine the pose category of the target recognition object in the frame image. If the pose category is an interaction start pose, the interaction mode is activated, and the pose of the target recognition object in the current frame image is recognized. If the pose of the target recognition object in the current frame image is recognized as an interaction command pose, a preset control command with a mapping relationship to the interaction command pose is determined according to the interaction command pose, and the preset control command is executed in the vehicle's infotainment system.
[0042] in, Figure 1 The computer device 102 shown can be any computing device that provides computing power support and performs logical operations, such as a microcomputer, embedded computer, or network computer, but is not limited to these. Figure 1The personnel shown are only targets to be identified and do not include a specific number of targets or specific referents. In some implementation environments, the personnel 101 may include multiple personnel for identification processing, and this document does not impose any restrictions on this.
[0043] In the practical application of human-computer interaction in in-vehicle systems, it can be specifically divided into: acquiring the scene image to be recognized; recognizing and determining the target from the image; determining and judging the target's behavior; determining the execution instruction based on the target's behavior; and sending the execution instruction to the in-vehicle controller and execution device for specific instruction execution. In some extended solutions, feedback on the execution result may also be included. In the implementation of this application, an example of an implementable environment is described as follows: When a vehicle is being displayed at an exhibition, in an environment with numerous background personnel and other identifiable targets, when the display personnel, experience personnel, or related technical personnel control the in-vehicle system's interactive functions, the vehicle's recognition of their interactive actions may result in errors in identifying the interactive object or action recognition due to the large number of identifiable targets in the image. This leads to low accuracy or even errors in human-computer interaction recognition.
[0044] The problems illustrated in the examples above are universally applicable to general human-computer interaction processes. It can be seen that in complex interaction environments, the presence of numerous recognition targets in the acquired images can lead to various problems. To address these issues, embodiments of this application propose a human-computer posture interaction method, a human-computer posture interaction system, an electronic device, and a computer-readable storage medium, which will be described in detail below.
[0045] Please see Figure 2 , Figure 2 This is a flowchart illustrating a human-computer gesture interaction method in an exemplary embodiment of this application. This method can be applied to... Figure 2 The implementation environment shown is specifically executed by computer device 102 within that implementation environment. It should be understood that this method can also be applied to other exemplary implementation environments and executed by devices in other implementation environments; this embodiment does not limit the implementation environment to which the method is applicable.
[0046] like Figure 2 As shown, in an exemplary embodiment, the human-computer gesture interaction method includes at least steps S210 to S250, which are described in detail below:
[0047] In step S210, the frame image to be processed is acquired.
[0048] In one embodiment of this application, there is at least one object to be identified in the above-mentioned frame image to be processed. When there is only one object to be identified in the frame image to be processed, after determining the key points and mask to be processed of the target object, the pose is directly identified and a preset control command with a mapping relationship to the pose is executed. When there are multiple objects to be identified in the frame image to be processed, the target object to be identified needs to be determined based on the key points and mask to be processed obtained later. In some possible implementations of this application, if there are multiple objects to be identified, the target object with the most identifiable key points is identified as the specific interactive object. In other implementations, the object to be identified that is at the foremost position in the visible range of the image can also be identified as the target object to be identified. It should be noted that the above exemplary examples are only examples of some possible implementations of this embodiment and do not specifically limit the specific implementation of this solution.
[0049] In one embodiment of this application, the above-mentioned frame image to be processed is acquired by an image acquisition device installed in the vehicle's infotainment system. The image acquisition device may be a front-view camera, a side-view camera, a surround-view camera, or a built-in camera, etc. The camera may specifically include a monocular camera, a binocular camera, a wide-angle camera, etc. Again, the specific type of image acquisition device is not limited.
[0050] In one embodiment of this application, the acquisition of the frame image to be processed is performed on the image at the front end of the vehicle's infotainment system in some embodiments of this solution. In some other embodiments, when the image acquisition direction changes, the direction of the image acquisition device needs to be adjusted accordingly. However, the specific change in the image direction does not substantially change the specific implementation of this solution. Therefore, the acquisition direction of the frame image to be processed is only an example of one possible implementation of this solution and does not limit the specific implementation of this solution.
[0051] In one embodiment of this application, before acquiring the frame image to be processed, the human body posture interaction method, in response to a preset posture library establishment request, acquires the posture image to be recorded, stores the posture image to be recorded in a preset posture storage area, performs image annotation based on the posture image to be recorded, and constructs a mapping relationship between the posture image to be recorded and preset control commands in a preset command library that have a mapping relationship with the image annotation, so as to perform preset posture recording in the preset posture library.
[0052] In one embodiment of this application, the above-mentioned posture image to be recorded may be a fixed-point photograph taken by relevant technicians or data collectors based on the posture to be collected; or the posture may be set and adjusted by a virtual digital human and realized by screenshot or image export; or the posture image to be recorded may be obtained by Internet data to meet the posture requirements of the collection. The specific acquisition method of the posture image to be recorded is not limited here.
[0053] In one embodiment of this application, before acquiring the frame image to be processed, the human body posture interaction method responds to the combined interaction command posture establishment request by calling a preset posture library, listening to the selected type and selection order of the preset posture, constructing an initial combined interaction command posture file based on the selected type and selection order of the preset posture, and constructing a mapping relationship between the initial combined interaction command posture file and the target control command to obtain the combined interaction command posture.
[0054] It should be noted that the above-mentioned combination of interactive gestures supports the reuse of a single preset gesture, but restricts the continuous arrangement of the same preset gestures.
[0055] In one embodiment of this application, the construction of combined interactive command gestures can improve the progress requirements for command determination during the interaction process, and avoid triggering the interaction and controlling the vehicle system to execute commands due to accidental actions in some scenarios. Secondly, the construction of combined interactive command gestures can enhance the fun of the interaction and realize personalized command interaction.
[0056] In step S220, key points are extracted from the image to be processed according to the first preset algorithm to obtain multiple key points to be processed, and instance segmentation is performed on the image to be processed according to the second preset algorithm to obtain multiple masks to be processed.
[0057] In one embodiment of this application, the first preset algorithm adopts the key point recognition algorithm of the YOLO model, and the second preset algorithm adopts the instance segmentation algorithm of the YOLO model.
[0058] In one embodiment of this application, key point extraction of the frame image to be processed according to the first preset algorithm includes obtaining key point prediction boxes through YOLO, determining the offset of each key point from the upper left vertex of the prediction box, and then obtaining the coordinates of the key points. In some scalable solutions, after obtaining the key point prediction boxes based on YOLO, the key points can be determined based on a convolutional neural network.
[0059] In one embodiment of this application, instance segmentation of the frame image to be processed according to the second preset algorithm includes grid division of the frame image to be processed, initial mask prediction, prediction result confidence calculation, prediction loss calculation, post-processing, and instance annotation to obtain multiple masks to be processed.
[0060] In step S230, based on each key point to be processed and each mask to be processed, the target recognition object is determined, and the pose of the target recognition object is recognized to determine the pose category of the target recognition object in the frame image to be processed.
[0061] In one embodiment of this application, determining the target recognition object based on each of the key points to be processed and each of the masks to be processed includes matching the mask regions to which each key point to be processed belongs and each mask to be processed to obtain multiple initial matching combinations, and performing overlap calculations on each of the initial matching combinations to obtain multiple matching combinations to be processed. If there are key points to be processed for multiple objects in each matching combination to be processed, then a visibility confidence calculation is performed on each key point to be processed, and the point with the highest visibility confidence is selected as the key point to be processed of the mask to be processed, thus obtaining the key point set corresponding to each mask to be processed. The number of key point sets in each mask to be processed is compared, and the mask to be processed with the largest number of key points to be processed in the key point set is determined as the target mask. The key points to be processed in the target mask are transformed into three-dimensional coordinates to obtain the three-dimensional coordinates of each key point to be processed in the target mask, and the target recognition object is determined based on the three-dimensional coordinates of each key point to be processed in the target mask.
[0062] In one embodiment of this application, the target identification object is a three-dimensional coordinate structure of a target control object contained within a target mask, such as the three-dimensional coordinate structure of a controller, development and testing personnel, etc., who controls and interacts with the vehicle.
[0063] In one embodiment of this application, the overlap calculation of each matching combination is performed based on a preset overlap algorithm using the IoU overlap algorithm. IoU (Intersection over Union) is a standard for measuring the accuracy of detecting corresponding objects in a specific dataset. IoU is a simple measurement standard, and tasks that derive a prediction range (bounding boxes) in the output can be measured using IoU.
[0064] In one embodiment of this application, the above-mentioned three-dimensional coordinate transformation of the key points to be processed within the target mask includes obtaining image size information of the frame image to be processed, performing pixel coordinate ratio conversion on the key points to be processed within the target mask according to the image size information to determine the two-dimensional coordinate parameters to be converted for each key point to be processed in the target mask, and calculating each two-dimensional coordinate parameter to be converted based on a preset coordinate transformation algorithm to obtain the three-dimensional coordinates of each key point to be processed within the target mask.
[0065] In one embodiment of this application, please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the distribution and recognition of specific human body key points, as shown in an exemplary embodiment of this application. Figure 3 As shown, after determining the key points to be processed within the target mask, since key points 1 and 2 (the left and right eyes) and key points 3 and 4 (the left and right ears) of the head are relatively fixed in position relative to key point 0 during movement, the head can be uniformly represented and calculated using key point 0. Please refer to... Figure 4 , Figure 4 This is a schematic diagram illustrating the specific sequence of coordinate transformation for key human body points, as shown in an exemplary embodiment of this application. Figure 4 As shown, the three-dimensional coordinates of each key point to be processed are calculated in the order of 0, 5→7→9, 6→8→10, 11→13→15, and 12→14→16 respectively.
[0066] In one embodiment of this application, when calculating three-dimensional coordinates, the depth direction is uniformly forward. The motion depth direction is then predicted based on the motion trajectory of consecutive frames combined with this feature. In some preferred embodiments, this can be achieved by restricting the user's posture. For example, if the motion is backward, it can be agreed that a small forward motion is performed first, followed by a backward motion, and the motion speed is relatively uniform to predict the depth direction.
[0067] In one embodiment of this application, based on the two-dimensional coordinate data of the interaction start gesture recognition, the distance between key points of the target object is calculated as a first parameter; based on the two-dimensional coordinate data of the interaction command gesture recognition, the distance between key points of the target object is calculated as a second parameter; and based on the second parameter and the first parameter, a three-dimensional depth parameter is obtained according to a projection algorithm to realize the conversion of two-dimensional coordinates to three-dimensional coordinates. Specifically, it can be calculated using the formula shown in equation (1):
[0068]
[0069] Where z is the z-axis coordinate of the three-dimensional coordinate system, i.e. the depth value, x is the first parameter, y is the second parameter, and l is the original size.
[0070] The original size l can be obtained through the 2D coordinates of the planar motion. Then the relative depth of all points is 0, and the projected distance l' is equal to the original size l. Therefore, for the preset pose, it can be obtained by uniformly collecting the planar poses of the batch.
[0071] In one embodiment of this application, the T-pose, also known as the binding pose, is a preset pose used in computer animation to bind the skeleton of a three-dimensional model. It can also be used as the basic movement of the captured object when using motion capture technology. In some embodiments of this application, the original size can also be obtained based on the pre-designed T-pose and then proportionally converted based on the image size to obtain the original size l.
[0072] In one embodiment of this application, after determining the key points to be processed within the target mask, the two-dimensional coordinates of the midpoint between key points 11 and 12 can be calculated first as the root node of the entire human body object, and the relative coordinates of each key point to be processed can be calculated sequentially based on the root node to reduce the computational amount of three-dimensional coordinate calculation.
[0073] In one embodiment of this application, after determining the target object based on the three-dimensional coordinates of each key point to be processed within the target mask, the maximum angular deviation value of each key point to be processed within the target mask is determined based on the three-dimensional coordinates of each key point to be processed within the target mask and the three-dimensional coordinates of each key point of a preset pose type. If the maximum angular deviation value of each key point to be processed is less than or equal to the preset recognition deviation value, then the preset pose type is taken as the pose recognition result of the target object.
[0074] In one embodiment of this application, when determining the maximum angular deviation value of each key point to be processed within the target mask based on the three-dimensional coordinates of each key point to be processed within the target mask and the three-dimensional coordinates of each key point of a preset pose type, the maximum angular deviation value of each key point to be processed within the target mask is first determined based on the head key point (key point 0) p0 among the key points to be processed within the target mask. x = (a1, a2, a3) and the head key point (key point 0) p0 in the three-dimensional coordinates of each key point of the preset pose type. i The angle deviation value can be obtained by calculating (b1, b2, b3). This angle deviation value can be calculated based on the following formula:
[0075]
[0076] Where, θ0 xi The angle deviation between the 3D coordinates of each key point to be processed within the target mask and the 3D coordinates of key points of preset pose types, specifically the 0th key point.
[0077] In one embodiment of this application, after determining the angle deviation between the three-dimensional coordinates of each key point to be processed within the target mask and the 0th key point among the three-dimensional coordinates of each key point of the preset pose type, the angle deviation values of each key point are solved sequentially, and the maximum value of the angle deviation values of each group of key points is calculated to obtain the deviation value between the pose of the target object and the preset pose. Then, the minimum deviation degree among each preset pose is calculated, and if the deviation degree is less than or equal to the preset recognition deviation value, the pose of the target object is considered to conform to the corresponding preset pose, and is output as the pose recognition result.
[0078] In one embodiment of this application, before determining the target recognition object based on each of the key points to be processed and each of the masks to be processed, loss weights are calculated for the key points to be processed in each target box based on a preset key point loss function, so as to reduce the reduction in recognition accuracy caused by the loss of key points to the frame image to be processed. The target box is the target bounding box generated for each object to be recognized during the key point extraction process.
[0079] In one embodiment of this application, the loss weight calculation can be based on the following preset keypoint loss function:
[0080]
[0081] Where n is the number of bounding boxes, and kpt_loss i nkpt is the keypoint loss value of the i-th target box in the image to be processed. i nkpt is the total number of keypoints in the i-th bounding box. i,v=1 is the total number of visible keypoints in the i-th bounding box, and kpt_loss is the total loss value of keypoints within the frame image to be processed.
[0082] In one embodiment of this application, since the objects to be identified are all foreground objects with no occlusion or missing keypoints, it is necessary to reduce the loss weight for predicted keypoints of objects with occlusion or missing keypoints. Therefore, the training method of the preset YOLO model is modified so that during the preprocessing of the dataset in the preset YOLO model training stage, occlusion in the three categories of visible, occluded, and invisible keypoints is processed as invisible. Secondly, the preset loss function of the YOLO model is improved to reduce the loss calculation weight for objects with fewer visible points, thereby improving the recognition accuracy of the trained YOLO model for objects with more visible points. When calculating the loss weight based on the above-mentioned preset keypoint loss function, the keypoint prediction deviation of human objects with more missing keypoints has a smaller impact on the total keypoint deviation.
[0083] In step S240, if the pose category is the interaction start pose, the interaction mode is activated and the pose of the target object in the current frame image is identified.
[0084] In one embodiment of this application, the aforementioned current frame image is a frame image following the frame image to be processed.
[0085] In one embodiment of this application, if the pose category is a non-interactive start pose, the current identification is stopped.
[0086] In one embodiment of this application, before controlling the activation of the interactive mode and recognizing the pose of the target object in the current frame image, a preset number of start verification frame images are obtained after the frame image to be processed, and the pose of the target object in each of the start verification frame images is recognized. The preset number of start verification frame images are a preset number of consecutive frame images after the frame image to be processed. If the pose of the target object in each of the start verification frame images is an interactive start pose, the interactive mode is activated and a voice broadcast is performed with a first preset prompt tone.
[0087] In one embodiment of this application, if the pose of the target object in each start verification frame image is a non-interactive start pose, then the current recognition is stopped.
[0088] In one embodiment of this application, the selection of a preset number of start verification frames after the above-mentioned frame image to be processed is used to ensure the accuracy of the interactive action. In a specific implementation, one possible implementation is to select all frame images within two seconds after the frame image to be processed for recognition, which effectively improves the certainty of each pose recognition and the accuracy of the interactive action.
[0089] In one embodiment of this application, the aforementioned first preset prompt tone is a voice prompt that announces the status of controlling the activation of the interactive mode. Its content can be voice prompts describing the activation of the mode, such as "Interactive mode is activated!" or "Start gesture interaction". The specific content can be determined according to the specific needs in the implementation of this solution, and can also be set according to user needs. Here, the first preset prompt tone is not specifically limited.
[0090] In one embodiment of this application, after receiving the prompt tone of the first prompt tone, the user can trigger the corresponding preset control command through a single preset posture or a combination of postures. For example, if the corresponding preset control command is triggered by a combination of postures, the user can continuously change the posture from the start of the combined interaction command in place according to the expected posture combination until the posture of the combined interaction command is completed, triggering the preset control command with a mapping relationship.
[0091] Preferably, if the user needs to issue continuous commands, a short pause time can be set and agreed upon between gesture combinations, which can effectively improve the recognition accuracy.
[0092] In one embodiment of this application, after recognizing the pose of the target object in the current frame image, if the pose of the target object in the current frame image is recognized as the start pose of a combined interaction command, and the recognition results of a preset number of consecutive frames after the current frame image are all the start pose of a combined interaction command, then a new frame image is acquired and the pose of the target object in each newly acquired frame image is recognized. The pose recognition results of the target object in the newly acquired frame image are then recorded sequentially until the pose of the target object in the newly acquired frame image returns to the preset end pose, thus obtaining the pose recognition result of this combined interaction command, and ending the recognition of the combined interaction command pose of the target object in the current frame image. Then, the pose recognition result of the combined interaction command is compared with the preset combined interaction command pose. If there is a preset combined interaction command pose that is consistent with the pose recognition result of the combined interaction command, then a preset control command that has a mapping relationship with the preset combined interaction command pose is determined, and the preset control command is executed in the vehicle system.
[0093] In one embodiment of this application, after recognizing that the pose of the target object in the current frame image is the start pose of the combined interaction command, the vehicle system, after receiving consecutive current frame images, removes duplicate poses from adjacent frames and then matches them with preset combined interaction command poses.
[0094] In one embodiment of this application, if the preset combined interaction command gesture belongs to a subset of the deduplicated data frames, the combination is considered to be a hit. Then, a preset control command that has a mapping relationship with the preset combined interaction command gesture is determined, and the preset control command is executed in the vehicle system.
[0095] In one embodiment of this application, if there are multiple hit results, a time-first and combination length-first strategy is adopted to sort them to obtain the optimal first matching result. The matched interval is truncated, and the data of subsequent frames is retained. Matching continues until there are no matching results for the preset combination interaction command gestures in the truncated frames. Then, the preset control commands that have a mapping relationship with the identified combination interaction command gestures are arranged according to the order in which the combination interaction command gestures are identified.
[0096] In one embodiment of this application, after identifying the pose of the target object in the current frame image, if the pose of the target object in the current frame image is identified as the interaction end pose, the interaction mode is turned off and a voice broadcast is performed with a second preset prompt tone.
[0097] In one embodiment of this application, the aforementioned second preset prompt tone is a voice prompt that announces the status of the control to turn off the interaction mode. Its content can be voice prompts such as "Interaction mode is turned off!" or "Gesture interaction ended" to describe the mode being turned on. The specific content can be determined according to the specific needs of the implementation of this solution. The first preset prompt tone can also be set according to user needs. The second preset prompt tone is not specifically limited here.
[0098] In step S250, if the pose of the target object in the current frame image is identified as an interactive command pose, a preset control command with a mapping relationship to the interactive command pose is determined based on the interactive command pose, and the preset control command is executed on the vehicle system.
[0099] In one embodiment of this application, after recognizing that the pose of the target object in the current frame image is an interactive command pose, a preset number of recognition verification frame images are obtained after the current frame image, and the pose of the target object in each of the recognition verification frame images is recognized, wherein the preset number of recognition verification frame images are a preset number of consecutive frame images after the current frame image.
[0100] In one embodiment of this application, if the pose of the target object in each of the recognition verification frame images is an interaction command pose, then a preset control command that has a mapping relationship with the interaction command pose is determined according to the interaction command pose, and the preset control command is executed in the vehicle system, and voice broadcast is performed based on the preset control command.
[0101] Please see Figure 5 , Figure 5 This is a flowchart illustrating a specific human-computer gesture interaction method in an exemplary embodiment of this application.
[0102] like Figure 5 As shown, in a specific embodiment of this application, an image captured by a camera on the vehicle is first obtained to obtain an image frame. This image frame is consistent with the image to be processed in the above embodiment. A preset pose recognition algorithm is used to identify human pose on the image. The preset pose recognition algorithm includes the first preset algorithm, the second preset algorithm, the preset overlap algorithm, the preset key point loss function, the preset coordinate transformation algorithm, and the maximum angle deviation value calculation method in the above embodiment. A pose category data frame is obtained, and it is determined whether the command mode is activated. The command mode is consistent with the interaction mode in the above embodiment.
[0103] In one specific embodiment of this application, if the command mode is activated, it is first determined whether the current frame conforms to the end gesture strategy. If it conforms to the end gesture strategy, the vehicle control command mode is exited, and a third prompt tone is used to prompt the user that the command mode has been successfully closed. The third prompt tone is consistent with the second preset prompt tone in the above embodiment.
[0104] In one specific embodiment of this application, if the ending posture strategy is not met, the command mode is confirmed to be activated. A vehicle control command strategy is then matched to the continuous posture data, and the vehicle is controlled according to the vehicle control command. A second prompt tone is used to notify the user to complete one command. The second prompt tone is consistent with the voice broadcast based on preset control commands in the above embodiment.
[0105] In one specific embodiment of this application, if the command mode is not activated, it is first determined whether the current frame conforms to the start posture strategy. If it conforms to the start posture strategy, the vehicle control command mode is activated, and a first prompt tone is used to prompt the user that the command mode has been successfully activated. The first prompt tone is consistent with the first preset prompt tone in the above embodiment.
[0106] In one specific embodiment of this application, if the starting posture strategy is not met, there is no response, and the identification ends.
[0107] Please see Figure 6 , Figure 6 This is a flowchart illustrating a specific posture recognition method as an exemplary embodiment of this application.
[0108] like Figure 6 As shown, in a specific embodiment of this application, after acquiring image data, key points to be processed and bounding boxes segmented are performed based on YOLO model instance segmentation and YOLO model keypoint recognition. When determining foreground and background, if overlapping bounding boxes exist under YOLO model instance segmentation, the foreground and background are determined based on the area of the instance mask region; if overlapping bounding boxes exist under YOLO model keypoint recognition, the foreground and background are determined based on the number of visible keypoints.
[0109] In one specific embodiment of this application, after foreground object screening, the target bounding box object is segmented based on IOU alignment and key point recognition. Based on the landing position of key points in the alignment instance mask area, the correspondence is corrected to obtain the target pose key points. Then, based on the starting pose key points, it is aligned with the target pose key point data to obtain basic skeletal parameters. Preset pose key points are also obtained. After aligning the preset pose with the acquired batch data to obtain basic skeletal parameters, 2D coordinates are converted to 3D coordinates. The maximum angle deviation between the target pose and the preset pose is calculated, and the preset pose with the smallest deviation is obtained. It is then determined whether it is lower than the sensitivity threshold. This sensitivity threshold is consistent with the preset recognition deviation value in the above embodiment. If it is lower than the sensitivity threshold, the preset pose with the smallest deviation is output as the pose category. If it is not lower than the sensitivity threshold, the current recognition ends and recognition is performed again.
[0110] It should be noted that the above Figure 6 The target bounding box described in the specific pose recognition method embodiment shown in the flowchart is consistent with the mask to be processed in the previous embodiment.
[0111] This invention discloses a human-computer posture interaction method, system, electronic device, and storage medium. By acquiring a frame image to be processed, key point extraction and mask segmentation are performed on the frame image according to a preset algorithm. The target object is determined based on the key points and mask, and posture recognition is performed to determine the posture category of the target object. If it is an interaction start posture, the interaction mode is activated and the posture of the target object in the current frame image is identified. If it is an interaction command posture, a preset control command is determined based on the interaction command posture and executed on the vehicle system. This method simultaneously extracts key points and masks, using them as common recognition factors to improve target recognition accuracy in multi-target environments. Furthermore, the interaction mode is activated only after the interaction start action is detected, further optimizing accurate target recognition and reducing the impact of other target actions. Other beneficial effects include that the construction of combined interaction command postures can improve the progress requirement for command determination during the interaction process, avoiding the triggering of interaction and control due to accidental actions in some scenarios. The vehicle-mounted machine executes commands. Secondly, the construction of combined interactive command gestures can enhance the fun of the interaction and achieve personalized command interaction. After determining the key points to be processed within the target mask, the root node of the human object is calculated, and the relative coordinates of each key point to be processed are calculated sequentially based on the root node to reduce the computational load of three-dimensional coordinate calculation. A preset number of start verification frames are selected after the processed frame image to ensure the accuracy of the interactive action. In a specific implementation, one possible approach is to select all frame images within two seconds after the processed frame image for recognition, effectively improving the certainty of each pose recognition and the accuracy of the interactive action. Human skeletal parameters are obtained through T-pose or start pose, and the same batch of data is calibrated. The coordinates of two-dimensional key points are converted to three-dimensional coordinates, and the deviation between poses is calculated through spatial angles, achieving the effect that the pose recognition accuracy is not affected by changes in skeletal parameters or distance. The implementation scheme of this application does not require a large amount of labeled data; only a few images are needed to achieve expanded support for new poses, effectively saving computational resources.
[0112] The following describes system embodiments of this application, which can be used to execute the human-computer posture interaction method described above in this application. For details not disclosed in the device embodiments of this application, please refer to the embodiments of the human-computer posture interaction method described above in this application.
[0113] Figure 7 This is a block diagram illustrating a human-computer gesture interaction system as an exemplary embodiment of this application. The system can be applied to... Figure 2 The implementation environment shown is specifically configured in computer device 102. This system can also be applied to other exemplary implementation environments and specifically configured in other devices; this embodiment does not limit the implementation environment to which the system is applicable.
[0114] like Figure 7 As shown, this exemplary human-computer posture interaction system includes: an image acquisition module 701, used to acquire a frame image to be processed, wherein at least one object to be identified exists in the frame image to be processed, and the frame image to be processed is acquired through an image acquisition device installed in the vehicle's infotainment system; a posture recognition module 702, used to extract key points from the frame image to be processed according to a first preset algorithm to obtain multiple key points to be processed, and to perform instance segmentation on the frame image to be processed according to a second preset algorithm to obtain multiple masks to be processed; based on each key point to be processed and each mask to be processed, a target object to be identified is determined, and the target object is... The system identifies the pose of the target object and determines the pose category of the target object in the frame image to be processed. If the pose category is an interaction start pose, it controls the activation of the interaction mode and identifies the pose of the target object in the current frame image, which is the frame image after the frame image to be processed. The instruction management module 703 is used to determine a preset control instruction that has a mapping relationship with the interaction instruction pose if the pose of the target object in the current frame image is identified as an interaction instruction pose. The controller management module 704 is used to execute the preset control instruction on the vehicle system.
[0115] Please see Figure 8 , Figure 8 This is a block diagram illustrating a specific human-computer gesture interaction system as an exemplary embodiment of this application. The system can be applied to... Figure 2 The implementation environment shown is specifically configured in computer device 102. This system can also be applied to other exemplary implementation environments and specifically configured in other devices; this embodiment does not limit the implementation environment to which the system is applicable.
[0116] like Figure 8 As shown, the specific human-computer posture interaction system includes an S100 posture management module, an S200 posture recognition module, an S300 instruction management module, and an S400 controller management module. The S100 posture management module includes S101 posture acquisition and input. Posture acquisition can be achieved by the acquisition personnel fixing their posture in front of a monocular camera and then taking pictures. The same acquisition batch by the same acquisition personnel needs to be photographed at a fixed point. The T-pose posture (hands outstretched, feet together, body upright) in front of the camera can also be completed by a virtual digital human. After the acquisition is completed, the data is entered into the system. It needs to be entered according to the acquisition batch, and a T-pose sample within one batch needs to be specified as the benchmark for subsequent human body specification calibration of this group of data.
[0117] In a specific system embodiment, the aforementioned S100 posture management module further includes S102 posture grouping and labeling. For the posture-collected image data from S101, posture groups can be created on the system as needed. Then, batches of data are manually labeled and transferred to the corresponding groups. The group information can specify an image as a representative posture for preview. Additionally, the range of key points to focus on during action recognition should be indicated. For example, for the hands-on-hips pose, in this embodiment, leg movements and head orientation are not restricted, and the range of key point recognition can be marked as 5–12. During posture recognition, this can be used to limit the similarity between the input object's posture and the target posture within this range to improve the recognition rate. Low, medium, and high generalization rate levels can also be set. These three levels correspond to the maximum angular deviation threshold between the collected posture and the target posture. In this embodiment, the preset thresholds for the three levels are: low (maximum deviation angle ≤ 10 degrees), medium (maximum deviation angle ≤ 20 degrees), and high (maximum deviation angle ≤ 30 degrees).
[0118] In a specific system embodiment, the above-mentioned S100 posture management module further includes S103 posture combination. When creating a posture combination, after creating a new posture combination, the preset posture grouping categories can be previewed and selected, and multiple selections of postures are supported. Within the posture combination, the selected posture categories can be arranged in order, supporting the reuse of a single posture, but restricting the continuous arrangement of the same posture.
[0119] In a specific system embodiment, the above-mentioned S200 pose recognition module consists of a preset example segmentation algorithm based on the YOLO model, an improved YOLO key point recognition algorithm, key point and mask correction, 2D coordinate to 3D coordinate conversion, and pose deviation angle calculation, etc.; the input is an image frame, and the output is the preset pose classification result recognized.
[0120] In a specific system embodiment, the S300 instruction management module further includes an S301 instruction strategy configuration. In the instruction strategy, strategies can be created according to the controller. Multiple trigger conditions and execution instructions can be configured within each strategy. For each trigger condition, a corresponding activated gesture combination can be configured. The following is an exemplary example of a specific implementation: For a music player strategy, a gesture combination of hands touching the ears followed by hands making a heart shape above the head can be configured as a condition to trigger the music player to start; hands touching the ears followed by raising the right hand and waving it forward and backward can be configured as a condition to trigger the music player to continue; hands touching the ears followed by a hands-on-hips gesture can be configured as a condition to trigger the music player to pause; hands touching the ears followed by raising the right hand and then lowering it until the right hand hangs down can be configured as a condition to trigger the music player to switch; and hands touching the ears followed by waving the right hand horizontally left and right can be configured as a condition to trigger the music player to exit. To facilitate user memorization and reduce intent entanglement between controllers, it is recommended that the first action in the gesture combination remain consistent for the same controller and be clearly distinguishable from other controllers.
[0121] In a specific system embodiment, the above-mentioned S300 instruction management module also includes S302 instruction recognition. The instruction recognition service will match the matched gesture combinations for the input data fragments, sort the instructions corresponding to the combinations according to the recognition order, and return the results.
[0122] In a specific system embodiment, after receiving the command stream, the S400 vehicle equipment control module will sequentially send the commands to the corresponding controllers, sending the next command after completing one command, until the batch of commands is completed and then returning a status.
[0123] It should be noted that the human-computer posture interaction system and the human-computer posture interaction method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments and will not be repeated here. In practical applications, the human-computer posture interaction system provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.
[0124] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the methods provided in the above embodiments.
[0125] Figure 9 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 9The computer system 900 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0126] like Figure 9 As shown, the computer system 900 includes a Central Processing Unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 902 or programs loaded from Storage Unit 908 into Random Access Memory (RAM) 903, such as performing the methods described in the above embodiments. The RAM 903 also stores various programs and data required for system operation. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An Input / Output (I / O) interface 905 is also connected to the bus 904.
[0127] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. Removable media 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 910 as needed so that computer programs read from them can be installed into storage section 908 as needed.
[0128] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs various functions defined in the system of this application.
[0129] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.
[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0131] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.
[0132] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the method described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.
[0133] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various embodiments described above.
[0134] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.
Claims
1. A human-computer gesture interaction method, characterized in that, The human-computer gesture interaction method includes: A frame image to be processed is acquired, wherein at least one object to be identified is present in the frame image to be processed, and the frame image to be processed is acquired by an image acquisition device installed in the vehicle's infotainment system; The key points of the frame image to be processed are extracted according to the first preset algorithm to obtain multiple key points to be processed, and the frame image to be processed is segmented according to the second preset algorithm to obtain multiple masks to be processed. Based on each of the key points to be processed and each of the masks to be processed, the target recognition object is determined; wherein, the mask region to which each of the key points to be processed belongs and each of the masks to be processed are matched to obtain multiple initial matching combinations, and the overlap degree of each of the initial matching combinations is calculated to obtain multiple matching combinations to be processed. If there are multiple objects with unprocessed key points in each of the matching combinations to be processed, then the visibility confidence of each unprocessed key point is calculated, and the point with the highest visibility confidence is selected as the unprocessed key point of the unprocessed mask, so as to obtain the key point set corresponding to each unprocessed mask. Compare the number of key points in each mask to be processed, and determine the mask to be processed with the largest number of key points to be processed as the target mask. The three-dimensional coordinates of the key points to be processed within the target mask are transformed to obtain the three-dimensional coordinates of each key point to be processed within the target mask, and the target recognition object is determined based on the three-dimensional coordinates of each key point to be processed within the target mask. The pose of the target object is identified to determine the pose category of the target object in the frame image to be processed; If the gesture category is the interaction start gesture, then the interaction mode is activated and the gesture of the target object in the current frame image is identified. The current frame image is the frame image after the frame image to be processed. If the pose of the target object in the current frame image is identified as an interaction command pose, then a preset control command with a mapping relationship to the interaction command pose is determined based on the interaction command pose, and the preset control command is executed in the vehicle system.
2. The human-computer posture interaction method according to claim 1, characterized in that, Before controlling the activation of the interactive mode and recognizing the pose of the target object in the current frame image, the human-computer posture interaction method further includes: After obtaining the frame image to be processed, a preset number of start verification frame images are obtained, and the pose of the target object in each of the start verification frame images is identified. The preset number of start verification frame images are a preset number of consecutive frame images after the frame image to be processed. If the pose of the target object in each of the start verification frame images is the interaction start pose, then the interaction mode is activated and a voice broadcast is given with the first preset prompt tone.
3. The human-computer posture interaction method according to claim 1, characterized in that, The three-dimensional coordinate transformation of the key points to be processed within the target mask includes: Obtain the image size information of the frame image to be processed; Based on the image size information, the pixel coordinate ratio of the key points to be processed in the target mask is converted to determine the two-dimensional coordinate parameters to be converted for each key point to be processed in the target mask. The three-dimensional coordinates of each key point to be processed within the target mask are obtained by calculating the two-dimensional coordinate parameters to be converted based on the preset coordinate transformation algorithm.
4. The human-computer posture interaction method according to claim 1, characterized in that, After determining the target object based on the three-dimensional coordinates of each key point to be processed within the target mask, the human-computer posture interaction method further includes: Based on the three-dimensional coordinates of each key point to be processed within the target mask and the three-dimensional coordinates of each key point of a preset pose type, the maximum angular deviation value of each key point to be processed within the target mask is determined. If the maximum angular deviation value of each of the key points to be processed is less than or equal to the preset recognition deviation value, then the preset posture type is taken as the posture recognition result of the target object.
5. The human-computer posture interaction method according to claim 1, characterized in that, After recognizing the pose of the target object in the current frame image as an interaction command pose, the human-computer pose interaction method further includes: After acquiring the current frame image, a preset number of recognition and verification frame images are obtained, and the pose of the target object in each recognition and verification frame image is recognized. The preset number of recognition and verification frame images are a preset number of consecutive frame images after the current frame image. If the pose of the target object in each of the recognition verification frame images is an interactive instruction pose, then a preset control instruction that has a mapping relationship with the interactive instruction pose is determined according to the interactive instruction pose, and the preset control instruction is executed in the vehicle system, and voice broadcast is performed based on the preset control instruction.
6. The human-computer posture interaction method according to claim 1, characterized in that, Before determining the target object based on each of the key points to be processed and each of the masks to be processed, the human-computer posture interaction method further includes: The loss weights of the key points to be processed in each target box are calculated based on a preset key point loss function to reduce the reduction in recognition accuracy caused by the loss of key points in the frame image to be processed. The target box is the bounding box of each object to be recognized generated during the key point extraction process.
7. The human-computer posture interaction method according to claim 6, characterized in that, The loss weight calculation is based on the following preset key point loss function: Where n is the number of bounding boxes. It is the keypoint loss value of the i-th target box in the frame image to be processed. It is the total number of key points in the i-th bounding box. It is the total number of visible keypoints of the i-th target bounding box. It is the total loss value of key points within the frame image to be processed.
8. The human-computer posture interaction method according to any one of claims 1-7, characterized in that, After identifying the pose of the target object in the current frame image, the human-computer pose interaction method further includes: If the pose of the target object in the current frame image is identified as the start pose of the combined interaction command, and the recognition results of the preset number of consecutive frames after the current frame image are all the start pose of the combined interaction command, then a new frame image is acquired and the pose of the target object in each newly acquired frame image is identified. The pose recognition results of the target object in the newly acquired frame image are recorded sequentially until the pose of the target object in the newly acquired frame image is restored to the preset end pose. The pose recognition result of this combined interaction command is obtained, and the combined interaction command pose recognition of the target object in the current frame image is ended. The combined interaction command gesture recognition result is compared with the preset combined interaction command gesture. If there is a preset combined interaction command gesture that matches the combined interaction command gesture recognition result, a preset control command that has a mapping relationship with the preset combined interaction command gesture is determined, and the preset control command is executed in the vehicle system.
9. The human-computer posture interaction method according to any one of claims 1-7, characterized in that, After identifying the pose of the target object in the current frame image, the human-computer pose interaction method further includes: If the pose of the target object in the current frame image is identified as the interaction end pose, the interaction mode is turned off and a voice broadcast is given with a second preset prompt tone.
10. A human-computer posture interaction system, characterized in that, The human-computer gesture interaction system includes: An image acquisition module is used to acquire a frame image to be processed, wherein there is at least one object to be identified in the frame image to be processed, and the frame image to be processed is acquired by an image acquisition device installed in the vehicle system; The pose recognition module is used to extract key points from the frame image to be processed according to a first preset algorithm to obtain multiple key points to be processed, and to perform instance segmentation on the frame image to be processed according to a second preset algorithm to obtain multiple masks to be processed; based on each key point to be processed and each mask to be processed, a target recognition object is determined; wherein, the mask region to which each key point to be processed belongs and each mask to be processed are matched to obtain multiple initial matching combinations, and the overlap of each initial matching combination is calculated to obtain multiple matching combinations to be processed; if there are multiple key points to be processed in the mask to be processed in each matching combination to be processed, the visibility confidence of each key point to be processed is calculated, and the point with the highest visibility confidence is selected as the key point to be processed. The key points to be processed in the mask are obtained, and the key point set corresponding to each mask to be processed is obtained; the number of key point sets in each mask to be processed is compared, and the mask to be processed with the largest number of key points to be processed in the key point set is determined as the target mask; the key points to be processed in the target mask are transformed into three-dimensional coordinates to obtain the three-dimensional coordinates of each key point to be processed in the target mask, and the target recognition object is determined based on the three-dimensional coordinates of each key point to be processed in the target mask; the pose of the target recognition object is recognized to determine the pose category of the target recognition object in the frame image to be processed; if the pose category is the interaction start pose, the interaction mode is activated, and the pose of the target recognition object in the current frame image is recognized, wherein the current frame image is the frame image after the frame image to be processed; The instruction management module is used to determine a preset control instruction that has a mapping relationship with the interaction instruction posture if the posture of the target object in the current frame image is identified as an interaction instruction posture. The controller management module is used to execute the preset control commands on the vehicle's infotainment system.
11. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the human-computer gesture interaction method as described in any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by the computer's processor, causes the computer to perform the human-computer gesture interaction method as described in any one of claims 1 to 9.
Citation Information
Patent Citations
Gesture recognition control method and device, electronic equipment and storage medium
CN112686169A
Automobile man-machine interaction method, device and equipment
CN114373193A
Multi-human-body-posture estimation detection and segmentation optimization method and device based on instance segmentation
CN114708647A
