Vehicle screen control method and device, electronic equipment and storage medium

Abstract

The invention relates to a vehicle screen control method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining at least one frame of gesture image and vehicle driving parameters associated with each frame of gesture image; acquiring a gesture state vector corresponding to each frame of gesture image and a driving state vector of the associated vehicle driving parameters; inputting the gesture state vector corresponding to each frame of gesture image and the associated driving state vector into a pre-trained gesture recognition model for gesture recognition to obtain a gesture category corresponding to each frame of gesture image output by the gesture recognition model; and controlling the vehicle screen based on the gesture category corresponding to the at least one frame of gesture image. According to the scheme, gesture recognition is carried out by combining the multi-modal data of the gesture state vector of the gesture image and the driving state vector of the vehicle driving parameter, so that the influence of the vehicle motion state on the user gesture is considered during gesture recognition, and the accuracy and robustness of gesture recognition in the vehicle motion state can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of screen control technology, and in particular to a vehicle screen control method, device, electronic device and storage medium. Background Technology

[0002] With the rapid development of intelligent vehicle technology and computer vision technology, it has become possible to control vehicle screens through gesture recognition.

[0003] Currently, related technologies involve capturing user gesture images using in-vehicle cameras, performing gesture recognition on these images to determine the user's gestures, and then controlling the vehicle screen based on the recognized gestures. However, since the vehicle's movement during operation can affect the user's gestures, the aforementioned method of recognizing gestures based on a single gesture image suffers from low recognition accuracy, thus impacting the control effectiveness of the vehicle screen. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, at least one embodiment of the present disclosure provides a vehicle screen control method, device, electronic device and storage medium.

[0005] In a first aspect, this disclosure provides a vehicle screen control method, including: Obtain at least one frame of gesture image and vehicle driving parameters associated with each frame of gesture image; Obtain the gesture state vector corresponding to each frame of the gesture image and the driving state vector of the associated vehicle driving parameters; The gesture state vector and the associated driving state vector corresponding to each frame of the gesture image are input into a pre-trained gesture recognition model for gesture recognition, and the gesture category corresponding to each frame of the gesture image is obtained from the output of the gesture recognition model. The vehicle screen is controlled based on the gesture category corresponding to the at least one frame of gesture image.

[0006] Secondly, this disclosure provides a vehicle screen control device, comprising: The first acquisition module is used to acquire at least one frame of gesture image and vehicle driving parameters associated with each frame of gesture image; The second acquisition module is used to acquire the gesture state vector corresponding to each frame of the gesture image and the driving state vector of the associated vehicle driving parameters. The third acquisition module is used to input the gesture state vector corresponding to each frame of the gesture image and the associated driving state vector into a pre-trained gesture recognition model for gesture recognition, so as to obtain the gesture category corresponding to each frame of the gesture image output by the gesture recognition model. The control module is used to control the vehicle screen based on the gesture category corresponding to the at least one frame of gesture image.

[0007] Thirdly, this disclosure provides an electronic device, including: a processor and a memory; The processor executes the vehicle screen control method as described in the first aspect by calling programs or instructions stored in the memory.

[0008] Fourthly, this disclosure provides a computer-readable storage medium storing a program or instructions that cause a computer to perform the vehicle screen control method as described in the first aspect.

[0009] Fifthly, this disclosure provides a computer program product, including a computer program or instructions, which, when executed by a processor, are used to implement the vehicle screen control method as described in the first aspect.

[0010] The technical solution provided in this disclosure has at least the following advantages compared with the prior art: In this embodiment, at least one frame of gesture image and vehicle driving parameters associated with each frame of gesture image are acquired; the gesture state vector corresponding to each frame of gesture image and the driving state vector of the associated vehicle driving parameters are acquired; then, the gesture state vector corresponding to each frame of gesture image and the associated driving state vector are input into a pre-trained gesture recognition model for gesture recognition, obtaining the gesture category corresponding to each frame of gesture image output by the gesture recognition model, and then controlling the vehicle screen based on the gesture category corresponding to at least one frame of gesture image. By adopting the above technical solution, by acquiring gesture images and their associated vehicle driving parameters, and combining multimodal data such as the gesture state vector of the gesture image and the driving state vector of the vehicle driving parameters for gesture recognition, the influence of vehicle motion state on user gestures is considered during gesture recognition, which can improve the accuracy and robustness of gesture recognition under vehicle motion state, thereby improving the accuracy of vehicle screen control and enhancing the user experience. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1A schematic flowchart of a vehicle screen control method provided as an exemplary embodiment of this disclosure; Figure 2 A schematic flowchart of a vehicle screen control method provided as another exemplary embodiment of this disclosure; Figure 3 A schematic diagram of key hand points of an exemplary embodiment of the present disclosure is shown; Figure 4 A schematic diagram of the structure of a gesture recognition model according to an exemplary embodiment of the present disclosure is shown; Figure 5 This is a schematic diagram of the structure of a vehicle screen control device provided in an embodiment of the present disclosure. Detailed Implementation

[0014] To better understand the above-described objectives, features, and advantages of this disclosure, the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the described embodiments are only some, not all, of the embodiments of this disclosure. The specific embodiments described herein are merely for explaining this disclosure and not for limiting it. Unless otherwise specified, the embodiments of this disclosure and the features within them can be combined with each other. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure are within the scope of protection of this disclosure.

[0015] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0016] With the rapid development of intelligent vehicle technology, rear-seat entertainment systems have gradually become a key feature for enhancing user experience, and their experience has become a crucial indicator of vehicle comfort. Many models are now equipped with large rear-seat entertainment screens, and their ease of control directly impacts the overall passenger satisfaction, especially in scenarios such as long-distance travel and family trips, where efficient interaction methods can significantly improve the journey experience. However, current rear-seat entertainment screen control methods have significant limitations. For example, traditional physical remote controls are easily lost during multi-person use, and third-row passengers, due to the distance, need to stand up to pass the screen or bend down to reach it, which is inconvenient and prone to causing physical contact. Touchscreen controls require passengers to be close to the screen, but rear-seat passengers are often in a relaxed posture such as reclining or leaning back; frequent forward leaning can lead to fatigue, and the distance between the third row and the screen exacerbates this problem. Voice control accuracy drops significantly in noisy environments with multiple people conversing in the rear.

[0017] With the development of computer vision technology, controlling screens through gesture recognition has become possible. However, in the in-vehicle environment, the Occupant Monitor System (OMS) camera is located on the roof, shooting downwards. Different passengers have different positions, and their angles, directions, and hand positions when making gestures may vary. Furthermore, vehicle turning and bumps can also affect passenger movements. At the same time, gesture recognition needs to be real-time; even a slight delay can negatively impact the passenger experience. These issues present challenges for in-vehicle gesture recognition.

[0018] Currently, the main method of controlling the screen through gesture recognition in related technologies involves recognizing gestures from captured images and then controlling the screen based on the recognized gestures. This method, which relies on a single gesture image for recognition, struggles to guarantee accuracy when the vehicle is in motion, especially during steering or bumpy driving conditions.

[0019] To address the aforementioned issues, this disclosure provides an efficient, convenient, and accurate gesture control solution for the rear entertainment screen. By combining collected gesture images with vehicle driving parameters for gesture recognition, it can improve the accuracy and robustness of gesture recognition under different user postures, positions, and vehicle movement states. It not only solves the problems of physical remote controls being easily lost and inconvenient operation for third-row passengers, touch operation requiring passengers to be close to the screen causing fatigue and the problem being more pronounced in the third row due to distance, and voice control having low recognition accuracy in noisy rear environments, but also solves the problem of low gesture recognition accuracy caused by inconsistencies in gesture perspective, direction, left and right hand positions, and tilt due to factors such as the overhead shooting angle of the OMS camera, differences in passenger positions, and vehicle steering and bumps in the in-vehicle environment. It also overcomes challenges such as insufficient real-time performance of gesture recognition (delay affects the experience).

[0020] The specific embodiments of the vehicle screen control method, device, electronic equipment, and storage medium disclosed herein are explained in detail below with reference to the accompanying drawings.

[0021] Figure 1 This is a flowchart illustrating a vehicle screen control method provided in an exemplary embodiment of the present disclosure. The method can be executed by a vehicle screen control device provided in the embodiments of the present disclosure. The vehicle screen control device can be implemented in software and / or hardware and can be integrated into an electronic device. The electronic device can be deployed in a vehicle for controlling the vehicle screen.

[0022] like Figure 1 As shown, the vehicle screen control method may include the following steps: Step 101: Obtain at least one frame of gesture image and the vehicle driving parameters associated with each frame of gesture image.

[0023] At least one frame of the gesture image is an image captured by a camera in the vehicle, such as an OMS camera. Each time the camera captures a frame, it uploads the current frame to the electronic device. The electronic device then obtains at least one frame (called the gesture image) including the current frame and acquires the vehicle driving parameters associated with each gesture image.

[0024] During vehicle operation, the vehicle's acceleration, bump frequency, and speed all affect its stability, thus causing a certain degree of deviation in hand gestures. To ensure the accuracy of hand gesture recognition while the vehicle is in motion, this embodiment not only acquires the hand gesture image but also acquires the vehicle's driving parameters associated with the gesture image. These driving parameters include at least one of longitudinal acceleration, lateral acceleration, vertical acceleration, bump frequency, and vehicle speed. These parameters are all dynamic state parameters during vehicle operation and can be acquired using the vehicle's existing control modules and sensors without requiring additional hardware. Therefore, this solution does not increase hardware costs. For example, an inertial measurement unit (IMU) can be used to acquire longitudinal, lateral, and vertical acceleration; a vehicle speed sensor can be used to acquire the vehicle's speed; and a vibration acceleration sensor can be used to determine the vehicle's bump frequency based on the vibration acceleration signal.

[0025] It is understandable that, due to the difference in the frequency of image acquisition by the camera and the frequency of vehicle driving parameters acquisition by the sensor, the vehicle driving parameters associated with different gesture images may be the same or different. When obtaining the vehicle driving parameters associated with each frame of gesture image, the associated vehicle driving parameters can be determined based on the timestamp of the gesture image. For example, for a frame of gesture image with timestamp t1, if the sensor does not report the acquired vehicle driving parameters at time t1, then the set of vehicle driving parameters closest to time t1 from the historical vehicle driving parameters is used as the vehicle driving parameters associated with that frame of gesture image; if the sensor reports vehicle driving parameters at time t1, then the vehicle driving parameters at time t1 are associated with that frame of gesture image.

[0026] Step 102: Obtain the gesture state vector corresponding to each frame of gesture image and the driving state vector of the associated vehicle driving parameters.

[0027] In this embodiment, for each frame of gesture image and its associated vehicle driving parameters, the gesture state vector corresponding to each frame of gesture image can be further obtained, as well as the driving state vector of the vehicle driving parameters associated with that frame of gesture image, wherein the driving state vector is determined according to the vehicle driving parameters.

[0028] As an example, when obtaining the gesture state vector corresponding to each frame of gesture image, each frame of gesture image can be input into a pre-trained gesture state detection model. The gesture state detection model will then perform gesture state detection on the input gesture image and output the corresponding gesture state vector.

[0029] As an example, when obtaining the driving state vector of vehicle driving parameters associated with each frame of gesture image, the vehicle driving parameters can be concatenated according to a preset concatenation order to obtain the corresponding driving state vector. For example, suppose the obtained vehicle driving parameters include longitudinal acceleration (denoted as a). x ), lateral acceleration (denoted as a) y Vertical acceleration (denoted as a) z ), bump frequency (denoted as f) and vehicle speed (denoted as v) c The driving state vector is denoted as X. e Then X e =(a x , a y , a z , f, v c ).

[0030] It is understandable that vehicle driving parameters are associated with gesture images, and the resulting driving state vector is also associated with gesture images.

[0031] Step 103: Input the gesture state vector and associated driving state vector corresponding to each frame of gesture image into the pre-trained gesture recognition model for gesture recognition, and obtain the gesture category corresponding to each frame of gesture image output by the gesture recognition model.

[0032] In this embodiment, after obtaining the gesture state vector and associated driving state vector corresponding to each frame of gesture image, the gesture state vector and driving state vector corresponding to the same gesture image can be sequentially input into the pre-trained gesture recognition model for each frame of gesture image to perform gesture recognition. The gesture recognition model then performs gesture prediction based on the input gesture state vector and driving state vector and outputs the recognized gesture category.

[0033] The gesture categories can be set according to actual needs. For example, gesture categories can include thumbs up, thumbs down, thumbs left, thumbs right, and the OK gesture. Different gesture categories correspond to different screen control operations, and the same gesture category can correspond to different screen control operations in different applications. For example, in multimedia applications, the thumbs up gesture is used to increase the volume, the thumbs right gesture is used to switch to the next playback content, and the OK gesture is used to control the playback content to switch from play to pause and from pause to play, and so on. In game applications, the thumbs up gesture is used to control the game character to move forward, the thumbs right gesture is used to control the game character to move right, and the OK gesture is used to control the game character to perform combat actions, and so on.

[0034] The gesture recognition model in this embodiment is pre-trained. It collects sample images of passengers making gestures to the screen and associated vehicle driving parameters. It obtains a set of training samples, including a gesture state vector corresponding to a sample image and a driving state vector corresponding to the vehicle driving parameters. The training samples are then labeled with gesture categories. The labeled training samples are used to train the initial model. Training is complete when the number of iterations reaches a threshold or the model converges (loss value is less than a preset value), resulting in a trained gesture recognition model. When collecting sample images, some images without gestures are collected as negative samples. When labeling the training samples, not only are positive samples containing gestures labeled with the gesture type, but negative samples are also labeled with the category of "no gesture" to reduce the false detection rate. Thus, the trained gesture recognition model can not only output the recognized gesture category but also output the recognition result of "no gesture" for images without gestures.

[0035] It is understood that in this embodiment, the input data for each input gesture recognition model is the gesture state vector and the associated driving state vector corresponding to the same gesture image. When there are multiple frames of gesture images, it is necessary to input the gesture recognition model multiple times to perform gesture recognition in order to obtain the gesture category corresponding to each frame of gesture images.

[0036] Step 104: Control the vehicle screen based on the gesture category corresponding to at least one frame of gesture image.

[0037] In this embodiment, after obtaining the gesture category corresponding to each gesture image in at least one frame of gesture images, the vehicle screen can be controlled according to the gesture category corresponding to at least one frame of gesture images.

[0038] As an example, at least one gesture image frame includes a single frame image, which is the current frame image. That is, after the camera captures one frame image (the current frame image), the electronic device performs gesture recognition based on the current frame image uploaded by the camera each time and the associated vehicle driving parameters to obtain the gesture category corresponding to the current frame image, and then performs vehicle screen control based on the gesture category. If the gesture category corresponding to the current frame is one of the preset gesture categories, the electronic device performs control operations on the vehicle screen according to the control command corresponding to the recognized gesture category; if the gesture recognition model outputs a recognition result of no gesture, then when the camera captures the next frame image, the electronic device continues to perform gesture recognition based on the next frame image and the associated vehicle driving parameters.

[0039] As an example, at least one frame of gesture image includes a preset number of consecutive images. That is, the number of acquired gesture images is the preset number of frames, and these preset number of gesture images form a consecutive image. The specific value of the preset number of frames can be set according to actual needs. For example, setting the preset number of frames to 25 frames means that the acquired 25 frames of gesture images are consecutive. When controlling the vehicle screen based on the gesture category corresponding to at least one frame of gesture image, the number of times the gesture category corresponding to the preset number of consecutive images appears can be obtained based on the gesture category of each frame of gesture image. That is, the number of times each gesture category appears in the gesture category corresponding to the consecutive images. Based on the number of times each gesture category appears, the target gesture category corresponding to the preset number of consecutive images is determined. The number of times the target gesture category appears is greater than a threshold value. The specific value of the threshold value can be set according to actual needs. The threshold value should be less than the preset number of frames but not less than half of the preset number of frames to avoid situations where the number of times different gesture categories appear all reach the threshold value. For example, if the preset number of frames is 25, the threshold value can be set to 15, and then the vehicle screen can be controlled based on the target gesture category. In other words, in this embodiment, after obtaining the gesture categories corresponding to a preset number of consecutive images, the frequency of each gesture category in the preset number of consecutive images is counted. If the frequency of a certain gesture category reaches a threshold, then that gesture category is determined as the target gesture category, and the vehicle screen is controlled based on the control command corresponding to the target gesture category. By determining the gesture category whose frequency reaches the threshold as the target gesture category and controlling the vehicle screen based on the target gesture category, the problem of misoperation caused by screen control based on a single frame of gesture images can be avoided, ensuring the accuracy of screen control.

[0040] In one optional embodiment of this disclosure, for vehicles that include multiple screens, the target screen associated with the camera that captures gesture images can be determined based on the association between different screens and cameras, and then the target screen can be controlled according to the recognized gesture category.

[0041] In practical applications, for scenarios where at least one gesture image is a continuous image of a preset number of frames, in order to ensure the real-time performance of gesture control of the screen, the electronic device performs gesture recognition on the current frame image after acquiring the current frame image uploaded by the camera, and obtains the gesture category of the previously recognized historical frame images. If the gesture category of a continuous image of a preset number of frames (including the current frame image) can be obtained, the vehicle screen is controlled by determining the target gesture category. If the number of recognized images is less than the preset number of frames, the device continues to wait for the next frame image for gesture recognition until the preset number of frames is met, and then controls the vehicle screen by determining the target gesture category. If the target gesture category cannot be determined from the gesture recognition results of a continuous image of a preset number of frames (the number of times each gesture category appears is less than the number of times thresholded), the device also waits for the next frame image for gesture recognition, and determines the target gesture category and controls the screen based on the latest image to determine a continuous image of a preset number of frames. In other words, a sliding window technique can be used to determine a preset number of consecutive images. The window length is fixed at the preset number of frames. Positioning is based on the current frame image. Each time a new frame arrives, the window content slides to the right, adding the new frame to the end of the frame. The oldest frame in the window is discarded, maintaining the total number of frames at the preset number. Gesture categories for previously recognized frames can be cached. When a new frame arrives, gesture recognition is performed on the new frame, and the recognition results of the previous frame are directly retrieved from the cache and combined with the recognition results of the new frame to determine the target gesture category for vehicle screen control. This reduces the repeated recognition of the same gesture images, reduces gesture recognition time, solves the gesture recognition latency problem, and ensures fast operation response.

[0042] The vehicle screen control method of this disclosure acquires at least one frame of gesture image and vehicle driving parameters associated with each frame of gesture image; acquires the gesture state vector corresponding to each frame of gesture image and the driving state vector of the associated vehicle driving parameters; then, inputs the gesture state vector corresponding to each frame of gesture image and the associated driving state vector into a pre-trained gesture recognition model for gesture recognition, obtaining the gesture category corresponding to each frame of gesture image output by the gesture recognition model, and then controls the vehicle screen based on the gesture category corresponding to at least one frame of gesture image. By adopting the above technical solution, by acquiring gesture images and their associated vehicle driving parameters, and combining multimodal data such as the gesture state vector of the gesture image and the driving state vector of the vehicle driving parameters for gesture recognition, the influence of vehicle motion state on user gestures is considered during gesture recognition, which can improve the accuracy and robustness of gesture recognition under vehicle motion state, thereby improving the accuracy of vehicle screen control and enhancing the user experience.

[0043] Because the feature area of ​​the hand keypoint region is small in in-vehicle scenarios, scaling the integrated keypoint detection model to a fixed size may result in the loss of input features, leading to poor performance. Conversely, directly using high-resolution images for detection would significantly increase computational load. To address this issue, in this embodiment, when acquiring the gesture state vector of each frame of the gesture image, the detection of the hand region and the recognition of hand keypoints are performed separately. First, the hand is detected and the hand region bounding box is predicted. Then, the hand region bounding box is cropped from the original image for keypoint regression. This avoids the poor performance caused by directly performing keypoint regression at a low resolution, ensuring the accuracy of hand detection without increasing computational overhead. Therefore, in one optional embodiment of this disclosure, as... Figure 2 As shown, based on the aforementioned embodiment, step 102, obtaining the gesture state vector corresponding to each frame of gesture image, may include the following sub-steps: Step 201: Input each frame of gesture image into the pre-trained hand detection model to perform hand detection, and obtain the hand region location information and left / right hand category labels corresponding to each frame of gesture image output by the hand detection model.

[0044] The hand detection model can be trained using an object detection model. For example, an initial model can be built based on the YOLO11n model, and sample images can be collected to annotate hand detection boxes and left / right hand categories. Then, the annotated data is used to train the initial model to obtain the hand detection model. The YOLO11n model is a lightweight object detection model in the YOLO11 series, but it cannot distinguish between left and right hands. Therefore, in this embodiment, a binary classification fully connected layer is added after the last feature map layer of the YOLO11n model. This branch is responsible for predicting the left and right hand categories. The trained hand detection model can then output the hand region location information and left / right hand category labels in the gesture image. Using a lightweight object detection model to train the hand detection model can improve the speed of hand region detection to a certain extent, thereby shortening the overall gesture recognition time and ensuring fast operation response.

[0045] For example, the hand region location information can be represented as the coordinates of the four vertices of the hand detection box, or as the coordinates of the top-left and bottom-right vertices of the hand detection box. Left and right hand category identifiers can be represented by preset identifiers, such as using the number "0" to represent the left hand and the number "1" to represent the right hand.

[0046] In this embodiment, for each acquired gesture image, the gesture image can be input into a trained hand detection model. The hand detection model performs hand detection on the input gesture image and outputs the hand region location information and left / right hand category identifiers corresponding to the input gesture image.

[0047] It is understandable that each time a frame of gesture image is input into the hand detection model, the hand detection model performs hand detection on the frame of gesture image and outputs the corresponding hand region location information and left and right hand category labels.

[0048] For example, when performing hand detection, the gesture image can be scaled up to a preset resolution. For instance, since a smaller resolution has less impact on the accuracy of hand detection, the gesture image can be scaled down to a smaller size, such as 320*192, to reduce the computational load of hand detection.

[0049] Since the key points on the left and right hands are distributed in different locations, in order to ensure the accuracy of the recognition results, in this embodiment, the left and right hand category identifiers are detected simultaneously when performing hand detection, and the left and right hand category identifiers are used as part of the gesture state vector, so that the difference between the left and right hands is taken into account when performing gesture recognition.

[0050] Step 202: Determine the hand region image from each frame of gesture image based on the hand region location information.

[0051] In this embodiment, after obtaining the hand region position information of each frame of gesture image, the hand region image can be cropped from the corresponding gesture image based on the hand region position information, thereby obtaining the hand region image corresponding to each frame of gesture image. For example, if the hand region position information of gesture image A is position 1 and the hand region position information of gesture image B is position 2, then the hand region image of gesture image A can be cropped from gesture image A based on position 1, and the hand region image of gesture image B can be cropped from gesture image B based on position 2.

[0052] Step 203: Input the hand region image into the pre-trained keypoint detection model to detect hand keypoints, and obtain the hand keypoint information of each frame of gesture image output by the keypoint detection model.

[0053] The keypoint detection model can be constructed and trained using a lightweight backbone network and a fully connected layer; for example, the lightweight MobileNetV4 backbone network can be used. When training the keypoint detection model, the training samples used to train the hand detection model can be utilized. After training the hand detection model, the detected hand bounding box regions are directly cropped from the training samples to obtain sample hand images, and keypoints are annotated. The keypoint detection model is then trained using these annotated hand images. Once trained, the keypoint detection model can be used to detect hand keypoints in the input hand region image and output hand keypoint information.

[0054] The key information for the hand includes key point identifiers and key point coordinates. The key points for the hand include wrist key points and finger key points.

[0055] Figure 3 A schematic diagram of key points of the hand in an exemplary embodiment of this disclosure is shown, taking the left hand as an example, as follows: Figure 3 As shown, the hand includes 21 key points, namely 0-wrist, 1-thumb metacarpophalangeal joint, 2-thumb proximal interphalangeal joint, 3-thumb distal interphalangeal joint, 4-thumb tip, 5-index finger metacarpophalangeal joint, 6-index finger proximal interphalangeal joint, 7-index finger distal interphalangeal joint, 8-index finger tip, 9-middle finger metacarpophalangeal joint, 10-middle finger proximal interphalangeal joint, 11-middle finger distal interphalangeal joint, 12-middle finger tip, 13-ring finger metacarpophalangeal joint, 14-ring finger proximal interphalangeal joint, 15-ring finger distal interphalangeal joint, 16-ring finger tip, 17-little finger metacarpophalangeal joint, 18-little finger proximal interphalangeal joint, 19-little finger distal interphalangeal joint, 20-little finger tip.

[0056] In this embodiment, the hand region image cropped from each frame of the gesture image is input into the key point detection model to detect hand key points. The key point detection model outputs the identified hand key point information, thereby obtaining the hand key point information corresponding to each frame of the gesture image.

[0057] For example, the hand keypoint information output by the keypoint detection model can be represented in the following form: .

[0058] in, H n Indicates the first n Hand key point information in frame gesture images. Indicates the first n Key points in the frame gesture image are marked as m The key coordinates of the key points.

[0059] Step 204: Determine the gesture state vector of each frame of gesture image based on the left and right hand category identifiers and hand key point information.

[0060] In this embodiment, after obtaining the hand key point information corresponding to each frame of gesture image, the gesture state vector corresponding to the frame of gesture image can be determined based on the left and right hand category identifiers and hand key point information corresponding to the same gesture image, thereby obtaining the gesture state vector of each frame of gesture image.

[0061] As an example, the left and right hand category identifiers and hand key point information of the same gesture image can be concatenated to obtain the gesture state vector of the gesture image.

[0062] Since hand gestures are based on skeletal movements, relying solely on hand key points to teach the model the type of gesture, especially in complex in-vehicle perspectives (e.g., different passenger positions, left and right hands, different gesture directions; sometimes the palm faces the camera, sometimes the back of the hand faces the camera, sometimes the hand is raised, sometimes the hand is on the armrest), the model would have difficulty converging and the recognition effect would be poor. Therefore, this embodiment proposes a method of training the model using skeletal vectors. By learning the movement direction of each finger, the model can improve its recognition effect. Thus, in this embodiment, skeletal vectors can be calculated based on the identified hand key point information.

[0063] Therefore, in one optional embodiment of this disclosure, when determining the gesture state vector of each frame of gesture image based on the left and right hand category identifiers and hand key point information, for each frame of gesture image, the wrist key point coordinates and the finger key point coordinates corresponding to each finger can be determined according to the key point identifier of each hand key point in the hand key point information. The finger key point coordinates include the metacarpophalangeal joint key point coordinates, for example, in Figure 3 In the hand keypoint diagram shown, keypoint 0 corresponds to the wrist keypoint coordinates, keypoints 1-20 correspond to the finger keypoint coordinates, and keypoints 1, 5, 9, 13, and 17 correspond to the metacarpophalangeal joint keypoint coordinates. Next, based on the wrist keypoint coordinates and the metacarpophalangeal joint keypoint coordinates for each finger, the first skeletal vector from the wrist to the metacarpophalangeal joint is determined. Based on the finger keypoint coordinates for each finger, the second skeletal vector between adjacent finger keypoints on each finger is determined. Continuing with... Figure 3 Taking the schematic diagram shown as an example, the first skeleton vector includes and The second skeletal vector includes , and Next, the left and right hand category identifiers, the first bone vector, and the second bone vector of the same gesture image are concatenated to obtain the gesture state vector of each frame of the gesture image. That is, in this embodiment, for a frame of gesture image, the first bone vector and the second bone vector are determined based on the key hand points of the gesture image in that frame, and then the left and right hand category identifiers of the gesture image in that frame are concatenated to obtain the gesture state vector of that frame of the gesture image. In this way, the gesture state vector corresponding to each frame of the gesture image can be obtained.

[0064] The vehicle screen control method of this disclosure first inputs each frame of gesture image into a hand detection model to perform hand detection, obtaining the hand region location information and left / right hand category identifiers corresponding to each frame of gesture image. Then, based on the hand region location information, the hand region image is determined from each frame of gesture image. The hand region image is then input into a keypoint detection model to perform hand keypoint detection, obtaining the hand keypoint information of each frame of gesture image. Finally, based on the left / right hand category identifiers and hand keypoint information of each frame of gesture image, the gesture state vector of each frame of gesture image is determined. This separates hand region detection and keypoint recognition, which avoids the poor performance caused by directly performing keypoint regression on low-resolution images and ensures the accuracy of keypoint recognition.

[0065] In one optional embodiment of this disclosure, the gesture recognition model includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, an attention layer, and a fully connected layer. When inputting the gesture state vector and associated driving state vector corresponding to each frame of gesture image into the gesture recognition model to obtain the gesture category corresponding to each frame of gesture image, for each frame of gesture image, the gesture state vector corresponding to the same frame of gesture image can be input into the first feature extraction layer for feature extraction to obtain gesture features, and the driving state vector can be input into the second feature extraction layer for feature extraction to obtain driving features; then, the fused feature of the obtained gesture features and driving features is input into the third feature extraction layer for feature extraction to obtain multimodal features, wherein the fused feature can be obtained by adding the gesture features and driving features; next, the gesture features are used as key-value pairs and the multimodal features are used as queries to input attention features into the attention layer to obtain target attention features; finally, the target attention features are input into the fully connected layer for gesture recognition to obtain the gesture category.

[0066] The network structures of the first feature extraction layer, the second feature extraction layer, and the third feature extraction layer can be the same or different, and this disclosure does not impose any restrictions on this.

[0067] In this embodiment, for any gesture image, the gesture state vector of the gesture image is input into the first feature extraction layer to obtain gesture features, the driving state vector associated with the gesture image is input into the second feature extraction layer to extract features to obtain driving features, the driving features and gesture features corresponding to the gesture image are fused to obtain fused features, the fused features are input into the third feature extraction layer to extract features to obtain multimodal features of the gesture image, and finally, the gesture features of the gesture image frame are used as key-value pairs, and the multimodal features of the gesture image frame are used as query Q. Query Q, key K and value V are the three main inputs in the attention mechanism. The gesture features and multimodal features corresponding to the same gesture image are input into the attention layer to extract attention features to obtain target attention features. The fully connected layer performs gesture recognition based on the target attention features and outputs the gesture category of the gesture image.

[0068] Figure 4 A schematic diagram of the structure of a gesture recognition model according to an exemplary embodiment of the present disclosure is shown, as follows: Figure 4 As shown, the gesture recognition model includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, an attention layer, and a fully connected layer. The first, second, and third feature extraction layers have the same structure, each including a feed-forward neural network (FFN) layer and a normalization layer. The first feature extraction layer processes the gesture state vector (denoted as...) X v Feature extraction is performed on the driving state vector (denoted as ) and the second feature extraction layer is used to process the driving state vector. X e After feature extraction, X v The gesture features obtained from feature extraction are used as keys K and values ​​V (K=V) in one path for the attention layer, and as input to the input field in the other path. X e The driving features obtained from feature extraction are fused and further encoded by a third feature extraction layer. The resulting multimodal features serve as the query Q for the attention layer. The target attention features extracted by the attention layer are input into a fully connected (FC) layer for gesture category prediction, and the recognized gesture category is output. By inputting the gesture state vector and driving state vector of a gesture image frame into this gesture recognition model, the gesture category corresponding to that frame can be obtained, thus obtaining the gesture category of each frame of gesture images in at least one frame.

[0069] In this embodiment, feature extraction is performed on the gesture state vector and the associated driving state vector respectively, and the extracted features are fused and then extracted to obtain multimodal features. Based on the multimodal features and gesture features, an attention mechanism is used to extract more attentional features of the target, thereby identifying the gesture category, which can ensure the accuracy and robustness of gesture recognition.

[0070] To avoid unnecessary waste of computing resources caused by gesture recognition, in one optional embodiment of this disclosure, before acquiring gesture images and vehicle driving parameters, it can be first detected whether the control mode of the vehicle screen supports gesture control mode. If it is determined that gesture control mode is supported, it is further detected whether the gesture control mode is enabled. If the gesture control mode is enabled, at least one frame of gesture image and the vehicle driving parameters associated with each frame of gesture image are acquired. If the gesture control mode is disabled, gesture images and vehicle driving parameters are not acquired.

[0071] In one specific implementation of this disclosure, when the user activates the gesture control mode on the vehicle screen, the hand detection model continuously detects gesture images captured by the camera. When a hand region is detected, the coordinates of the passenger's hand region and left / right hand identifiers are returned. After the electronic device crops the hand region image based on the hand region detection coordinates, it uses a keypoint detection model to perform keypoint regression on the hand region image to obtain the keypoint coordinates. Then, it calculates the gesture state vector and obtains the associated driving state vector. The driving state vector and gesture state vector are then input into the gesture recognition model to obtain the gesture category. If more than 15 out of 25 consecutive frames are of the same gesture category, then that gesture category is determined as the current gesture category. If the current gesture category is determined to be a preset category such as up, down, left, right, or confirm, the electronic device controls the screen according to the current gesture category. If the gesture recognition model outputs that no gesture was performed, the hand region detection continues in the next frame.

[0072] In summary, the vehicle screen control solution provided in this disclosure has the following advantages: (1) Break through the limitations of traditional control: get rid of the dependence on physical remote control, avoid the trouble of loss, inconvenience of transmission or getting up to operate; passengers do not need to get close to the screen to touch, so that passengers in the back row who are in a relaxed posture such as reclining or leaning to the side (especially the third row) do not need to lean forward frequently, greatly reducing operating fatigue.

[0073] (2) Adapt to complex in-vehicle scenarios: Optimize the OMS camera overhead view and the inconsistent gesture view, direction and left and right hands caused by the difference in passenger position. Significantly improve the robustness and accuracy of gesture recognition under different postures, positions and vehicle movement states; and is not affected by noisy environments such as multiple people talking in the back row. Compared with voice control, the interaction stability is stronger.

[0074] (3) Optimize the interactive experience: The delay of gesture recognition has been resolved, ensuring rapid operation response. At the same time, the recognition effect has been optimized, allowing rear passengers to operate the entertainment system naturally and efficiently in scenarios such as long-distance travel and family outings, further improving the comfort and satisfaction of the ride.

[0075] To implement the above embodiments, this disclosure also provides a vehicle screen control device, which can be implemented in software and / or hardware and can be integrated into an electronic device.

[0076] Figure 5 This is a schematic diagram of the structure of a vehicle screen control device provided in an embodiment of the present disclosure, as shown below. Figure 5 As shown, the vehicle screen control device 30 may include: a first acquisition module 310, a second acquisition module 320, a third acquisition module 330, and a control module 340.

[0077] The first acquisition module 310 is used to acquire at least one frame of gesture image and vehicle driving parameters associated with each frame of gesture image. The second acquisition module 320 is used to acquire the gesture state vector corresponding to each frame of gesture image and the driving state vector of the associated vehicle driving parameters. The third acquisition module 330 is used to input the gesture state vector and the associated driving state vector corresponding to each frame of gesture image into the pre-trained gesture recognition model for gesture recognition, and obtain the gesture category corresponding to each frame of gesture image output by the gesture recognition model. The control module 340 is used to control the vehicle screen based on the gesture category corresponding to at least one frame of gesture image.

[0078] Optionally, the second acquisition module 320 includes: The first detection unit is used to input each frame of gesture image into a pre-trained hand detection model for hand detection, and obtain the hand region location information and left / right hand category labels corresponding to each frame of gesture image output by the hand detection model. The first determining unit is used to determine the hand region image from each frame of gesture image based on the hand region position information; The second detection unit is used to input the hand region image into the pre-trained keypoint detection model to detect hand keypoints and obtain the hand keypoint information of each frame of gesture image output by the keypoint detection model. The second determining unit is used to determine the gesture state vector of each frame of gesture image based on the left and right hand category identifiers and hand key point information of each frame of gesture image.

[0079] Further optionally, the second determining unit is also used for: For each frame of gesture image, based on the key point identifier of each hand key point in the hand key point information, determine the wrist key point coordinates and the finger key point coordinates corresponding to each finger in the hand key point information. The finger key point coordinates include the metacarpophalangeal joint key point coordinates. Based on the coordinates of the wrist key points and the coordinates of the key points of the metacarpophalangeal joints corresponding to each finger, the first skeletal vector from the wrist to the metacarpophalangeal joint is determined. Based on the coordinates of the key points of each finger, determine the second bone vector between adjacent key points on each finger; By concatenating the left and right hand category identifiers, the first bone vector, and the second bone vector of the same gesture image, the gesture state vector of each frame of the gesture image is obtained.

[0080] Optionally, the gesture recognition model includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, an attention layer, and a fully connected layer; the third acquisition module 330 is further used for: For each frame of gesture image, the gesture state vector is input into the first feature extraction layer for feature extraction to obtain gesture features; The driving state vector is input into the second feature extraction layer for feature extraction to obtain driving features; The fused features of gesture features and driving features are input into the third feature extraction layer for feature extraction to obtain multimodal features; The attention layer extracts the target attention features by using gesture features as key-value pairs and multimodal features as query input. The target attention features are input into a fully connected layer for gesture recognition to obtain the gesture category.

[0081] Optionally, the number of gesture images is a preset number of frames, and the preset number of gesture images form a continuous image; the control module 340 is also used for: Get the number of times the gesture category appears in consecutive images of a preset number of frames; Based on the frequency of each gesture category, determine the target gesture category corresponding to a preset number of consecutive images; Control the vehicle screen based on the target gesture category.

[0082] Optionally, the vehicle driving parameters include at least one of the following: longitudinal acceleration, lateral acceleration, vertical acceleration, bump frequency, and vehicle speed.

[0083] Optionally, the first acquisition module 310 is also used for: Check if the vehicle screen's control mode supports gesture control. In response to support for gesture control mode, detect whether the gesture control mode is enabled; In response to the gesture control mode being enabled, at least one frame of gesture image and vehicle driving parameters associated with each frame of gesture image are acquired.

[0084] The vehicle screen control device applicable to electronic devices provided in this disclosure can execute the vehicle screen control method provided in this disclosure, and has the corresponding functional modules and beneficial effects for executing the method. Content not described in detail in the device embodiments of this disclosure can be referred to the description in any method embodiment of this disclosure.

[0085] This disclosure also provides an electronic device, including a processor and a memory; the processor executes the steps of the aforementioned vehicle screen control method embodiments by calling programs or instructions stored in the memory. To avoid repetition, these steps will not be repeated here.

[0086] This disclosure also provides a computer-readable storage medium that is non-transitory and stores a program or instructions that cause a computer to perform the steps of the aforementioned vehicle screen control method embodiments. To avoid repetition, these steps will not be repeated here.

[0087] This disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, are used to implement the steps of the aforementioned vehicle screen control method embodiments.

[0088] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0089] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vehicle screen control method, characterized in that, The method includes: Obtain at least one frame of gesture image and vehicle driving parameters associated with each frame of gesture image; Obtain the gesture state vector corresponding to each frame of the gesture image and the driving state vector of the associated vehicle driving parameters; The gesture state vector and the associated driving state vector corresponding to each frame of the gesture image are input into a pre-trained gesture recognition model for gesture recognition, and the gesture category corresponding to each frame of the gesture image is obtained from the output of the gesture recognition model. The vehicle screen is controlled based on the gesture category corresponding to the at least one frame of gesture image.

2. The method according to claim 1, characterized in that, The step of obtaining the gesture state vector corresponding to each frame of the gesture image includes: Each frame of the gesture image is input into a pre-trained hand detection model for hand detection, and the hand region location information and left / right hand category identifiers corresponding to each frame of the gesture image are obtained from the output of the hand detection model. Based on the hand region location information, a hand region image is determined from each frame of the gesture image; The hand region image is input into a pre-trained keypoint detection model to detect hand keypoints, and the hand keypoint information of each frame of the gesture image output by the keypoint detection model is obtained. Based on the left and right hand category identifiers and hand key point information of each frame of the gesture image, the gesture state vector of each frame of the gesture image is determined.

3. The method according to claim 2, characterized in that, The step of determining the gesture state vector for each frame of the gesture image based on the left and right hand category identifiers and the hand key point information includes: For each frame of the gesture image, based on the key point identifier of each hand key point in the hand key point information, the wrist key point coordinates and the finger key point coordinates corresponding to each finger in the hand key point information are determined, and the finger key point coordinates include the metacarpophalangeal joint key point coordinates. Based on the coordinates of the wrist key points and the coordinates of the metacarpophalangeal joint key points corresponding to each finger, the first skeletal vector from the wrist to the metacarpophalangeal joint is determined. Based on the coordinates of the key points of each finger, determine the second skeletal vector between adjacent key points on each finger; By splicing the left and right hand category identifiers, the first bone vector, and the second bone vector of the same gesture image, the gesture state vector of each frame of the gesture image is obtained.

4. The method according to claim 1, characterized in that, The gesture recognition model includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, an attention layer, and a fully connected layer; The step of inputting the gesture state vector corresponding to each frame of the gesture image and the associated driving state vector into a pre-trained gesture recognition model for gesture recognition, and obtaining the gesture category corresponding to each frame of the gesture image output by the gesture recognition model, includes: For each frame of the gesture image, the gesture state vector is input into the first feature extraction layer for feature extraction to obtain gesture features; The driving state vector is input into the second feature extraction layer for feature extraction to obtain driving features; The fused features of the gesture features and the driving features are input into the third feature extraction layer for feature extraction to obtain multimodal features; The attention layer extracts attention features by using the gesture features as key-value pairs and the multimodal features as query inputs to obtain the target attention features; The target attention features are input into the fully connected layer for gesture recognition to obtain the gesture category.

5. The method according to claim 1, characterized in that, The number of gesture images is a preset number of frames, and the preset number of gesture images form a continuous image; controlling the vehicle screen based on the gesture category corresponding to the at least one frame of gesture images includes: Obtain the number of times the gesture category appears in the consecutive images corresponding to a preset number of frames; Based on the number of times each gesture category appears, the target gesture category corresponding to the consecutive images of a preset number of frames is determined; The vehicle screen is controlled based on the target gesture category.

6. The method according to claim 1, characterized in that, The vehicle driving parameters include at least one of the following: longitudinal acceleration, lateral acceleration, vertical acceleration, bump frequency, and vehicle speed.

7. The method according to any one of claims 1-6, characterized in that, The acquisition of at least one frame of gesture image and the vehicle driving parameters associated with each frame of gesture image includes: Detect whether the control mode of the vehicle screen supports gesture control mode; In response to supporting the gesture control mode, detect whether the gesture control mode is enabled; In response to the gesture control mode being in the enabled state, at least one frame of gesture image and vehicle driving parameters associated with each frame of gesture image are acquired.

8. A vehicle screen control device, characterized in that, include: The first acquisition module is used to acquire at least one frame of gesture image and vehicle driving parameters associated with each frame of gesture image; The second acquisition module is used to acquire the gesture state vector corresponding to each frame of the gesture image and the driving state vector of the associated vehicle driving parameters. The third acquisition module is used to input the gesture state vector corresponding to each frame of the gesture image and the associated driving state vector into a pre-trained gesture recognition model for gesture recognition, so as to obtain the gesture category corresponding to each frame of the gesture image output by the gesture recognition model. The control module is used to control the vehicle screen based on the gesture category corresponding to the at least one frame of gesture image.

9. An electronic device, characterized in that, include: Processor and memory; The processor executes the vehicle screen control method as described in any one of claims 1 to 7 by calling the program or instructions stored in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that cause a computer to perform the vehicle screen control method as described in any one of claims 1 to 7.

Images