Vehicle air conditioning control method, device, electronic device and storage medium

Abstract

The present application provides a method, device, electronic device, and storage medium for controlling an in-vehicle air conditioner. The method includes the following steps: acquiring an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture user gestures appearing on an in-vehicle screen; determining a quality index value of the image frame, wherein the quality index value of the image frame represents the lighting quality of the image frame; and when the quality index value of the image frame is less than a first preset threshold, performing quality enhancement processing on the image frame so that the quality index value of the image frame is greater than the first preset threshold. The present application utilizes image frames generated by a monocular camera for recognition, eliminating the need for multiple cameras for imaging. This overcomes the high cost disadvantage of the prior art and offers the advantage of low cost. Furthermore, the present application can reduce the impact of the monocular camera's lighting environment on gesture recognition through image quality enhancement processing.

Description

Technical Field

[0001] The present application relates to the field of vehicle air conditioning control, and more specifically, to a vehicle air conditioning control method, device, electronic device, and storage medium. Background Art

[0002] In an era of rapid development of intelligent systems, there are more and more auxiliary operations in the car, but the related operations are also becoming more and more complex. When changing the auxiliary operations in the car, the driver's eyes will inevitably stray from the driving route, which is prone to traffic accidents. In order to simplify the auxiliary operations of related systems in the car and prevent the driver's eyes from straying from the driving route, the following technical solutions for in-vehicle air conditioning control are currently available:

[0003] 1. Traditional human-computer interaction. The driver manually adjusts and sets various air conditioning parameters, including temperature, wind speed, and direction, by turning knobs or physical buttons on the air conditioning control panel on the center console and following the parameters displayed on the vehicle's display. However, this technical solution hinders the driver's ability to maintain high concentration while driving. The driver needs to observe the operating interface to control related functions, which inevitably diverts their gaze from the driving route, hindering safe driving.

[0004] 2. Voice human-computer interaction. The driver uses the vehicle's built-in microphone to recognize voice commands. The voice processing unit parses and understands the voice commands, converting them into corresponding control commands to adjust the vehicle's air conditioning system. However, this technical solution is limited by the noise level of the driving environment and the differences in drivers' accents and dialects, resulting in poor recognition results.

[0005] 3. Gesture-based human-computer interaction. This method utilizes a 3D depth-sensing camera inside the vehicle to capture 3D depth data, which is then transmitted to the onboard system for processing and recognition. This 3D depth data can simplify gesture recognition processing, reducing the computing power required for image processing and gesture recognition. However, multi-module cameras are expensive, making them difficult to popularize. Summary of the Invention

[0006] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for controlling an in-vehicle air-conditioning system, which is used to recognize user gestures based on image frames generated by a monocular camera, and to control the in-vehicle air-conditioning system based on the user gestures. Compared with the prior art, the present application uses image frames generated by a monocular camera for recognition while realizing the control of the in-vehicle air-conditioning system based on gestures, without the need for multiple cameras for imaging, thereby at least overcoming the technical defect of high cost in the prior art, thereby having the advantage of low cost. On the other hand, the present application can reduce the influence of the lighting environment of the monocular camera on the recognition of gestures through image quality enhancement processing.

[0007] In a first aspect, the present invention provides a vehicle air conditioning control method, the method comprising:

[0008] Acquire an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture a user's gesture action appearing in front of the vehicle screen;

[0009] Determining a quality index value of the image frame, wherein the quality index value of the image frame represents a lighting quality of the image frame;

[0010] When the quality index value of the image frame is less than a first preset threshold, performing quality enhancement processing on the image frame so that the quality index value of the image frame is greater than the first preset threshold;

[0011] When the number of image frames meeting the condition is greater than or equal to a second preset threshold, identifying the user gesture based on the image frames meeting the condition and the internal and external parameters of the monocular camera, wherein the image frames meeting the condition are image frames whose quality index values ​​are greater than the first preset threshold;

[0012] The vehicle air conditioner is controlled based on the user gesture action.

[0013] The method of the present application obtains an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture the user's gesture action appearing in front of the vehicle screen, thereby determining the quality index value of the image frame, wherein the quality index value of the image frame represents the lighting quality of the image frame, and then when the quality index value of the image frame is less than a first preset threshold, the image frame is quality enhanced so that the quality index value of the image frame is greater than the first preset threshold. Then, when the number of image frames that meet the conditions is greater than or equal to a second preset threshold, the user gesture action is recognized based on the image frames that meet the conditions and the internal and external parameters of the monocular camera, wherein the image frames that meet the conditions are image frames whose quality index value is greater than the first preset threshold, thereby controlling the vehicle air conditioner based on the user gesture action. Finally, the user gesture action is recognized based on the image frames generated by the monocular camera, and the vehicle air conditioner is controlled based on the user gesture action. Compared with the prior art, the present application uses the image frames generated by the monocular camera for recognition while realizing the control of the vehicle air conditioner based on gesture action, without requiring multiple cameras for imaging, thereby overcoming the technical disadvantage of the prior art of high cost, thereby having the advantage of low cost. On the other hand, the present application can reduce the impact of the lighting environment of monocular camera on gesture action recognition through image quality enhancement processing.

[0014] In an optional embodiment, after the image frame is enhanced and before the user gesture is recognized based on the image frame that meets the conditions and the internal and external parameters of the monocular camera, the method further includes:

[0015] It is determined whether the quality index value of the image frame is still less than the first preset threshold value. If the quality index value of the image frame is still less than the first preset threshold value, the image frame is discarded.

[0016] This optional implementation method can discard the image frame by judging whether the quality index value of the image frame is still less than the first preset threshold value, thereby reducing the difficulty of data processing for complex scenes.

[0017] In an optional embodiment, the identifying the user gesture based on the image frame that meets the conditions and the internal and external parameters of the monocular camera includes:

[0018] Using the image frame that meets the conditions as an input of a hand region detection model to identify a preselected hand region based on the hand region detection model;

[0019] determining a target hand area based on the preselected hand area;

[0020] Using the target hand area as an input to a gesture key point detection model to identify target gesture key points based on the gesture key point detection model;

[0021] The user gesture action is recognized based on the target gesture key points.

[0022] This optional implementation can use the image frame that meets the conditions as the input of the hand area detection model to identify the pre-selected hand area based on the hand area detection model, and then determine the target hand area based on the pre-selected hand area, and then use the target hand area as the input of the gesture key point detection model to identify the target gesture key point based on the gesture key point detection model, so as to identify the user gesture action based on the target gesture key point.

[0023] In an optional embodiment, the hand area detection model is a YOLOv7 algorithm model, wherein the YOLOv7 algorithm model is used to extract medium target features and large target features in the image frame based on an attention mechanism.

[0024] This optional implementation can identify the hand area based on the YOLOv7 algorithm model, and can introduce the attention mechanism into the YOLOv7 algorithm model, so that the YOLOv7 algorithm model pays more attention to the medium target features and large target features in the image frame, thereby improving its own recognition accuracy of the hand area.

[0025] In an optional embodiment, determining the target hand area based on the pre-selected hand area includes:

[0026] Determining the size information of the preliminary human head area and the distance from the preliminary human head area to the center of the monocular camera;

[0027] Whether the preliminary candidate's hand area is the target hand area is determined based on the size information of the preliminary candidate's hand area and the distance from the preliminary candidate's hand area to the center of the monocular camera.

[0028] This optional implementation method determines the size information of the preliminary contestant's hand area and the distance from the preliminary contestant's hand area to the center of the monocular camera, and then can determine whether the preliminary contestant's hand area is the target hand area based on the size information of the preliminary contestant's hand area and the distance from the preliminary contestant's hand area to the center of the monocular camera.

[0029] In an optional embodiment, the gesture key point detection model is a HandUNet network model, wherein the HandUNet network model is used to identify 21 gesture key points from the target hand area.

[0030] This optional implementation may use the HandUNet network model as a gesture key point detection model, and then identify 21 gesture keys from the target hand area based on the HandUNet network model.

[0031] In an optional embodiment, the identifying the user gesture action based on the target gesture key point includes:

[0032] Determining the coordinates of each of the 21 gesture key points and finger angle information;

[0033] The user gesture action is identified based on the coordinates of each gesture key point and the angle information of the finger.

[0034] This optional implementation determines the coordinates of each of the 21 gesture key points and the angle information of the finger, and can then identify the user gesture action based on the coordinates of each gesture key point and the angle information of the finger.

[0035] In a second aspect, the present invention provides a vehicle air conditioning control device, the device comprising:

[0036] An acquisition module, configured to acquire an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is configured to capture a user's gesture action appearing in front of the vehicle-mounted screen;

[0037] a determination module, configured to determine a quality index value of the image frame, wherein the quality index value of the image frame represents a lighting quality of the image frame;

[0038] an image enhancement processing module, configured to, when a quality index value of the image frame is less than a first preset threshold, perform quality enhancement processing on the image frame so that the quality index value of the image frame is greater than the first preset threshold;

[0039] a gesture recognition module, configured to recognize user gestures based on the image frames meeting the conditions and internal and external parameters of the monocular camera when the number of image frames meeting the conditions is greater than or equal to a second preset threshold, wherein the image frames meeting the conditions are image frames having a quality index value greater than the first preset threshold;

[0040] A control module is used to control the vehicle air conditioner based on the user gesture action.

[0041] The device of the present application executes a vehicle air conditioning control method, capable of acquiring image frames, wherein the image frames are generated by a monocular camera, the monocular camera being used to capture user gestures appearing in front of a vehicle screen, and then determining a quality index value of the image frames, wherein the quality index value of the image frames represents the lighting quality of the image frames. Furthermore, when the quality index value of the image frames is less than a first preset threshold, the image frames are quality enhanced so that the quality index value of the image frames is greater than the first preset threshold. Furthermore, when the number of image frames meeting the conditions is greater than or equal to a second preset threshold, the device recognizes user gestures based on the image frames meeting the conditions and internal and external parameters of the monocular camera, wherein the image frames meeting the conditions are image frames whose quality index values ​​are greater than the first preset threshold, thereby controlling the vehicle air conditioning based on the user gestures. Finally, the user gestures are recognized based on the image frames generated by the monocular camera, and the vehicle air conditioning is controlled based on the user gestures. Compared with the prior art, the present application uses image frames generated by the monocular camera for recognition while achieving gesture-based control of the vehicle air conditioning, eliminating the need for multiple cameras for imaging, thereby overcoming the technical disadvantage of the prior art of high cost and having the advantage of low cost. On the other hand, the present application can reduce the impact of the lighting environment of monocular camera on gesture action recognition through image quality enhancement processing.

[0042] In a third aspect, the present invention provides an electronic device, comprising:

[0043] processor; and

[0044] The memory is configured to store machine-readable instructions, which, when executed by the processor, execute the vehicle air conditioning control method as described in any one of the aforementioned embodiments.

[0045] The electronic device of the present application can obtain image frames by executing a vehicle air-conditioning control method, wherein the image frames are generated by a monocular camera, and the monocular camera is used to capture user gestures appearing in front of the vehicle screen, and then can determine the quality index value of the image frames, wherein the quality index value of the image frames represents the lighting quality of the image frames, and then can perform quality enhancement processing on the image frames when the quality index value of the image frames is less than a first preset threshold value, so that the quality index value of the image frames is greater than the first preset threshold value, and then can identify user gestures based on the image frames that meet the conditions and the internal and external parameters of the monocular camera when the number of image frames that meet the conditions is greater than or equal to a second preset threshold value, wherein the image frames that meet the conditions are image frames whose quality index values ​​are greater than the first preset threshold value, so that the vehicle air-conditioning can be controlled based on the user gestures, and finally, the user gestures are identified based on the image frames generated by the monocular camera, and the vehicle air-conditioning is controlled based on the user gestures. Compared to existing technologies, this application not only implements gesture-based control of vehicle air conditioning, but also uses image frames generated by a monocular camera for recognition, eliminating the need for multiple cameras. This overcomes the high cost disadvantage of existing technologies and offers the advantage of low cost. Furthermore, through image quality enhancement processing, this application can reduce the impact of the monocular camera's lighting environment on gesture recognition.

[0046] In a fourth aspect, the present invention provides a storage medium storing a computer program, wherein the computer program is executed by a processor to implement the vehicle air conditioning control method as described in any one of the aforementioned embodiments.

[0047] The storage medium of the present application can obtain image frames by executing the vehicle air-conditioning control method, wherein the image frames are generated by a monocular camera, and the monocular camera is used to capture user gestures appearing in front of the vehicle screen, and then can determine the quality index value of the image frames, wherein the quality index value of the image frames represents the lighting quality of the image frames, and then can perform quality enhancement processing on the image frames when the quality index value of the image frames is less than a first preset threshold value, so that the quality index value of the image frames is greater than the first preset threshold value, and then can identify user gestures based on the image frames that meet the conditions and the internal and external parameters of the monocular camera when the number of image frames that meet the conditions is greater than or equal to a second preset threshold value, wherein the image frames that meet the conditions are image frames whose quality index values ​​are greater than the first preset threshold value, so that the vehicle air-conditioning can be controlled based on the user gestures, and finally, the user gestures are identified based on the image frames generated by the monocular camera, and the vehicle air-conditioning is controlled based on the user gestures. Compared to existing technologies, this application not only implements gesture-based control of vehicle air conditioning, but also uses image frames generated by a monocular camera for recognition, eliminating the need for multiple cameras. This overcomes the high cost disadvantage of existing technologies and offers the advantage of low cost. Furthermore, through image quality enhancement processing, this application can reduce the impact of the monocular camera's lighting environment on gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 This is a flow chart of a vehicle air-conditioning controller disclosed in an embodiment of the present application;

[0050] Figure 2 This is a schematic structural diagram of a vehicle air-conditioning control device disclosed in an embodiment of the present application;

[0051] Figure 3 This is a structural diagram of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0053] Example 1

[0054] See also Figure 1 , Figure 1 This is a flow chart of a vehicle air conditioning control method disclosed in an embodiment of the present application, such as Figure 1 As shown, the method of the embodiment of the present application includes the following steps:

[0055] 101. Acquire an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture a user's gesture action appearing in front of the vehicle screen;

[0056] 102. Determine a quality index value of the image frame, wherein the quality index value of the image frame represents a lighting quality of the image frame;

[0057] 103. When the quality index value of the image frame is less than a first preset threshold, perform quality enhancement processing on the image frame so that the quality index value of the image frame is greater than the first preset threshold;

[0058] 104. When the number of image frames meeting the condition is greater than or equal to a second preset threshold, identifying a user gesture based on the image frames meeting the condition and internal and external parameters of the monocular camera, wherein the image frames meeting the condition are image frames having a quality index value greater than the first preset threshold;

[0059] 105. Control the car air conditioner based on user gestures.

[0060] The method of the embodiment of the present application obtains an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture the user's gesture action appearing in front of the vehicle screen, thereby determining the quality index value of the image frame, wherein the quality index value of the image frame represents the lighting quality of the image frame, and then when the quality index value of the image frame is less than a first preset threshold, the image frame is quality enhanced so that the quality index value of the image frame is greater than the first preset threshold. Then, when the number of image frames meeting the conditions is greater than or equal to a second preset threshold, the method can recognize the user's gesture action based on the image frames meeting the conditions and the internal and external parameters of the monocular camera, wherein the image frames meeting the conditions are image frames whose quality index value is greater than the first preset threshold, thereby controlling the vehicle air conditioner based on the user's gesture action. Finally, the user's gesture action is recognized based on the image frames generated by the monocular camera, and the vehicle air conditioner is controlled based on the user's gesture action. Compared with the prior art, the present application uses the image frames generated by the monocular camera for recognition while realizing the control of the vehicle air conditioner based on the gesture action, without requiring multiple cameras for imaging, thereby overcoming the technical disadvantage of the prior art of high cost, thereby having the advantage of low cost. On the other hand, the present application can reduce the impact of the lighting environment of monocular camera on gesture action recognition through image quality enhancement processing.

[0061] In the embodiments of this application, a monocular camera refers to a camera device with only one lens, which is less expensive than the multi-lens cameras used in the prior art. Furthermore, the prior art does not use a monocular camera because it acquires more images, including multi-angle imaging, to ensure the accuracy of gesture recognition. Multi-angle imaging is beneficial for improving gesture recognition accuracy, but it is more expensive.

[0062] In the embodiment of the present application, an image frame refers to a static image formed by a monocular camera at a certain moment, wherein the monocular camera is aimed at the front and back of the vehicle-mounted screen, and will capture the user's gestures appearing in front of the vehicle-mounted screen in real time and form multiple image frames.

[0063] In the embodiment of the present application, the quality index values ​​of the image frame include image correlation loss, brightness distortion, and contrast distortion.

[0064] In an embodiment of the present application, the first preset threshold refers to the comparison value of the quality index value of the image frame. If the quality index value of the image frame is less than the first preset threshold, it means that the image quality of the image frame is not very good, which may affect subsequent steps such as hand area recognition and gesture key point recognition. Based on this, by enhancing such image frames, the impact of such image frames on hand area recognition and gesture key point recognition can be reduced, thereby improving the accuracy of hand area recognition and gesture key point recognition.

[0065] In the embodiments of the present application, since user gestures are continuous and continuous user gestures are represented by multiple image frames, user gesture recognition must be based on multiple image frames. Furthermore, in general, a user gesture can be fully recorded by 30 image frames. Therefore, the second preset threshold value can be 30. That is, when the number of image frames that meet the conditions is greater than or equal to 30, the user gesture is recognized based on the image frames that meet the conditions and the internal and external parameters of the monocular camera.

[0066] In an embodiment of the present application, the intrinsic parameters of a monocular camera refer to parameters related to the characteristics of the camera itself, including focal length, principal point offset, and pixel scale factor. These parameters are usually obtained through a camera calibration process and are used to convert points in an image coordinate system into a camera coordinate system. On the other hand, the extrinsic parameters of a monocular camera refer to parameters that describe the position and posture of the camera relative to the world coordinate system, mainly including a rotation matrix and a translation vector. The extrinsic parameters reflect the specific position and direction of the camera in space, which is very important for multi-view geometry calculations. In an embodiment of the present application, as an optional implementation method, after the image frame is enhanced, based on the image frame that meets the conditions and the internal and external parameters of the monocular camera, before recognizing the user's gesture action, the method of the embodiment of the present application further includes the following steps:

[0067] It is determined whether the quality index value of the image frame is still less than the first preset threshold value. If the quality index value of the image frame is still less than the first preset threshold value, the image frame is discarded.

[0068] This optional implementation method can discard the image frame by judging whether the quality index value of the image frame is still less than the first preset threshold, thereby reducing the difficulty of data processing for complex scenes.

[0069] For this optional implementation, after the image frame is quality enhanced, its quality index value may still be less than the first preset threshold. In the case where the quality index value is still less than the first preset threshold, the image frame is discarded to avoid further processing of the image frame using complex processing algorithms, thereby reducing the difficulty of data processing for complex scenes.

[0070] In an embodiment of the present application, as an optional implementation, recognizing user gestures based on image frames that meet conditions and internal and external parameters of a monocular camera includes the following sub-steps:

[0071] Using the image frame that meets the conditions as the input of the hand region detection model to identify the pre-selected hand region based on the hand region detection model;

[0072] Determining a target hand area based on a preselected hand area;

[0073] Taking the target hand region as the input of the gesture key point detection model to identify the target gesture key points based on the gesture key point detection model;

[0074] Recognize user gestures based on target gesture key points.

[0075] This optional implementation can use image frames that meet the conditions as input to the hand area detection model, so as to identify the pre-selected hand area based on the hand area detection model, and then determine the target hand area based on the pre-selected hand area, and then use the target hand area as input to the gesture key point detection model, so as to identify the target gesture key points based on the gesture key point detection model, and thus recognize the user gesture action based on the target gesture key points.

[0076] In the above optional implementation, the hand area detection model usually outputs several image frames, wherein the image frames mark the areas in the image where gestures may exist. These areas are temporarily determined to have gestures, and are therefore called preliminary hand areas, while the areas where gestures have been determined to exist are called target gesture areas.

[0077] In the above optional implementation manner, the user gesture action can be determined by the geometric relationship formed by the gesture key points.

[0078] In an embodiment of the present application, in an optional implementation, the hand area detection model is a YOLOv7 algorithm model, wherein the YOLOv7 algorithm model is used to extract medium target features and large target features in the image frame based on the attention mechanism.

[0079] This optional implementation can identify the hand area based on the YOLOv7 algorithm model, and can introduce the attention mechanism into the YOLOv7 algorithm model, so that the YOLOv7 algorithm model pays more attention to the medium target features and large target features in the image frame, thereby improving its own recognition accuracy of the hand area.

[0080] In this optional embodiment, the "YOLOv7 algorithm model" refers to an advanced real-time object detection algorithm. The YOLO (You Only Look Once) family of algorithms is a well-known class of algorithms for object detection in the field of computer vision. YOLOv7, the seventh version of the family, maintains the YOLO family's fast detection speed while further improving detection accuracy and efficiency. YOLOv7 is particularly suitable for high-performance, low-latency object detection applications such as autonomous driving and video surveillance.

[0081] In an embodiment of the present application, as an optional implementation, determining the target hand area based on the pre-selected hand area includes the following sub-steps:

[0082] Determine the size of the pre-selected head area and the distance from the pre-selected head area to the center of the monocular camera;

[0083] Based on the size information of the preliminary hand area and the distance from the preliminary hand area to the center of the monocular camera, it is determined whether the preliminary hand area is the target hand area.

[0084] This optional implementation method determines the size information of the preliminary contestant's head area and the distance from the preliminary contestant's head area to the center of the monocular camera, and then can determine whether the preliminary contestant's head area is the target hand area based on the size information of the preliminary contestant's head area and the distance from the preliminary contestant's head area to the center of the monocular camera.

[0085] Regarding the above-mentioned optional implementation, generally, the distance from the pre-selected head area to the center of the monocular camera is associated with the size information of the pre-selected head area, wherein the greater the distance from the hand area to the center of the monocular camera, the larger the size of the hand area in theory. In this way, the target gesture area can be screened out from multiple pre-selected head areas through the size information of the pre-selected head area and the distance from the pre-selected head area to the center of the monocular camera. For example, if the distance from the pre-selected head area to the center of the monocular camera is A, and the size of the pre-selected head area is B, theoretically, when the distance from the hand area to the center of the monocular camera is A, the size of the hand area should be C. In this case, it can be determined that the pre-selected head area is not the target gesture area. However, if the distance from the hand area to the center of the monocular camera is A, the size of the hand area should be B, then the pre-selected head area can be determined as the target gesture area.

[0086] In an embodiment of the present application, as an optional implementation, the gesture key point detection model is a HandUNet network model, wherein the HandUNet network model is used to identify 21 gesture key points from the target hand area.

[0087] This optional implementation may use the HandUNet network model as a gesture key point detection model, and then identify 21 gesture keys from the target hand area based on the HandUNet network model.

[0088] For this optional implementation, the "HandUNet network model" is a deep learning model specifically designed for hand recognition or hand posture estimation. This model is usually based on the UNet architecture, which is a convolutional neural network (CNN) commonly used for image segmentation tasks. UNet is characterized by an encoder-decoder structure, in which the encoder is responsible for extracting features from the input image, and the decoder maps these features back to an output with the same size as the input image, thereby achieving pixel-level prediction. HandUNet makes specific adjustments and optimizations based on UNet to better adapt to the processing requirements of hand images, such as improving the ability to capture hand details and enhancing the robustness of the model.

[0089] In an embodiment of the present application, as an optional implementation, identifying a user gesture action based on a target gesture key point includes the following steps:

[0090] Determine the coordinates of each of the 21 gesture key points, as well as the angle information of the fingers;

[0091] Identify user gestures based on the coordinates of each gesture key point and the angle information of the fingers.

[0092] This optional implementation determines the coordinates of each of the 21 gesture key points and the angle information of the fingers, and can then identify the user's gesture action based on the coordinates of each gesture key point and the angle information of the fingers.

[0093] For the embodiment of this application, as an example:

[0094] Visual images are obtained through a monocular camera, and each frame of hand data collected is subjected to a low-light test to determine whether low-light image enhancement is required. The evaluation metric used for low-light testing is the Universal Quality Image Index (UQI). (The Universal Quality Image Index (UQI) combines three factors: loss of image correlation, brightness distortion, and contrast distortion to reflect image quality.) If the UQI is greater than 0.8, the frame is directly transmitted to the hand area detection module. If the UQI is less than 0.8, low-light image enhancement is required. The enhanced image is then evaluated again. If the UQI is greater than 0.8, it is transmitted to the hand area detection module. If it is still less than 0.8, the frame is directly discarded. When the number of image frames that pass the low-light test reaches 30, the image is input into the hand area detection module along with the internal and external parameter data of the monocular camera.

[0095] Furthermore, after 30 frames of low-light testing are transferred to the hand region detection module, image features are acquired using the Backbone and Neck layers of the YOLOv7 model. The Backbone layer passes through four CBS layers with a Maxpool layer before being transferred to the ELAN module. The output of the triple MP+ELAN is then connected to increase the grid level, improve model accuracy, and acquire image features.

[0096] Furthermore, in the Neck layer, the input feature map is encoded along the horizontal and vertical coordinates, and the feature map embedded with the width and height coordinate information is obtained by average pooling. The formulas for the two feature maps are as follows:

[0097]

[0098] Where: x c The feature vector representing the input; Represents the output at height h;

[0099] H represents the height of the initial input feature vector;

[0100] W represents the width of the initial input feature vector.

[0101] Furthermore, the two average pooled feature maps are spliced ​​and batch normalized to obtain the feature map F1, and then the feature map f is obtained through the nonlinear activation function. The formula of the feature map f is as follows:

[0102] f=δ(F([z h ,z w ]));

[0103] Then the feature map f is split vertically and horizontally into feature maps F with the same number of channels as the original h and F w , and then use the sigmoid activation function to obtain the attention weight information g in the height and width directions respectively h and g w , where the attention weight information g h and g w The formula is as follows:

[0104]

[0105] Finally, the attention weight information g in the height and width directions is h and g w , perform weighted multiplication and output as a feature map y with attention weights c (i,j). The calculation formula is as follows:

[0106]

[0107] After completing the above steps, the CA attention mechanism module was added to MP-2 in the original Neck layer, enabling the network model to focus more on medium and large objects. This enhanced the YOLOv7 algorithm's feature extraction of medium and large objects in the feature map, improving the accuracy of hand area detection. Subsequently, all detected hand areas were evaluated based on the size of the target box and the distance from the camera center to determine whether they belonged to the driver's hand.

[0108] Furthermore, the image, which had been hand-region detected using the improved YOLOv7 network model, was input into the HandUNet network model for hand keypoint detection, resulting in 21 hand keypoints of the operator. Each keypoint data contains four pieces of information: x-axis coordinate, y-axis coordinate, z-axis coordinate, and confidence level. Therefore, the resulting 21 hand keypoints contain a total of 84 pieces of information. Each finger has four keypoints, plus one keypoint at the base of the palm. Lines are drawn between the two adjacent points on each finger, and the points at the base of the finger are drawn with the points at the base of the palm. It can be seen that three angles can be obtained for one finger, and a total of 15 angles can be collected for five fingers. Gesture labels are classified based on the angle information and the hand keypoint information. The BiGRU network is then fused with the Capsule network to extract the driver's hand keypoints and angle information, achieving highly accurate gesture command recognition.

[0109] Example 2

[0110] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of a vehicle air-conditioning control device disclosed in an embodiment of the present application. Figure 2 As shown, the device of the embodiment of the present application includes the following functional modules:

[0111] An acquisition module 201 is configured to acquire an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is configured to capture a user's gestures appearing in front of the vehicle-mounted screen;

[0112] A determination module 202 is configured to determine a quality index value of an image frame, wherein the quality index value of the image frame represents a lighting quality of the image frame;

[0113] The image enhancement processing module 203 is configured to perform quality enhancement processing on the image frame when the quality index value of the image frame is less than a first preset threshold value, so as to make the quality index value of the image frame greater than the first preset threshold value;

[0114] a gesture recognition module 204 configured to recognize user gestures based on the image frames meeting the conditions and internal and external parameters of the monocular camera when the number of image frames meeting the conditions is greater than or equal to a second preset threshold, wherein the image frames meeting the conditions are image frames having a quality index value greater than the first preset threshold;

[0115] The control module 205 is used to control the vehicle air conditioner based on user gestures.

[0116] The device of the embodiment of the present application performs a vehicle air conditioning control method, which can obtain image frames, wherein the image frames are generated by a monocular camera, and the monocular camera is used to capture user gestures appearing on the vehicle screen, and then determine the quality index value of the image frames, wherein the quality index value of the image frames represents the lighting quality of the image frames, and then, when the quality index value of the image frames is less than a first preset threshold, the image frames can be quality enhanced so that the quality index value of the image frames is greater than the first preset threshold. Then, when the number of image frames that meet the conditions is greater than or equal to a second preset threshold, the device can recognize the user gesture based on the image frames that meet the conditions and the internal and external parameters of the monocular camera, wherein the image frames that meet the conditions are image frames whose quality index value is greater than the first preset threshold, thereby controlling the vehicle air conditioning based on the user gesture. Finally, the user gesture is recognized based on the image frames generated by the monocular camera, and the vehicle air conditioning is controlled based on the user gesture. Compared with the prior art, the present application uses image frames generated by the monocular camera for recognition while achieving gesture control of the vehicle air conditioning, without requiring multiple cameras for imaging, thereby overcoming the technical disadvantage of the prior art of high cost, thereby having the advantage of low cost. On the other hand, the present application can reduce the impact of the lighting environment of monocular camera on gesture action recognition through image quality enhancement processing.

[0117] It should be noted that for other detailed descriptions of the device in the embodiment of the present application, please refer to the relevant description of the first embodiment of the present application, and the embodiment of the present application will not go into details thereon.

[0118] Example 3

[0119] See also Figure 3 , Figure 3 is a structural diagram of an electronic device disclosed in an embodiment of the present application, such as Figure 3 As shown, the electronic device of the embodiment of the present application includes:

[0120] Processor 301; and

[0121] The memory 302 is configured to store machine-readable instructions. When the instructions are executed by the processor 301, the vehicle air conditioning control method according to any one of the aforementioned embodiments is executed.

[0122] The electronic device of the embodiment of the present application can obtain image frames by executing a vehicle air conditioning control method, wherein the image frames are generated by a monocular camera, and the monocular camera is used to capture user gestures appearing in front of the vehicle screen, thereby determining the quality index value of the image frames, wherein the quality index value of the image frames represents the lighting quality of the image frames, and then when the quality index value of the image frames is less than a first preset threshold, the image frames can be quality enhanced so that the quality index value of the image frames is greater than the first preset threshold. Then, when the number of image frames that meet the conditions is greater than or equal to a second preset threshold, the electronic device can recognize the user gesture based on the image frames that meet the conditions and the internal and external parameters of the monocular camera, wherein the image frames that meet the conditions are image frames whose quality index value is greater than the first preset threshold, thereby controlling the vehicle air conditioning based on the user gesture. Finally, the user gesture is recognized based on the image frames generated by the monocular camera, and the vehicle air conditioning is controlled based on the user gesture. Compared with the prior art, the present application uses image frames generated by the monocular camera for recognition while realizing gesture control of the vehicle air conditioning, without requiring multiple cameras for imaging, thereby overcoming the technical disadvantage of the prior art of high cost, thereby having the advantage of low cost. On the other hand, the present application can reduce the impact of the lighting environment of monocular camera on gesture action recognition through image quality enhancement processing.

[0123] Example 4

[0124] An embodiment of the present application provides a storage medium storing a computer program, and the computer program is executed by a processor as a vehicle air conditioning control method as described in any of the aforementioned embodiments.

[0125] The storage medium of the embodiment of the present application can obtain image frames by executing a vehicle air conditioning control method, wherein the image frames are generated by a monocular camera, and the monocular camera is used to capture user gestures appearing in front of the vehicle screen, thereby determining the quality index value of the image frames, wherein the quality index value of the image frames represents the lighting quality of the image frames, and then when the quality index value of the image frames is less than a first preset threshold, the image frames can be quality enhanced so that the quality index value of the image frames is greater than the first preset threshold, and then when the number of image frames that meet the conditions is greater than or equal to a second preset threshold, the user gestures can be recognized based on the image frames that meet the conditions and the internal and external parameters of the monocular camera, wherein the image frames that meet the conditions are image frames whose quality index values ​​are greater than the first preset threshold, thereby controlling the vehicle air conditioning based on the user gestures, and finally recognizing the user gestures based on the image frames generated by the monocular camera, and controlling the vehicle air conditioning based on the user gestures. Compared with the prior art, the present application uses image frames generated by the monocular camera for recognition while realizing gesture-based control of the vehicle air conditioning, without requiring multiple cameras for imaging, thereby overcoming the technical disadvantage of the prior art of high cost, thereby having the advantage of low cost. On the other hand, the present application can reduce the impact of the lighting environment of monocular camera on gesture action recognition through image quality enhancement processing.

[0126] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0127] In addition, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] Furthermore, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0129] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0130] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0131] The above are merely examples of the present application and are not intended to limit the scope of protection of the present application. Those skilled in the art will appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A vehicle air conditioning control method, characterized in that: The method comprises: Acquire an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture a user's gesture action appearing in front of the vehicle screen; Determining a quality index value of the image frame, wherein the quality index value of the image frame represents a lighting quality of the image frame; When the quality index value of the image frame is less than a first preset threshold, performing quality enhancement processing on the image frame so that the quality index value of the image frame is greater than the first preset threshold; When the number of image frames meeting the condition is greater than or equal to a second preset threshold, identifying the user gesture based on the image frames meeting the condition and the internal and external parameters of the monocular camera, wherein the image frames meeting the condition are image frames whose quality index values ​​are greater than the first preset threshold; controlling the vehicle air conditioner based on the user gesture; Furthermore, the identifying of user gestures based on the image frames meeting the conditions and the internal and external parameters of the monocular camera includes: Using the image frame that meets the conditions as an input of a hand region detection model to identify a preselected hand region based on the hand region detection model; determining a target hand area based on the preselected hand area; Using the target hand area as an input to a gesture key point detection model to identify target gesture key points based on the gesture key point detection model; Recognizing the user gesture action based on the target gesture key points; Furthermore, the hand region detection model is a YOLOv7 algorithm model, wherein the YOLOv7 algorithm model is used to extract medium target features and large target features in the image frame based on an attention mechanism; Furthermore, determining the target hand area based on the preliminary hand area includes: Determining the size information of the preliminary human head area and the distance from the preliminary human head area to the center of the monocular camera; determining whether the preliminary hand area is the target hand area based on the size information of the preliminary hand area and the distance from the preliminary hand area to the center of the monocular camera; And, the gesture key point detection model is a HandUNet network model, wherein the HandUNet network model is used to identify 21 gesture key points from the target hand area; And, the identifying the user gesture action based on the target gesture key point includes: Determining the coordinates of each of the 21 gesture key points and finger angle information; The user gesture action is identified based on the coordinates of each gesture key point and the angle information of the finger.

2. The method according to claim 1, wherein After the image frame is enhanced, and before the user gesture is recognized based on the image frame that meets the conditions and the internal and external parameters of the monocular camera, the method further includes: It is determined whether the quality index value of the image frame is still less than the first preset threshold value. If the quality index value of the image frame is still less than the first preset threshold value, the image frame is discarded.

3. A vehicle air conditioning control device, characterized in that: The device comprises: An acquisition module, configured to acquire an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is configured to capture a user's gesture action appearing in front of the vehicle-mounted screen; a determination module, configured to determine a quality index value of the image frame, wherein the quality index value of the image frame represents a lighting quality of the image frame; an image enhancement processing module, configured to, when a quality index value of the image frame is less than a first preset threshold, perform quality enhancement processing on the image frame so that the quality index value of the image frame is greater than the first preset threshold; a gesture recognition module, configured to recognize user gestures based on the image frames meeting the conditions and internal and external parameters of the monocular camera when the number of image frames meeting the conditions is greater than or equal to a second preset threshold, wherein the image frames meeting the conditions are image frames having a quality index value greater than the first preset threshold; A control module, configured to control the vehicle air conditioner based on the user's gestures; Furthermore, the identifying of user gestures based on the image frames meeting the conditions and the internal and external parameters of the monocular camera includes: Using the image frame that meets the conditions as an input of a hand region detection model to identify a preselected hand region based on the hand region detection model; determining a target hand area based on the preselected hand area; Using the target hand area as an input to a gesture key point detection model to identify target gesture key points based on the gesture key point detection model; Recognizing the user gesture action based on the target gesture key points; Furthermore, the hand region detection model is a YOLOv7 algorithm model, wherein the YOLOv7 algorithm model is used to extract medium target features and large target features in the image frame based on an attention mechanism; Furthermore, determining the target hand area based on the preliminary hand area includes: Determining the size information of the preliminary human head area and the distance from the preliminary human head area to the center of the monocular camera; determining whether the preliminary hand area is the target hand area based on the size information of the preliminary hand area and the distance from the preliminary hand area to the center of the monocular camera; And, the gesture key point detection model is a HandUNet network model, wherein the HandUNet network model is used to identify 21 gesture key points from the target hand area; And, the identifying the user gesture action based on the target gesture key point includes: Determining the coordinates of each of the 21 gesture key points and finger angle information; The user gesture action is identified based on the coordinates of each gesture key point and the angle information of the finger.

4. An electronic device, characterized in that: include: processor; as well as A memory configured to store machine-readable instructions, which, when executed by the processor, execute the vehicle air conditioning control method according to any one of claims 1 to 2.

5. A storage medium, characterized in that The storage medium stores a computer program, and the computer program is executed by a processor to implement the vehicle air conditioning control method according to any one of claims 1 to 2.

Images