Air conditioner air supply angle adjustment method and system based on gesture control and control device

Abstract

The application discloses a gesture control-based air conditioner air supply angle adjusting method, system and control device, and specifically comprises the following steps: collecting image data and preprocessing; performing feature extraction and human body model prediction processing to obtain a confidence value; judging the size of the confidence value and a threshold value, if the confidence value is greater than or equal to the threshold value, the image is retained, otherwise the image is discarded; obtaining the skeletal key points of the human body in the image and positioning; recognizing the gesture in the image, comparing the determined gesture with the meanings of each gesture in a gesture gesture library, finding the local area of the human body corresponding to the gesture made by the user; and adjusting the air supply angle to the area specified by the user. The application can more accurately and directly receive the thermal comfort feedback of the subject through the relatively accurate hand gesture, when the subject needs to adjust a specific part of the body, only needs to make the gesture corresponding to the part to accurately adjust, and can meet the demand of the special disabled people for personalized thermal comfort adjustment.

Description

Technical Field

[0001] This invention belongs to the field of intelligent air conditioning adjustment technology, specifically relating to a method, system and control device for adjusting the air supply angle of an air conditioner based on gesture control. Background Technology

[0002] People with disabilities have become an important and indispensable part of today's society, and addressing their thermal comfort needs has become a pressing issue. Currently, air conditioning, a crucial tool for thermal comfort, plays a vital role in daily life. However, the existing control modes of air conditioners cannot fully meet users' needs. Users can only adjust overall thermal comfort, and the up-and-down airflow angle can only be adjusted via remote control, which cannot meet the precise adjustment requirements for specific body parts. This is especially difficult for people with disabilities, who find it challenging to use air conditioner remotes. They desire air conditioners that can autonomously adjust parameters based on their posture and hand movements, ideally using easily operable hand gestures. For example, they could determine the user's current thermal needs based on their posture and provide adjustment commands to the air conditioning system, creating a personalized thermal comfort environment.

[0003] In existing technologies, thermal comfort adjustment systems can infer a subject's thermal comfort sensation based on their posture and adjust corresponding thermal comfort parameters. This involves collecting data on human postures under different thermal comfort conditions through questionnaires to define several different thermal comfort postures. Then, video is captured using cameras or similar methods, and the subject's postures in the video are analyzed. These are compared to the defined thermal comfort postures to predict the subject's thermal comfort feedback and provide appropriate adjustments. However, this method has several limitations: First, the thermal comfort posture samples obtained through questionnaires are limited and not representative; furthermore, this method does not consider the thermal comfort needs of people with disabilities. Second, it's important to understand that the purpose of collecting thermal comfort posture data is to predict the user's thermal feedback. After collecting information through questionnaires, further analysis and prediction are required, and this method cannot accurately reflect the subject's subjective feelings. Third, current technologies only support overall thermal comfort adjustment and generally suffer from limitations in adjusting angles or insufficient precision in angle adjustment. Summary of the Invention

[0004] To address the above technical requirements, this invention proposes a method, system, and control device for adjusting the air supply angle of an air conditioner based on gesture control, providing thermal comfort adjustment for localized body parts for people with disabilities, thereby achieving the goal of providing users with a personalized thermal comfort environment.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] The invention first discloses a method for adjusting the air supply angle of an air conditioner based on gesture control, which specifically includes the following steps:

[0007] Step 1: Establish a skeletal key point model of the human body and divide the model into local areas of the human body; set up a gesture posture library, which defines several gestures related to local areas of the human body, one of which represents a local area of ​​the human body that needs to be ventilated.

[0008] Step 2: Use a computer vision device to capture image data of the user.

[0009] Step 3: Preprocess the acquired image data;

[0010] Step 4: Perform feature extraction and human body model prediction on the preprocessed image data to obtain the confidence value of the image;

[0011] Step 5: Determine the relationship between the confidence value and the threshold. If the confidence value is greater than or equal to the threshold, retain the image and proceed to Step 6; otherwise, discard the image.

[0012] Step 6: Obtain the skeletal key points of the human body in the image, locate these skeletal key points, and connect the located skeletal key points to obtain the final human body detection result.

[0013] Step 7: Recognize the gestures in the image and determine the gestures; compare the determined gestures with the meanings of each gesture in the gesture pose library to find the local human body area corresponding to the user's gesture.

[0014] Step 8: Adjust the air supply angle to the user-specified area determined in Step 8;

[0015] Step 9: Real-time acquisition of image data. For each acquired image frame, process it according to steps 3 to 8.

[0016] Optionally, step 3 specifically includes: performing noise reduction processing on the image data of each frame, as well as alignment and occlusion enhancement processing.

[0017] Optionally, step 4 specifically includes: using the BlazePose algorithm to extract features from the image, generating a pixel matrix, inputting the basic parameter matrix corresponding to the generated pixel matrix into the model of the BlazePose algorithm for prediction, and obtaining the confidence value of the frame image.

[0018] Optionally, the threshold value in step 5 is 0.5.

[0019] Optionally, the method for locating skeletal key points in step 6 specifically includes: using the BlazePose algorithm to obtain skeletal key points in a human image, generating a heatmap for each skeletal key point and refining the offset of each coordinate, thereby obtaining the two-dimensional information coordinates of each skeletal key point, and then converting the two-dimensional information coordinates into three-dimensional coordinates.

[0020] Optionally, the specific steps for determining the gesture in step 7 are as follows: obtain key finger points through MediaPipe, determine the open fingers based on the obtained key finger points, and then calculate the angle γ between the open fingers using the law of cosines. If the angle between adjacent fingers is less than 90°, count it as 2 fingers; if it is greater than or equal to 90°, count it as 1 finger; finally determine the gesture.

[0021]

[0022] In the above formula, γ represents the angle between adjacent fingers, and a, b, and c represent the lengths of the three sides of the triangle formed between the two fingers.

[0023] Optionally, step 8 specifically involves: firstly calculating the Euclidean distance between a point in the local area of ​​the human body corresponding to the user's gesture and any skeletal key point in that area; taking the skeletal key point corresponding to the smallest Euclidean distance as the representative skeletal key point of that area; and adjusting the air supply angle to supply air to the location of the representative skeletal key point.

[0024] The present invention also discloses an air conditioning air supply angle adjustment system based on gesture control, the adjustment system comprising:

[0025] The gesture library stores several gestures related to specific areas of the human body, with one gesture representing a specific area of ​​the human body that requires airflow.

[0026] The video capture module is used to capture and collect image data from users.

[0027] The preprocessing module is used to preprocess the acquired image data and output the region of interest image data;

[0028] The feature extraction module is used to extract features from the preprocessed image data;

[0029] The human body detector module is used to acquire the confidence value of the image and filter it frame by frame according to the confidence value to obtain the required image data;

[0030] The human skeleton key point localization module is used to acquire the skeletal key points of the human body in the image and locate these skeletal key points.

[0031] Human gesture recognition module, used to recognize and determine gestures in images;

[0032] The region retrieval module is used to determine the local human body region corresponding to the identified gesture, locate the skeletal key points of the determined local human body region, and determine the precise air delivery angle.

[0033] The present invention also discloses a control device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the method described above in the present invention.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] (1) The present invention uses more accurate hand gestures to receive thermal comfort feedback from the subject more accurately and directly. When the subject needs to adjust a specific part of the body, he / she only needs to make the corresponding gesture to make precise adjustment. For example, when the subject needs to send air to his / her left hand, he / she only needs to make the gesture of "5" with his / her left hand, without having to make gestures such as "rubbing hands" to let the system make predictions.

[0036] (2) This invention is developed with the aim of facilitating thermal comfort adjustment for people with disabilities. It uses easy-to-operate hand gestures to provide thermal comfort feedback and adjustment, which can meet the needs of special people with disabilities for personalized thermal comfort adjustment and help ensure the thermal comfort of people with disabilities. Attached Figure Description

[0037] Figure 1 This is a flowchart of the adjustment method of the present invention.

[0038] Figure 2 This is a distribution model diagram of skeletal key points as described in the embodiments of the present invention.

[0039] Figure 3 These are some examples of gestures described in the embodiments of the present invention.

[0040] Figure 4 This refers to the subject's hand posture identified by the system of this invention. Detailed Implementation

[0041] The preferred embodiments of the present invention are described in detail below. However, the present invention is not limited to the specific embodiments described below. Those skilled in the art can make modifications or equivalent transformations within the scope of the claims, and all such modifications or transformations should be included within the protection scope of the present invention.

[0042] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0043] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0044] Example 1

[0045] This embodiment discloses a method for adjusting the air supply angle of an air conditioner based on gesture control, which specifically includes the following steps:

[0046] Step 1: First, a skeletal keypoint model of the human body needs to be established, such as... Figure 2 The diagram shows the distribution of key nodes in the human skeleton. Table 1 shows the human body parts corresponding to the defined key nodes in the human skeleton.

[0047] Table 1 Key Points of the Human Skeleton

[0048]

[0049]

[0050] A gesture library also needs to be established, defining several gestures related to specific areas of the human body. Each gesture represents a specific area requiring airflow. The first and second columns of Table 2 show the different gestures corresponding to the specific areas of the human body defined in this embodiment. For example, gesture 1 with the left hand represents the head area. The second and third columns show the skeletal key points corresponding to different specific areas of the human body. In this embodiment, the human body is divided into 10 areas, including the head area, shoulder area, chest area, and left arm area, as shown in Table 2. Each specific area is composed of different key points; for example, skeletal key points 1-10 form the head area.

[0051] Table 2. Correspondence between gesture parts and key points

[0052]

[0053] Step 2: Capture image data in real time for the user using a computer vision device; in this embodiment, the real-time image data consists of no more than 30 frames per second of continuous frames, and the following steps are performed on each frame of data.

[0054] Step 3: Preprocess the acquired image data. Specifically:

[0055] The image data is denoised to remove salt-and-pepper noise and Gaussian noise. In addition, considering that the acquired image data is not always frontal or completely of the human body, this embodiment performs centroid symmetry alignment on the denoised image data and performs occlusion enhancement processing on the image before performing alignment and occlusion enhancement processing.

[0056] Step 4 involves feature extraction and human body model prediction processing of the preprocessed image data to obtain the image's confidence value. Specifically, this includes:

[0057] This embodiment uses the BlazePose algorithm to perform operations such as line feature extraction and image segmentation on the current frame image, generating a 960*929*3 pixel matrix. The basic parameter matrix corresponding to the generated image pixel matrix is ​​input into the BlazePose algorithm model for prediction. This invention adopts the top-down method in the BlazePose algorithm, focusing on detecting the bounding boxes of relatively rigid body parts (such as faces or torsos) to obtain the prediction results, i.e., the confidence values ​​of the image.

[0058] Step 5: In human detection and model prediction, some images may be displaced due to algorithm errors or abnormal image data. Therefore, this invention sets a confidence value and a threshold to filter images. In this embodiment, the threshold is set to 0.5. First, the size of the confidence value and the threshold is judged. If the confidence value is greater than or equal to the threshold, i.e., the confidence value ≥ 0.5, the image is retained as a human image and proceeds to step 6; otherwise, the image is discarded.

[0059] Step 6: Obtain the skeletal key points of the human body in the image, locate these skeletal key points, and then connect the located skeletal key points to obtain the final detection result of the human body.

[0060] This embodiment uses the BlazePose algorithm to obtain skeletal key points in human images. The specific principle is as follows: after detecting the human body, the input image is processed using a topological structure. Only a minimum number of key points on the face, hands, and feet are used to estimate the pose interest region, and this is extended to obtain the whole body skeletal nodes.

[0061] The specific positioning method is as follows: a heatmap is generated for each skeletal key point, and the offset of each coordinate is refined to obtain the two-dimensional information coordinates of each skeletal key point, i.e., the parameter i(x,y), where i represents the skeletal node number, and x and y represent the coordinate values ​​of each skeletal node in the coordinate system of the region of interest image. A combination of heatmap, offset, and regression methods is used to directly regress the two-dimensional key point coordinates to the three-dimensional coordinate information i(x,y,z), ensuring the accuracy and precision of the three-dimensional skeletal key point prediction, where z represents the position information of the human body's centroid from the camera.

[0062] Step 7: Recognize and determine the gestures in the image, specifically as follows:

[0063] Finger key points are obtained through MediaPipe. During the MediaPipe detection process, each finger corresponds to four key points. If no key points are identified or fewer than four key points are identified, the finger is set to be closed, and thus the open fingers are determined. Then, the angle γ between the open fingers is calculated using the law of cosines, specifically as shown in equation (1). If the angle between adjacent fingers is less than 90°, it is counted as two fingers; if it is greater than or equal to 90°, it is counted as one finger. Finally, the gesture is determined.

[0064]

[0065] In the above formula, γ represents the angle between adjacent fingers, and a, b, and c represent the lengths of the three sides of the triangle formed between the two fingers.

[0066] The determined gesture is compared with the meaning of each gesture in the gesture library to find the local human body area corresponding to the user's gesture.

[0067] Step 8: Adjust the airflow angle to the user-specified area. Specifically:

[0068] First, calculate the Euclidean distance d between a point in the local human body region corresponding to the user's gesture and any key point on the skeleton in that region. The skeletal key point corresponding to the smallest Euclidean distance is taken as the representative skeletal key point for this region. Then, the air supply angle is adjusted to deliver air to the location of this representative skeletal key point. The mass point is determined by calculating the Euclidean distance d between any two skeletal key points in this local area of ​​the human body.

[0069] Example 2

[0070] This embodiment discloses an air conditioning air supply angle adjustment system based on gesture control. The system specifically includes a gesture library, a video acquisition module, a preprocessing module, a feature extraction module, a human body detector module, a human skeleton key point localization module, a human gesture recognition module, and a region retrieval module. The gesture library stores several gestures related to local areas of the human body, with one gesture representing a local area requiring air supply. The video acquisition module captures and collects image data from the user. The preprocessing module preprocesses the acquired image data and outputs region of interest image data. The feature extraction module extracts features from the preprocessed image data. The human body detector module obtains the confidence value of the image and filters it frame by frame based on the confidence value to obtain the required image data. The human skeleton key point localization module acquires and locates the skeletal key points of the human body in the image. The human gesture recognition module identifies and determines the gestures in the image. The region retrieval module determines the local area of ​​the human body corresponding to the identified gesture, locates the skeletal key points of the determined local area, and determines the precise air supply angle.

[0071] The implementation of the above module is based on steps 2 to 9 in embodiment 1, where the example and application scenario are the same.

[0072] The present invention uses the method and system described in the above embodiments to perform thermal comfort adjustment on a simulated person with disabilities. The specific implementation process is as follows:

[0073] First, the subject turned on the air conditioner and perceived the thermal comfort environment it provided.

[0074] Secondly, the subject enters the system developed in this invention. The subject is familiar with the hand postures corresponding to different areas of the human body beforehand, and then makes corresponding hand postures based on their own thermal comfort sensations or thermal comfort needs for a specific area. The system recognizes the user's hand postures, such as... Figure 4 The figure shows different hand postures identified by the system of the present invention, and personalized thermal comfort adjustment is performed on the corresponding parts based on these different hand postures.

Claims

1. A method for adjusting the air supply angle of an air conditioner based on gesture control, characterized in that, Includes the following steps: Step 1: Establish a skeletal key point model of the human body, and divide the human body into local areas on the model; set up a gesture posture library, which defines several gestures related to local areas of the human body, one of which represents a local area of ​​the human body that needs air supply. Step 2: Use a computer vision device to capture image data of the user. Step 3: Preprocess the acquired image data; Step 4: Perform feature extraction and human body model prediction on the preprocessed image data to obtain the confidence value of the image; Step 5: Determine the relationship between the confidence value and the threshold. If the confidence value is greater than or equal to the threshold, retain the image and proceed to Step 6; otherwise, discard the image. Step 6: Obtain the skeletal key points of the human body in the image, locate these skeletal key points, and connect the located skeletal key points to obtain the final human body detection result. Step 7: Recognize the gestures in the image and determine the gestures; compare the determined gestures with the meanings of each gesture in the gesture pose library to find the local human body area corresponding to the user's gesture. Step 8: Adjust the air supply angle to the user-specified area determined in Step 7; Step 9: Real-time acquisition of image data. For each acquired image frame, process it according to steps 3 to 8.

2. The air conditioning air supply angle adjustment method based on gesture control as described in claim 1, characterized in that, Step 3 specifically includes: performing noise reduction processing on the image data of each frame, as well as alignment and occlusion enhancement processing.

3. The air conditioning air supply angle adjustment method based on gesture control as described in claim 1, characterized in that, Step 4 specifically includes: using the BlazePose algorithm to extract features from the image, generating a pixel matrix, and inputting the basic parameter matrix corresponding to the generated pixel matrix into the model of the BlazePose algorithm for prediction to obtain the confidence value of the frame image.

4. The air conditioning air supply angle adjustment method based on gesture control as described in claim 1, characterized in that, The threshold value in step 5 is 0.

5.

5. The air conditioning air supply angle adjustment method based on gesture control as described in claim 1, characterized in that, The method for locating skeletal key points in step 6 specifically includes: using the BlazePose algorithm to obtain skeletal key points in a human image, generating a heatmap for each skeletal key point and refining the offset of each coordinate, thereby obtaining the two-dimensional information coordinates of each skeletal key point, and then converting the two-dimensional information coordinates into three-dimensional coordinates.

6. The air conditioning air supply angle adjustment method based on gesture control as described in claim 1, characterized in that, The specific steps for determining the gesture in step 7 are as follows: obtain finger key points through MediaPipe, determine the open fingers based on the obtained finger key points, and then calculate the angle between the open fingers using the cosine theorem. If the angle between adjacent fingers is less than 90°, count as 2 fingers; if it is greater than or equal to 90°, count as 1 finger; finally determine the gesture. （1） In the above formula, The angle between adjacent fingers is represented by a, b, and c, which represent the lengths of the three sides of the triangle formed between the two fingers.

7. The air conditioning air supply angle adjustment method based on gesture control as described in claim 1, characterized in that, Step 8 specifically involves: firstly calculating the Euclidean distance between a point in the local area of ​​the human body corresponding to the user's gesture and any skeletal key point in that area; taking the skeletal key point corresponding to the smallest Euclidean distance as the representative skeletal key point of that area; obtaining the position information of the representative skeletal key point; and adjusting the air supply angle to supply air to the location of the representative skeletal key point.

8. A gesture-controlled air conditioning airflow angle adjustment system, characterized in that, The regulation system includes: The gesture library stores several gestures related to specific areas of the human body, with one gesture representing a specific area of ​​the human body that requires airflow. The video capture module is used to capture and collect image data from users. The preprocessing module is used to preprocess the acquired image data and output the region of interest image data; The feature extraction module is used to extract features from the preprocessed image data; The human body detector module is used to acquire the confidence value of the image and filter it frame by frame according to the confidence value to obtain the required image data; The human skeleton key point localization module is used to acquire the skeletal key points of the human body in the image and locate these skeletal key points. Human gesture recognition module, used to recognize and determine gestures in images; The region retrieval module is used to determine the local human body region corresponding to the identified gesture, locate the skeletal key points of the determined local human body region, and determine the precise air delivery angle.

9. A control device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, performs the steps of the method according to any one of claims 1 to 7.

Images