Air conditioner and control method and device thereof, storage medium and computer program product

Abstract

The invention provides an air conditioner and a control method and device thereof, a storage medium and a computer program product, a binocular camera is arranged in the environment where the air conditioner is located and used for collecting images of the environment where the air conditioner is located, and the control method comprises the steps that a first image of the environment where the air conditioner is located is collected through the binocular camera; performing preset processing on the collected first image to obtain a depth map of the first image; determining the position of a target object in the environment according to the obtained depth map of the first image; and controlling the air conditioner to supply air according to the determined position of the target object in the environment. According to the scheme provided by the invention, the specific position of the user in the room can be accurately positioned.

Description

Technical Field

[0001] The present invention relates to the field of control, and in particular to an air conditioner and a control method, device, storage medium and computer program product thereof. Background Art

[0002] As people's living standards improve, the intelligence of air conditioners continues to improve, and more and more air conditioners have the functions of avoiding people and blowing with people. However, most of these functions are based on simple two-dimensional image recognition technology. Although this technology can basically identify the area where a person is located, it is limited to plane recognition and cannot accurately obtain the position of a person in three-dimensional space, resulting in the control effect of the air conditioner not being refined and humane enough. Therefore, it is difficult to optimize and adjust the air conditioner according to the specific situation during operation, and problems such as wind blowing directly on the human body, not blowing to the predetermined position, or blowing other objects by mistake may occur. In order to improve the user experience and meet people's higher requirements for smart home environments, it is necessary to explore and develop new air conditioning control methods based on more advanced image recognition technology to achieve three-dimensional spatial positioning of people or pets, so as to perform more precise intelligent control. Summary of the invention

[0003] The main purpose of the present invention is to overcome the defects of the above-mentioned related technologies and to provide an air conditioner and its control method, device, storage medium and computer program product to solve the problem that the two-dimensional image recognition technology of the air conditioner in the related technology can only obtain the planar position of the user, but cannot accurately judge the distance and height difference between the user and the air conditioner.

[0004] In one aspect, the present invention provides a method for controlling an air conditioner, wherein a binocular camera is provided in an environment where the air conditioner is located, for collecting images of the environment, and the control method comprises: collecting a first image of the environment by the binocular camera; performing preset processing on the collected first image to obtain a depth map of the first image; determining the position of a target object in the environment according to the obtained depth map of the first image; and controlling the air supply of the air conditioner according to the determined position of the target object in the environment. The target object comprises: a person or a pet.

[0005] Optionally, the binocular camera includes: an infrared binocular camera and a CMOS binocular camera; collecting a first image of the environment through the binocular camera includes: collecting a second image of the environment through the CMOS binocular camera at preset time intervals, and determining a current light intensity of the environment based on the collected second image; and collecting the first image of the environment using the infrared binocular camera or the CMOS binocular camera based on the determined current light intensity of the environment.

[0006] Optionally, the captured first image is subjected to preset processing to obtain a depth map of the first image, including: obtaining a relative rotation matrix and a relative translation matrix of a left camera and a right camera of the binocular camera used; performing stereo correction on the first image according to the obtained relative rotation matrix and relative translation matrix of the binocular camera used to obtain a stereo-corrected first image; performing stereo matching on the obtained stereo-corrected first image to obtain a depth map of the first image; and / or determining a position of a target object in the environment according to the obtained depth map of the first image, including: performing target object area segmentation on the first image, and obtaining the position of the target object in the environment according to the target object area obtained by the segmentation and the depth map of the first image.

[0007] Optionally, the first image includes a left image taken by a left camera and a right image taken by a right camera of the binocular camera used; stereo matching is performed on the obtained stereo-corrected first image to obtain a depth map of the first image, including: calculating the matching cost for each pair of corresponding pixels in the left image and the right image; based on the calculated matching cost, aggregating the matching cost within the neighborhood of each pixel to obtain the disparity value d with the minimum matching cost for each pixel, thereby obtaining the disparity map of the first image; based on the obtained disparity map of the first image, a depth map of the first image is calculated.

[0008] Optionally, the first image is segmented into target object regions, and the position of the target object in the environment is obtained based on the segmented target object regions and the depth map of the first image, including: segmenting the first image into target object regions to obtain target object regions; performing an AND operation on the segmented target object regions and the depth map of the first image to obtain a target object depth region; and locating the target object based on the obtained target object depth region to obtain the target object position.

[0009] Optionally, the target object is located according to the obtained target object depth area to obtain the target object position, including: calculating the average value of the depth values ​​of the pixel points in the target object depth area as the Z coordinate of the target object position; converting the coordinates of each pixel in the target object depth area into camera coordinate system coordinates according to the Z coordinate of the target object position and the camera internal parameters of the binocular camera used; converting the coordinates of each pixel in the target object depth area from the camera coordinate system to the world coordinate system to obtain the target object position in the environment.

[0010] Optionally, the air conditioner includes more than two air supply units, and the air supply of the air conditioner is controlled according to the determined target object position in the environment, including: controlling the air supply direction of the two or more air supply units according to the determined target object position in the environment to achieve a wind-following target air supply mode or a wind-avoiding target air supply mode, wherein: the wind-following target air supply mode includes: controlling the air supply direction of the two or more air supply units toward the area where the target object is located according to the determined target object position; the wind-avoiding target air supply mode includes: controlling the air supply direction of the two or more air supply units according to the determined target object position to avoid the area where the target object is located.

[0011] Optionally, according to the determined target object position in the environment, the air supply direction of the more than two air supply units is controlled, including: determining the air supply action area according to the determined target object position in the environment, and controlling the air supply direction of the more than two air supply units according to the determined air supply action area; wherein, determining the air supply action area according to the determined target object position in the environment includes: in a wind-following target air supply mode, according to the detected distance between the target object positions, determining a connected area formed by more than two target object positions as the air supply action area and / or a single area where a single target object position is located as the air supply action area; and / or, in a wind-avoiding target air supply mode, according to the detected distance between the target object positions, determining an area within a second preset distance around a connected area formed by more than two target object positions as the air supply action area and / or an area within a second preset distance around a single area formed by a single target object position as the air supply action area.

[0012] Optionally, it also includes: re-capturing a first image of the environment through the binocular camera at preset time intervals, and performing preset processing on the captured first image to obtain a depth map of the first image; re-determining the position of the target object in the environment based on the obtained depth map of the first image, and adjusting the air conditioning supply based on the re-determined position of the target object in the environment.

[0013] Optionally, when at least one of the following conditions is met, the position of the target object in the environment is re-determined, and the air conditioning air supply is adjusted according to the re-determined position of the target object in the environment: the number of target objects changes; the position difference of any target object in the first images captured twice is greater than a preset threshold; the air supply mode is switched.

[0014] Another aspect of the present invention provides a control device for an air conditioner, wherein a binocular camera is provided in the environment where the air conditioner is located, for collecting images of the environment, and the control device comprises: a collection unit for collecting a first image of the environment through the binocular camera; a processing unit for performing a preset process on the first image collected by the collection unit to obtain a depth map of the first image; a determination unit for determining the position of a target object in the environment according to the depth map of the first image obtained by the processing unit; and a control unit for controlling the air supply of the air conditioner according to the position of the target object in the environment determined by the determination unit. The target object comprises: a person or a pet.

[0015] Optionally, the binocular camera includes: an infrared binocular camera and a CMOS binocular camera; the acquisition unit acquires a first image of the environment through the binocular camera, including: acquiring a second image of the environment through the CMOS binocular camera at preset time intervals, and determining a current light intensity of the environment based on the acquired second image; and acquiring the first image of the environment using the infrared binocular camera or the CMOS binocular camera based on the determined current light intensity of the environment.

[0016] Optionally, the processing unit performs preset processing on the captured first image to obtain a depth map of the first image, including: obtaining a relative rotation matrix and a relative translation matrix of a left camera and a right camera of the binocular camera used; performing stereo correction on the first image according to the obtained relative rotation matrix and relative translation matrix of the binocular camera used to obtain a stereo-corrected first image; performing stereo matching on the obtained stereo-corrected first image to obtain a depth map of the first image; and / or the determination unit determines the position of the target object in the environment according to the depth map of the first image obtained by processing the processing unit, including: performing target object area segmentation on the first image, and obtaining the position of the target object in the environment according to the target object area obtained by the segmentation and the depth map of the first image.

[0017] Optionally, the first image includes a left image taken by a left camera and a right image taken by a right camera of the binocular camera used; the processing unit performs stereo matching on the obtained stereo-corrected first image to obtain a depth map of the first image, including: calculating the matching cost for each pair of corresponding pixels in the left image and the right image; based on the calculated matching cost, aggregating the matching cost within the neighborhood of each pixel to obtain the disparity value of each pixel with the minimum matching cost, thereby obtaining the disparity map of the first image; and calculating the depth map of the first image based on the obtained disparity map of the first image.

[0018] Optionally, the determination unit performs target object region segmentation on the first image, and obtains the target object position in the environment based on the segmented target object region and the depth map of the first image, including: performing target object region segmentation on the first image to obtain the target object region; performing an AND operation on the segmented target object region and the depth map of the first image to obtain the target object depth region; and performing target object positioning based on the obtained target object depth region to obtain the target object position.

[0019] Optionally, the determination unit performs target object positioning according to the obtained target object depth area to obtain the target object position, including: calculating the average value of the depth values ​​of the pixel points in the target object depth area as the Z coordinate of the target object position; converting the coordinates of each pixel in the target object depth area into camera coordinate system coordinates according to the Z coordinate of the target object position and the camera internal parameters of the binocular camera used; converting the coordinates of each pixel in the target object depth area from the camera coordinate system to the world coordinate system to obtain the target object position in the environment.

[0020] Optionally, the air conditioner includes more than two air supply units, and the control unit controls the air supply of the air conditioner according to the determined position of the target object in the environment, including: controlling the air supply direction of the two or more air supply units according to the determined position of the target object in the environment, to achieve a wind-following target air supply mode or a wind-avoiding target air supply mode, wherein: the wind-following target air supply mode includes: controlling the air supply direction of the two or more air supply units toward the area where the target object is located according to the determined position of the target object; the wind-avoiding target air supply mode includes: controlling the air supply direction of the two or more air supply units according to the determined position of the target object to avoid the area where the target object is located.

[0021] Optionally, the control unit controls the air supply direction of the two or more air supply units according to the determined target object position in the environment, including: determining the air supply action area according to the determined target object position in the environment, and controlling the air supply direction of the two or more air supply units according to the determined air supply action area; wherein, determining the air supply action area according to the determined target object position in the environment includes: in a wind-following target air supply mode, determining a connected area formed by more than two target object positions as the air supply action area and / or a single area where a single target object position is located as the air supply action area according to the detected distance between the target object positions; and / or, in a wind-avoiding target air supply mode, determining an area within a second preset distance around a connected area formed by more than two target object positions as the air supply action area and / or an area within a second preset distance around a single area formed by a single target object position as the air supply action area according to the detected distance between the target object positions.

[0022] Optionally, the acquisition unit is further used to: re-capture the first image of the environment through the binocular camera at preset time intervals; the processing unit is further used to: perform preset processing on the first image re-captured by the acquisition unit to obtain a depth map of the first image; the determination unit is further used to: re-determine the position of the target object in the environment based on the obtained depth map of the first image; the control unit is further used to: adjust the air supply of the air conditioner based on the position of the target object in the environment re-determined by the determination unit.

[0023] Optionally, when at least one of the following conditions is met, the determination unit redetermines the position of the target object in the environment based on the first image re-captured by the acquisition unit, and the control unit adjusts the air conditioning air supply based on the position of the target object in the environment re-determined by the determination unit: the number of target objects changes; the position difference between any target object in the first images captured twice is greater than a preset threshold; the air supply mode is switched.

[0024] Another aspect of the present invention provides a storage medium having a computer program stored thereon, wherein the program implements the steps of any of the aforementioned methods when executed by a processor.

[0025] In another aspect, the present invention provides an air conditioner, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the aforementioned methods when executing the program.

[0026] In another aspect, the present invention provides an air conditioner, comprising any of the aforementioned control devices.

[0027] In yet another aspect, the present invention provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the aforementioned methods are implemented.

[0028] According to the technical solution of the present invention, binocular vision technology is introduced to obtain depth information in three-dimensional space, thereby realizing accurate positioning of the user's specific position in the room. Through this three-dimensional positioning capability, the air conditioner can adjust the wind direction according to the user's specific position; through binocular vision technology, multiple targets can be identified at the same time, and positioned and tracked respectively, thereby realizing intelligent control of multiple targets; the air supply direction is automatically adjusted according to the position of the located target object, realizing all-round wind following the target and wind avoiding the target, improving the user's comfort, improving the energy efficiency and service life of the air conditioner, and having significant social and economic benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0030] Figure 1 is a method schematic diagram of an embodiment of the air conditioner control method provided by the present invention;

[0031] Figure 2a A schematic diagram showing wind following the target is shown;

[0032] Figure 2b A schematic diagram of a wind avoidance target is shown;

[0033] Figure 3 A schematic diagram showing the determination rules for connected regions and single regions is shown;

[0034] Figure 4a A schematic diagram of the air supply action area in the air supply mode of wind following the target is shown;

[0035] Figure 4b A schematic diagram of the air supply action area in the air supply mode of the wind avoidance target is shown;

[0036] Figure 5a A schematic diagram of the air supply mode of wind following the target when there are more than two air supply action areas is shown;

[0037] Figure 5b A schematic diagram showing an air supply mode of a wind avoidance target when there are more than two air supply action areas;

[0038] Figure 6 A flowchart showing a specific implementation of the step of performing preset processing on the acquired first image to obtain a depth map of the first image according to the present invention;

[0039] Figure 7 is a method schematic diagram of a specific embodiment of the air conditioner control method provided by the present invention;

[0040] Figure 8 It is a structural block diagram of an embodiment of the air conditioner control device provided by the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0043] The invention provides a control method for an air conditioner.

[0044] A binocular camera is arranged in the environment where the air conditioner is located to collect images of the environment, and the binocular camera includes: an infrared binocular camera and a COMS binocular camera. The COMS binocular camera is specifically a binocular camera using a COMS image sensor. The binocular camera is installed in a suitable position in the room where the air conditioner is located to ensure that its shooting range can cover the entire working area of ​​the air conditioner. For example, it is installed at a high place, facing the main activity area of ​​the room, so as to better capture all activities in the room.

[0045] Figure 1 It is a method schematic diagram of an embodiment of the air conditioner control method provided by the present invention.

[0046] like Figure 1 As shown, according to one embodiment of the present invention, the air conditioner control method at least includes step S110, step S120, step S130 and step S140.

[0047] Step S110: collecting a first image of the environment through the binocular camera.

[0048] In a specific implementation, a second image of the environment is collected by a CMOS binocular camera at a preset time interval, and the current light intensity of the environment is determined based on the collected second image. Based on the determined current light intensity of the environment, an infrared binocular camera or a CMOS binocular camera is used to collect a first image of the environment.

[0049] Specifically, a second image of the environment is collected by a CMOS binocular camera at every preset time interval, and the average brightness value of the collected second image is calculated using image processing technology to determine the light intensity of the current environment. Then, according to the determined current light intensity of the environment, an infrared binocular camera or a CMOS binocular camera is used to collect the first image of the environment. It is determined whether the current light intensity of the environment is greater than a preset threshold. If it is determined that the current light intensity is less than or equal to the preset threshold, it means that the light intensity is low, and the infrared binocular camera is used to collect the second image of the environment; if it is determined that the current light intensity is greater than the preset threshold, it means that the light intensity is high, and the CMOS binocular camera is used to collect the second image of the environment. For example, when the light intensity is low (at night), an infrared binocular camera is used for image collection, and when the light intensity is high (during the day), a CMOS binocular camera is used for image collection.

[0050] Step S120: performing preset processing on the collected first image to obtain a depth map of the first image.

[0051] Figure 6 FIG. 1 is a flowchart showing a specific implementation of the step of performing preset processing on the first image acquired according to the present invention to obtain a depth map of the first image. Figure 6 As shown, step S120 includes step S121, step S122 and step S123.

[0052] Step S121, obtaining the relative rotation matrix and relative translation matrix of the left and right cameras of the binocular camera used.

[0053] Specifically, before positioning a person or a pet, the captured image is first subjected to preset processing, including image correction, image matching, and depth calculation.

[0054] Image correction includes distortion correction and stereo correction. Specifically, the first image is subjected to distortion correction according to the radial distortion coefficient and tangential distortion coefficient of the camera; lens distortion includes radial distortion and tangential distortion.

[0055] Radial distortion: causes the edge of the image to deform and can be corrected using the following formula:

[0056] x distort =x(1+k1r 2 +k2r 4 +k3r 6 )

[0057] y distort =y(1+k1r 2 +k2r 4 +k3r 6 )

[0058] Among them, x distort ,y distort are the corrected x and y coordinates, k1, k2, k3 are the radial distortion coefficients, and r is the distance from the pixel to the center of the image.

[0059] Tangential distortion: caused by the lens not being completely parallel to the image plane, and can be corrected by the following formula:

[0060] x distort =x+[2p1xy+p2(r 2 +2x 2 )]

[0061] y distort =y+[p1(r 2 +2y 2 )+2p2xy]

[0062] Among them, x distort ,y distort are the corrected x and y coordinates, p1 and p2 are the tangential distortion coefficients, and r is the distance from the pixel to the center of the image.

[0063] According to the radial distortion coefficient and the tangential distortion coefficient of the camera, the first image can be corrected for distortion using the above formula.

[0064] The camera intrinsic parameters of the binocular camera used are calibrated to obtain the camera intrinsic parameters of the binocular camera used. Specifically, the camera intrinsic parameter matrix K and the distortion coefficient can be obtained through a calibration board (such as a chessboard). The camera intrinsic parameter matrix K includes the focal length f x 、f y and the principal point coordinates c x 、c y .

[0065]

[0066] After obtaining the camera internal parameters of the binocular camera used, the binocular camera used is calibrated in stereo to obtain the relative rotation matrix and relative translation matrix of the left and right cameras of the binocular camera used. The stereo calibration of the binocular camera used may specifically include:

[0067] a. Prepare the 3D point coordinates of the calibration plate. The position of each corner point on the calibration plate in the world coordinate system is known and can be expressed as (X, Y, Z);

[0068] b. Correct the image using the camera's internal parameters and distortion coefficients, and calculate the coordinates of the corner points extracted from each pair of images taken by the left and right cameras in their respective image coordinate systems.

[0069] c. According to the calculated coordinates of the corner points extracted from each pair of images taken by the left and right cameras in their respective image coordinate systems, optimize the extrinsic parameters of the left and right cameras (extrinsic parameters refer to the translation matrix and rotation matrix of the camera coordinate system relative to the world coordinate system), and calculate the relative rotation matrix R and relative translation matrix T between the left and right cameras.

[0070] The objective function for minimizing the reprojection error during stereo calibration is:

[0071]

[0072] in, It means to sum the difference between the coordinates of the Mth corner point of the Nth pair of images and the reprojected coordinates. R represents the rotation matrix between the left and right cameras, and T represents the translation matrix between the left and right cameras. represents the 3D position of the jth corner point in the i-th image in the world coordinate system, represents the position of the jth corner point in the image coordinate system in the i-th left camera image (referred to as the left image), Indicates the position of the jth corner point in the image coordinate system of the i-th right camera image (referred to as the right image). The project function projects the point X in the three-dimensional space onto the two-dimensional image plane, which is accomplished through the camera's intrinsic and extrinsic matrix. The formula is as follows:

[0073] x=K·[R·X+T]

[0074] Wherein, R and T are the relative rotation matrix and relative translation matrix between the left and right cameras respectively, and K is the camera intrinsic parameter matrix.

[0075] The two camera extrinsic parameters R1, T1, R2, and T2 are optimized by an iterative optimization algorithm (such as the Levenberg-Marquardt algorithm) to minimize the reprojection error.

[0076] Calculate the relative rotation matrix R:

[0077] Calculate the relative translation matrix T: T = T2-R·T

[0078] R1 and T1 are the external parameters of the left camera, which represent the rotation matrix and translation matrix relative to the world coordinate system. R2 and T2 are the external parameters of the right camera, which represent the rotation matrix and translation matrix relative to the world coordinate system. In the dual-camera system, the world coordinate system is used as the intermediate point of the transformation to obtain the relative rotation matrix R and the relative translation matrix T.

[0079] Step S122: performing stereoscopic correction on the first image according to the acquired relative rotation matrix and relative translation matrix of the binocular camera used to obtain a stereoscopically corrected first image.

[0080] Specifically, after obtaining the relative rotation matrix and relative translation matrix of the left and right cameras of the binocular camera used, stereo correction is performed on the first image according to the relative rotation matrix and relative translation matrix to obtain the stereo-corrected first image, that is, the left and right camera images in the first image are transformed to the same plane.

[0081] Stereo Rectification is to transform the images taken by the two cameras in the stereo camera system to the same plane, that is, according to the relative rotation matrix and the relative translation matrix, the left and right camera images are transformed to the same plane so that the corresponding pixels are on the same row, thereby simplifying the parallax calculation and depth estimation. The main goal of stereo rectification is to align the images so that the cameras are aligned and the effects of non-parallel optical axes are eliminated.

[0082] The core of R1′R2′ binocular stereo vision is to calculate the disparity of each pixel by comparing the images captured by the left and right cameras, and then infer the depth of the point. In order to accurately calculate the depth information, it is necessary to ensure that the internal and external parameters of the two cameras are accurately known. Stereo calibration not only needs to calibrate the intrinsic parameters of each camera, but also needs to determine the relative position (translation vector) and relative posture (rotation matrix) between the two cameras. These external parameters are the basis for calculating disparity and depth information. Since the two cameras are installed in different positions, the images taken will have certain perspective differences. Stereo calibration aligns the perspectives of the two cameras through correction transformation, so that each pair of pixels corresponds to the same object point in the two images.

[0083] Step S123: performing stereo matching on the obtained first image after stereo correction to obtain a depth map of the first image.

[0084] Specifically, the stereo matching may include: image preprocessing and image matching. Before stereo matching, the left and right camera images are preprocessed to improve the accuracy of matching. The image is corrected using the distortion parameters obtained by camera calibration to eliminate lens distortion. The image is filtered and at least one of contrast enhancement is performed to improve image quality. Image matching is to extract features from the image for matching. The extracted features may specifically include at least one of pixel values, gradients, and textures. Since the two images captured at the same time are very similar, pixel values ​​can be used as features to reduce the amount of calculation.

[0085] Image matching includes matching cost calculation and matching cost aggregation. First, matching cost calculation is performed. The first image includes a left image taken by a left camera of the binocular camera used and a right image taken by a right camera. Specifically, a matching cost calculation is performed for each pair of corresponding pixels in the left image and the right image.

[0086] In stereo image matching, the pixels of the left image and the right image (the image taken by the left camera, referred to as the left image, and the image taken by the right camera, referred to as the right image) are usually represented as points in the same scene taken at the same time, but their positions in the image are slightly different due to the different camera perspectives. Each pair of pixels refers to a pixel in the left image (or a window centered on the pixel) and a corresponding pixel in the same scene in the right image (or a window centered on the pixel). In order to find such matching points, the area in the right image that is most similar to the pixels in the left image can be searched along the disparity direction.

[0087] The purpose of calculating the matching cost is to measure whether a certain pixel and its neighborhood in two images are similar. In a specific implementation, the correlation between two pixel blocks can be calculated using Normalized Cross-Correlation (NCC).

[0088]

[0089] Among them, I1(x+i,y+j) and I2(x+i+d,y+j) represent the pixel values ​​of the left image and the right image at positions (x+i,y+j) and (x+i+d,y+j), respectively. and are the means of the left image and the right image in the window respectively, w is the window radius, the window size is (2w+1)*(2w+1), and d is the disparity.

[0090] The best matching point is found by calculating the normalized cross-correlation coefficient of the pixel values ​​within the window. The cross-correlation coefficient value ranges from -1 to 1, with 1 indicating complete similarity and -1 indicating complete dissimilarity.

[0091] After calculating the matching cost for each pair of corresponding pixels in the left image and the right image, based on the calculated matching cost, the matching cost within the neighborhood of each pixel is aggregated to obtain the disparity value d with the minimum matching cost for each pixel, thereby obtaining the disparity map of the first image, and based on the obtained disparity map of the first image, a depth map of the first image is calculated.

[0092] Specifically, by aggregating the matching costs within the neighborhood of each pixel (the neighborhood refers to the pixel and its eight adjacent pixels, forming a 3×3 pixel area), the impact of noise on the matching results is reduced and the accuracy of disparity estimation is improved. Filtering technology is used to smooth the matching costs, making the disparity map more accurate and coherent.

[0093] In a specific implementation, bilateral filtering can be used to smooth the matching cost. Bilateral filtering not only considers the spatial distance, but also the similarity between pixel values. When calculating the weighted average, both the spatial distance of the pixels and the similarity of the pixel values ​​are considered, so as to better preserve the image edge.

[0094] Specifically, the following formula can be used to calculate the matching cost of each pixel after aggregation:

[0095]

[0096] is the matching cost after aggregation, C(q) is the initial matching cost, N(p) is the neighborhood of pixel p, q is any pixel in the neighborhood N(p) of pixel p, pq represents the pixel coordinate distance, _I(p) and I(q) represent the grayscale values ​​of pixel p and q, σ s is the parameter that controls the effect of spatial distance, σ r is a parameter that controls the influence of pixel value similarity, W p is the normalization coefficient, ensuring the weight is 1.

[0097] After the matching cost aggregation, the disparity value d with the minimum matching cost is selected for each pixel to obtain the disparity map.

[0098] Calculate the baseline distance B: B = ‖T‖

[0099] Calculate the depth map:

[0100] Where: T represents the baseline vector between the two cameras, that is, the distance vector between the two cameras in the horizontal or spatial direction, Z is the depth (the distance from the camera), f is the focal length of the camera after stereo correction, B is the baseline distance between the left and right cameras, and d is the disparity (the horizontal position difference between the corresponding points in the left and right images).

[0101] Step S130: determining the position of the target object in the environment according to the obtained depth map of the first image.

[0102] The target object includes: a person or a pet. Specifically, the first image is segmented into target object regions, and the position of the target object in the environment is obtained according to the segmented target object regions and the depth map of the first image.

[0103] After stereo matching, a depth map is obtained, but it is impossible to know which part of the depth map belongs to the target object, so the target object area needs to be segmented. Specifically, spatial positioning includes target object segmentation and target object positioning (such as character segmentation and character positioning). First, the target object area of ​​the first image is segmented to obtain the target object area (such as character segmentation), and then the segmented target object area is ANDed with the depth map of the first image to obtain the target object depth area, and then the target object is positioned according to the obtained target object depth area to obtain the target object position.

[0104] In a specific embodiment, the encoder-decoder architecture of the U-Net neural network is used to segment the target object region. The encoder-decoder architecture of the U-Net neural network enables it to capture high-level features while maintaining high-resolution feature maps. This architecture helps to accurately segment complex shapes and details. U-Net introduces jump connections between each encoder layer and the corresponding decoder layer. These connections enable the network to utilize high-resolution features lost during the encoding process during the decoding process, enhance the accuracy of segmentation and the retention of boundary details, and facilitate accurate target object contour segmentation, such as human body contour segmentation.

[0105] After performing target object region segmentation on the first image, an AND operation is performed on the segmented target object region and the depth map of the first image to obtain a target object depth region; an average value of the depth values ​​of the pixel points in the target object depth region is calculated as the Z coordinate of the target object position, and the coordinates (X, Y, Z) of each pixel point in the world coordinate system are obtained in combination with the camera internal parameters.

[0106] The average value Z of the depth values ​​of the pixels in the depth area of ​​the target object is calculated according to the following formula:

[0107]

[0108] Where D is the depth area, u and v are the horizontal and vertical coordinates of the pixel respectively, d is the disparity value of each pair of pixels, and n is the number of pixels in the depth area of ​​the target object. Combined with the camera internal parameters, the coordinates (u, v) and the depth average Z are converted into camera coordinate system coordinates to obtain the coordinates (X, Y, Z) of each pixel:

[0109]

[0110] Then convert the obtained target object area coordinates (X, Y, Z) from the camera coordinate system to the world coordinate system to obtain the target object position coordinates:

[0111]

[0112] Among them, Xw, Yw, and Zw are the position coordinates of the target object converted from the camera coordinate system to the world coordinate system, and R and T are the relative rotation matrix and relative translation matrix between the left and right cameras, respectively.

[0113] Step S140, controlling the air supply of the air conditioner according to the determined position of the target object in the environment.

[0114] In a specific embodiment, the air conditioner includes more than two air supply units, and the air supply direction can be controlled by a guide plate, which can realize an air supply mode of wind following a target (e.g., wind following a person) and an air supply mode of wind avoiding a target (e.g., wind avoiding a person), so that the temperature of different rooms or areas can be adjusted as needed, thereby improving the user's comfort and energy saving effect. It can effectively reduce energy consumption, reduce operating costs, and be more environmentally friendly.

[0115] Specifically, according to the determined position of the target object in the environment, the air supply direction of the two or more air supply units is controlled to realize the wind-following-target air supply mode or the wind-avoiding-target air supply mode. The wind-following-target air supply mode includes: determining the air supply action area according to the determined position of the target object in the environment, toward the area where the target object is located, wherein the area where the target object is located is an area within a first preset distance from the detected target object position. For example, Figure 2a A schematic diagram showing the wind following the target is shown. Figure 2a As shown, the dot represents the target object. When the wind follows the target (for example, the wind blows with a person), the air supply direction is toward the area where the target object (human body or pet) is located. In this mode, users can quickly feel the air conditioning effect, especially in hot or cold weather, avoiding ineffective cooling or heating of unmanned areas and saving energy. The wind-avoiding target air supply mode includes: according to the determined target object position, controlling the air supply direction of the two or more air supply units to avoid the area where the target object is located, wherein the area where the target object is located is an area within a first preset distance from the detected target object position, such as toward the surrounding areas of the target object area. For example, Figure 2b A schematic diagram of a wind avoidance target is shown, which can be referred to Figure 2b As shown in the figure, the dot represents the target object. When the wind avoids the target (for example, the wind avoids people), the air supply direction is toward the surroundings of the target object area. In this mode, direct wind is avoided, reducing the discomfort caused by it, such as headaches, dry eyes, colds, etc. Through indirect air supply, the indoor temperature is more uniform and the overall comfort is improved. It is especially suitable for people who need to stay in an air-conditioned environment for a long time, such as office workers, the elderly and children. It can reach a comfortable temperature relatively quickly and avoid ineffective cooling or heating in unmanned areas, saving energy.

[0116] Preferably, the air supply action area is determined according to the position of the target object in the determined environment, and the air supply direction of the more than two air supply units is controlled according to the determined air supply action area to achieve an air supply mode in which the wind follows the target (for example, the wind follows the person) or an air supply mode in which the wind avoids the target (for example, the wind avoids the person).

[0117] In a specific embodiment, the air supply action area is determined according to the detected target object position, including, in the wind-following air supply mode, determining the connected area formed by more than two target object positions as the air supply action area and / or the single area where a single target object position is located as the air supply action area according to the distance between the detected target object positions. In the wind-avoiding air supply mode, determining the area within a second preset distance around the connected area formed by more than two target object positions as the air supply action area and / or the area within a second preset distance around the single area formed by a single target object position as the air supply action area according to the distance between the detected target object positions. Wherein, the single area where the single target object position is located is the area within the first preset distance from the target object position.

[0118] Specifically, if the distance d between any two or more single target object positions is less than the set distance threshold T, then in the wind-following air supply mode, a connected area within a first preset distance from the two or more target object positions is determined as the air supply action area, or in the wind-avoiding air supply mode, an area within a second preset distance around the connected area within the first preset distance from the two or more target object positions is determined as the air supply action area; if the distance d between any single target object position and other single target object positions is greater than or equal to the set distance threshold T, then in the wind-following air supply mode, a single area within the first preset distance from the single target object position is determined as the air supply action area, or in the wind-avoiding air supply mode, an area within a second preset distance around the single area within the first preset distance from the single target object position is determined as the air supply action area. The distance between a connected area and a connected area or a single area is determined by the two target object positions with the smallest distance.

[0119] Figure 3 The figure shows the schematic diagram of the determination rules of connected areas and single areas. Figure 3 As shown in the figure, the dot is the detected target object position (i.e., the human body area shown in the figure), and the surrounding shadow area (i.e., the wind blowing with the person area shown in the figure) is a single area where a single target object position is determined or a connected area formed by two or more target object positions. Set a distance threshold T. When the distance between the positions of two people is less than or equal to T, it can be considered as a connected area. When the distance between the positions of two people is greater than T, it is considered as a single area.

[0120] Figure 4a Figure 2 shows a schematic diagram of the air supply action area in the air supply mode of wind following the target. Figure 4a As shown in FIG, in the wind-follows-the-target air supply mode, the air supply action area is a connected area formed by the positions of two or more target objects, or a single area where a single target object is located, that is, Figure 4a Area 1 (i.e. the area where the wind blows with people) is shown in . Figure 4b FIG. 1 shows a schematic diagram of the air supply action area in the air supply mode of the wind avoidance target. Figure 4b As shown, in the air supply mode of the wind avoidance target, the air supply action area is the area within the second preset distance around the connected area formed by the positions of more than two target objects, or the area within the second preset distance around a single area where a single target object is located, that is, Figure 4b The area 2 shown in FIG. 1 (i.e., the area where people are sheltered from the wind).

[0121] In a specific embodiment, the air supply direction of the more than two air supply units is controlled according to the determined air supply action area, including: when there is only one air supply action area, each air supply unit supplies air to the partial area closest to the air supply action area; that is, when there is only one connected area and / or single area, each air supply unit supplies air to the partial area closest to the corresponding air supply action area of ​​the connected area and / or single area (the connected area or single area when the wind avoids the target, and the area within a second preset distance around the connected area or single area when the wind avoids the target).

[0122] When there are more than two air supply action areas, a corresponding number of air supply units are allocated to supply air to the two or more air supply action areas according to the area ratio of the two or more air supply action areas. That is, when there are more than two connected areas and / or single areas, a corresponding number of air supply units are allocated to supply air to the corresponding air supply action areas of the two or more connected areas and / or single areas (the two or more connected areas or single areas when the wind avoids the target, and the area within the second preset distance around the two or more connected areas or single areas when the wind avoids the target) according to the area ratio of the two or more connected areas and / or single areas.

[0123] For example, Figure 4a The figure shows the schematic diagram of the air supply mode when there is only one air supply action area. Figure 4b A schematic diagram showing the air supply pattern of the wind avoidance target when there is only one air supply action area. Figure 4a , Figure 4bAs shown, assuming that there are four independent air supply units, when there is only one connected area and / or a single area, each air supply unit is responsible for a part of the air supply action area. When the action area is relatively concentrated, the wind direction of each air outlet is fixed. When the action area is relatively dispersed, in order to avoid the wind from different air outlets interfering with each other in the air, each air outlet is responsible for a part of the area. When the action area is relatively concentrated, the area that each air supply unit is responsible for is very small and can be considered fixed. When the action area is relatively dispersed, each air supply unit is responsible for more areas. In order to cool down as evenly as possible, the air supply unit will sweep the air back and forth or around the annular area.

[0124] When there are more than two action areas, a corresponding number of air supply units are allocated according to the area ratio of the air supply action areas. Figure 5a The figure shows the schematic diagram of the air supply mode when there are more than two air supply action areas. Figure 5b A schematic diagram showing the air supply pattern of a wind avoidance target when there are more than two air supply action areas.

[0125] Optionally, it also includes: re-capturing a first image of the environment through the binocular camera at preset time intervals, and performing preset processing on the captured first image to obtain a depth map of the first image; re-determining the position of the target object in the environment based on the obtained depth map of the first image, and adjusting the air conditioning supply based on the re-determined position of the target object in the environment.

[0126] Specifically, the image acquisition frame rate is set, and when the latest image frame acquisition is completed, stereo matching and character segmentation are repeated to obtain the number and position of people in the visible area. The position of the target object in the environment is re-determined, and according to the re-determined position of the target object in the environment, the air supply conditions of the air conditioner are adjusted as follows:

[0127] The number of target objects (number of people) at the old and new moments changes, with new people entering or existing people leaving; the number of target objects remains unchanged, and the position difference of any target object in the first images acquired twice is greater than a preset threshold; the air supply mode switches, that is, from wind following the target to wind avoiding the target or from wind avoiding the target to wind following the target.

[0128] In order to clearly illustrate the technical solution of the present invention, the execution process of the air conditioner control method provided by the present invention is described below with reference to a specific embodiment.

[0129] Figure 7 FIG. 1 is a schematic diagram of a specific embodiment of the air conditioning control method provided by the present invention. Figure 7As shown, images are collected by a CMOS camera at preset time intervals (for example, every 5 minutes) to analyze the light intensity. The lighting conditions of the current environment are determined. According to whether the calculated light intensity value is greater than the preset threshold, it is determined whether to use an infrared binocular camera or a CMOS binocular camera for image acquisition. After image acquisition, the characters are segmented, and the image is distorted and corrected. Stereo correction is performed through stereo calibration. The disparity map is calculated through stereo matching to obtain the depth map. According to the target object area obtained by character segmentation, the character space is located in combination with the depth map to obtain the target object position. Intelligent air supply is performed according to the target object position. When the mode is switched, or when it is determined that the number of people at the new and old times has changed, or when there are new people entering or the original people leave or the number of people remains unchanged, or when the position difference of any person in the first image collected twice is greater than the preset threshold, the air supply action area is re-determined to plan the air supply path, and the air conditioning operation parameters are adjusted.

[0130] The invention also provides a control device for an air conditioner.

[0131] A binocular camera is arranged in the environment where the air conditioner is located to collect images of the environment, and the binocular camera includes: an infrared binocular camera and a COMS binocular camera. The COMS binocular camera is specifically a binocular camera using a COMS image sensor. The binocular camera is installed in a suitable position in the room where the air conditioner is located to ensure that its shooting range can cover the entire working area of ​​the air conditioner. For example, it is installed at a high place, facing the main activity area of ​​the room, so as to better capture all activities in the room.

[0132] Figure 8 FIG. 1 is a structural block diagram of an embodiment of the air conditioner control device provided by the present invention. Figure 8 As shown, the control device 100 includes: a collection unit 110 , a processing unit 120 , a determination unit 130 and a control unit 140 .

[0133] The acquisition unit 110 is used to acquire a first image of the environment through the binocular camera.

[0134] In a specific embodiment, the binocular camera includes: an infrared binocular camera and a CMOS binocular camera; the acquisition unit 110 acquires a first image of the environment through the binocular camera, including: acquiring a second image of the environment through the CMOS binocular camera at preset time intervals, and determining a current light intensity of the environment based on the acquired second image; and acquiring the first image of the environment using the infrared binocular camera or the CMOS binocular camera based on the determined current light intensity of the environment.

[0135] Specifically, a second image of the environment is collected by a CMOS binocular camera at every preset time interval, and the average brightness value of the collected second image is calculated using image processing technology to determine the light intensity of the current environment. Then, according to the determined current light intensity of the environment, an infrared binocular camera or a CMOS binocular camera is used to collect the first image of the environment. It is determined whether the current light intensity of the environment is greater than a preset threshold. If it is determined that the current light intensity is less than or equal to the preset threshold, it means that the light intensity is low, and the infrared binocular camera is used to collect the second image of the environment; if it is determined that the current light intensity is greater than the preset threshold, it means that the light intensity is high, and the CMOS binocular camera is used to collect the second image of the environment. For example, when the light intensity is low (at night), an infrared binocular camera is used for image collection, and when the light intensity is high (during the day), a CMOS binocular camera is used for image collection.

[0136] The processing unit 120 is used to perform preset processing on the first image acquired by the acquisition unit to obtain a depth map of the first image.

[0137] In a specific implementation, the processing unit performs preset processing on the collected first image to obtain a depth map of the first image, including: obtaining a relative rotation matrix and a relative translation matrix of a left camera and a right camera of the binocular camera used; performing stereo correction on the first image according to the obtained relative rotation matrix and relative translation matrix of the binocular camera used to obtain a stereo-corrected first image; and performing stereo matching on the obtained stereo-corrected first image to obtain a depth map of the first image.

[0138] Specifically, before locating the target object, the collected image is first subjected to preset processing, including: image correction, image matching and depth calculation.

[0139] Image correction includes distortion correction and stereo correction. Specifically, the first image is subjected to distortion correction according to the radial distortion coefficient and tangential distortion coefficient of the camera; lens distortion includes radial distortion and tangential distortion.

[0140] Radial distortion: causes the edge of the image to deform and can be corrected using the following formula:

[0141] x distort =x(1+k1r 2 +k2r 4 +k3r 6 )

[0142] y distort =y(1+k1r 2 +k2r 4 +k3r 6 )

[0143] Among them, x distort ,y distort are the corrected x and y coordinates, k1, k2, k3 are the radial distortion coefficients, and r is the distance from the pixel to the center of the image.

[0144] Tangential distortion: caused by the lens not being completely parallel to the image plane, and can be corrected by the following formula:

[0145] x distort =x+[2p1xy+p2(r 2 +2x 2 )]

[0146] y distort =y+[p1(r 2 +2y 2 )+2p2xy]

[0147] Among them, x distort ,y distort are the corrected x and y coordinates, p1 and p2 are the tangential distortion coefficients, and r is the distance from the pixel to the center of the image.

[0148] According to the radial distortion coefficient and the tangential distortion coefficient of the camera, the first image can be corrected for distortion using the above formula.

[0149] The camera intrinsic parameters of the binocular camera used are calibrated to obtain the camera intrinsic parameters of the binocular camera used. Specifically, the camera intrinsic parameter matrix K and distortion coefficients (radial distortion coefficient and tangential distortion coefficient) can be obtained through a calibration board (such as a chessboard). The camera intrinsic parameter matrix K includes the focal length f x 、f y and the principal point coordinates c x 、c y .

[0150]

[0151] After obtaining the camera internal parameters of the binocular camera used, the binocular camera used is calibrated in stereo to obtain the relative rotation matrix and relative translation matrix of the left and right cameras of the binocular camera used. The stereo calibration of the binocular camera used may specifically include:

[0152] a. Prepare the 3D point coordinates of the calibration plate. The position of each corner point on the calibration plate in the world coordinate system is known and can be expressed as (X, Y, Z);

[0153] b. Correct the image using the camera's internal parameters and distortion coefficients, and calculate the coordinates of the corner points extracted from each pair of images taken by the left and right cameras in their respective image coordinate systems.

[0154] c. According to the calculated coordinates of the corner points extracted from each pair of images taken by the left and right cameras in their respective image coordinate systems, optimize the extrinsic parameters of the left and right cameras (extrinsic parameters refer to the translation matrix and rotation matrix of the camera coordinate system relative to the world coordinate system), and calculate the relative rotation matrix R and relative translation matrix T between the left and right cameras.

[0155] The objective function for minimizing the reprojection error during stereo calibration is:

[0156]

[0157] in, It means to sum the difference between the coordinates of the Mth corner point of the Nth pair of images and the reprojected coordinates. R represents the rotation matrix between the left and right cameras, and T represents the translation matrix between the left and right cameras. represents the 3D position of the jth corner point in the i-th image in the world coordinate system, represents the position of the jth corner point in the image coordinate system in the i-th left camera image (referred to as the left image), Indicates the position of the jth corner point in the image coordinate system of the i-th right camera image (referred to as the right image). The project function projects the point X in the three-dimensional space onto the two-dimensional image plane, which is accomplished through the camera's intrinsic and extrinsic matrix. The formula is as follows:

[0158] x=K·[R·X+T]

[0159] Wherein, R and T are the relative rotation matrix and relative translation matrix between the left and right cameras respectively, and K is the camera intrinsic parameter matrix.

[0160] The two camera extrinsic parameters R1, T1, R2, and T2 are optimized by an iterative optimization algorithm (such as the Levenberg-Marquardt algorithm) to minimize the reprojection error.

[0161] Calculate the relative rotation matrix R:

[0162] Calculate the relative translation matrix T: T = T2-R·T

[0163] R1 and T1 are the extrinsic parameters of the left camera, which represent the rotation matrix and translation matrix relative to the world coordinate system. R2 and T2 are the extrinsic parameters of the right camera, which represent the rotation matrix and translation matrix relative to the world coordinate system. In the dual-camera system, the world coordinate system is used as the intermediate point of the transformation to obtain the relative rotation matrix R and the relative translation matrix T.

[0164] Step S122: performing stereoscopic correction on the first image according to the acquired relative rotation matrix and relative translation matrix of the binocular camera used to obtain a stereoscopically corrected first image.

[0165] Specifically, after obtaining the relative rotation matrix and relative translation matrix of the left and right cameras of the binocular camera used, stereo correction is performed on the first image according to the relative rotation matrix and relative translation matrix to obtain the stereo-corrected first image, that is, the left and right camera images in the first image are transformed to the same plane.

[0166] Stereo Rectification is to transform the images taken by the two cameras in the stereo camera system to the same plane, that is, according to the relative rotation matrix and the relative translation matrix, the left and right camera images are transformed to the same plane so that the corresponding pixels are on the same row, thereby simplifying the parallax calculation and depth estimation. The main goal of stereo rectification is to align the images so that the cameras are aligned and the effects of non-parallel optical axes are eliminated.

[0167] The core of binocular stereo vision is to calculate the disparity of each pixel by comparing the images captured by the left and right cameras, and then infer the depth of the point. In order to accurately calculate the depth information, it is necessary to ensure that the internal and external parameters of the two cameras are accurately known. Stereo calibration not only requires calibrating the intrinsic parameters of each camera, but also needs to determine the relative position (translation vector) and relative posture (rotation matrix) between the two cameras. These external parameters are the basis for calculating disparity and depth information. Since the two cameras are installed in different positions, the images taken will have a certain perspective difference. Stereo calibration aligns the perspectives of the two cameras through correction transformation, so that each pair of pixels corresponds to the same object point in the two images.

[0168] In a specific embodiment, the first image includes a left image taken by a left camera and a right image taken by a right camera of the binocular camera used; the processing unit 120 performs stereo matching on the obtained stereo-rectified first image to obtain a depth map of the first image, including: calculating a matching cost for each pair of corresponding pixels in the left image and the right image; based on the calculated matching cost, aggregating the matching costs within the neighborhood of each pixel to obtain a disparity value d with a minimum matching cost for each pixel, thereby obtaining a disparity map of the first image; and calculating a depth map of the first image based on the obtained disparity map of the first image.

[0169] Specifically, the stereo matching may include: image preprocessing and image matching. Before stereo matching, the left and right camera images are preprocessed to improve the accuracy of matching. The image is corrected using the distortion parameters obtained by camera calibration to eliminate lens distortion. The image is filtered and at least one of contrast enhancement is performed to improve image quality. Image matching is to extract features from the image for matching. The extracted features may specifically include at least one of pixel values, gradients, and textures. Since the two images captured at the same time are very similar, pixel values ​​can be used as features to reduce the amount of calculation.

[0170] Image matching includes matching cost calculation and matching cost aggregation. First, matching cost calculation is performed. The first image includes a left image taken by a left camera of the binocular camera used and a right image taken by a right camera. Specifically, a matching cost calculation is performed for each pair of corresponding pixels in the left image and the right image.

[0171] In stereo image matching, the pixels of the left image and the right image (the image taken by the left camera, referred to as the left image, and the image taken by the right camera, referred to as the right image) are usually represented as points in the same scene taken at the same time, but their positions in the image are slightly different due to the different camera perspectives. Each pair of pixels refers to a pixel in the left image (or a window centered on the pixel) and a corresponding pixel in the same scene in the right image (or a window centered on the pixel). In order to find such matching points, the area in the right image that is most similar to the pixels in the left image can be searched along the disparity direction.

[0172] The purpose of calculating the matching cost is to measure whether a certain pixel and its neighborhood in two images are similar. In a specific implementation, the correlation between two pixel blocks can be calculated using Normalized Cross-Correlation (NCC).

[0173]

[0174] Among them, I1(x+i,y+j) and I2(x+i+d,y+j) represent the pixel values ​​of the left image and the right image at positions (x+i,y+j) and (x+i+d,y+j), respectively. and are the means of the left image and the right image in the window respectively, w is the window radius, the window size is (2w+1)*(2w+1), and d is the disparity.

[0175] The best matching point is found by calculating the normalized cross-correlation coefficient of the pixel values ​​within the window. The cross-correlation coefficient value ranges from -1 to 1, with 1 indicating complete similarity and -1 indicating complete dissimilarity.

[0176] After calculating the matching cost for each pair of corresponding pixels in the left image and the right image, the matching cost within the neighborhood of each pixel is aggregated based on the calculated matching cost to obtain the disparity value d with the minimum matching cost for each pixel, thereby obtaining the disparity map of the first image, and the depth map of the second image is calculated based on the obtained disparity map of the first image.

[0177] Specifically, by aggregating the matching costs within the neighborhood of each pixel (the neighborhood refers to the pixel and its eight adjacent pixels, forming a 3X3 pixel area), the impact of noise on the matching results is reduced and the accuracy of disparity estimation is improved. Filtering technology is used to smooth the matching costs, making the disparity map more accurate and coherent.

[0178] In a specific implementation, bilateral filtering can be used to smooth the matching cost. Bilateral filtering not only considers the spatial distance, but also the similarity between pixel values. When calculating the weighted average, both the spatial distance of the pixels and the similarity of the pixel values ​​are considered, so as to better preserve the image edge.

[0179] Specifically, the following formula can be used to calculate the matching cost of each pixel after aggregation:

[0180]

[0181] is the matching cost after aggregation, C(q) is the initial matching cost, N(p) is the neighborhood of pixel p, q is any pixel in the neighborhood N(p) of pixel p, pq represents the pixel coordinate distance, _I(p) and I(q) represent the grayscale values ​​of pixel p and q, σ s is the parameter that controls the effect of spatial distance, σ r is a parameter that controls the influence of pixel value similarity, W p is the normalization coefficient, ensuring the weight is 1.

[0182] After the matching cost aggregation, the disparity value d with the minimum matching cost is selected for each pixel to obtain the disparity map.

[0183] Calculate the baseline distance B: B = ‖T‖

[0184] Calculate the depth map:

[0185] Where: T represents the baseline vector between the two cameras, that is, the distance vector between the two cameras in the horizontal or spatial direction, Z is the depth (the distance from the camera), f is the focal length of the camera after stereo correction, B is the baseline distance between the left and right cameras, and d is the disparity (the horizontal position difference between the corresponding points in the left and right images).

[0186] The determination unit 130 is configured to determine a position of a target object in the environment according to the depth map of the first image processed by the processing unit.

[0187] Specifically, the target object region is segmented on the first image, and the position of the target object in the environment is obtained according to the segmented target object region and the depth map of the first image.

[0188] After stereo matching, a depth map is obtained, but it is impossible to know which part of the depth map belongs to the target object, so the target object area needs to be segmented. Specifically, spatial positioning includes target object segmentation and target object positioning (such as character segmentation and character positioning). First, the target object area of ​​the first image is segmented to obtain the target object area (such as character segmentation), and then the segmented target object area is ANDed with the depth map of the first image to obtain the target object depth area, and then the target object is positioned according to the obtained target object depth area to obtain the target object position.

[0189] In a specific embodiment, the target object region segmentation is performed using the encoder-decoder architecture of the U-Net neural network. The encoder-decoder architecture of the U-Net neural network enables it to capture high-level features while maintaining high-resolution feature maps. This architecture helps to accurately segment complex shapes and details. U-Net introduces jump connections between each encoder layer and the corresponding decoder layer. These connections enable the network to utilize high-resolution features lost during the encoding process during the decoding process, enhance the accuracy of segmentation and the retention of boundary details, and facilitate accurate target object contour segmentation.

[0190] After performing target object region segmentation on the first image, an AND operation is performed on the segmented target object region and the depth map of the first image to obtain a target object depth region; an average value of the depth values ​​of the pixel points in the target object depth region is calculated as the Z coordinate of the target object position, and the coordinates (X, Y, Z) of each pixel point in the world coordinate system are obtained in combination with the camera internal parameters.

[0191] The average value Z of the depth values ​​of the pixels in the depth area of ​​the target object is calculated according to the following formula:

[0192]

[0193] Where D is the depth area, u and v are the horizontal and vertical coordinates of the pixel respectively, d is the disparity value of each pair of pixels, and n is the number of pixels in the depth area of ​​the target object. Combined with the camera internal parameters, the coordinates (u, v) and the depth average Z are converted into camera coordinate system coordinates to obtain the coordinates (X, Y, Z) of each pixel:

[0194]

[0195] Then convert the obtained target object area coordinates (X, Y, Z) from the camera coordinate system to the world coordinate system to obtain the target object position coordinates:

[0196]

[0197] Among them, Xw, Yw, and Zw are the position coordinates of the target object converted from the camera coordinate system to the world coordinate system, and R and T are the relative rotation matrix and relative translation matrix between the left and right cameras, respectively.

[0198] The control unit 140 is used to control the air supply of the air conditioner according to the position of the target object in the environment determined by the determination unit.

[0199] In a specific embodiment, the air conditioner includes more than two air supply units, and the air supply direction can be controlled by a guide plate, which can realize an air supply mode of wind following a target (e.g., wind following a person) and an air supply mode of wind avoiding a target (e.g., wind avoiding a person), so that the temperature of different rooms or areas can be adjusted as needed, thereby improving the user's comfort and energy saving effect. It can effectively reduce energy consumption, reduce operating costs, and be more environmentally friendly.

[0200] Specifically, according to the determined position of the target object in the environment, the air supply direction of the two or more air supply units is controlled to realize the wind-following-target air supply mode or the wind-avoiding-target air supply mode. The wind-following-target air supply mode includes: determining the air supply action area according to the determined position of the target object in the environment, toward the area where the target object is located, wherein the area where the target object is located is an area within a first preset distance from the detected target object position. For example, Figure 2a A schematic diagram showing the wind following the target is shown. Figure 2a As shown, the dot represents the target object, and when the wind follows the target (for example, the wind follows a person), the air supply direction is toward the area where the target object is located. In this mode, users can quickly feel the air conditioning effect, especially in hot or cold weather, avoiding ineffective cooling or heating of unmanned areas and saving energy. The wind-avoiding target air supply mode includes: according to the determined target object position, controlling the air supply direction of the two or more air supply units to avoid the area where the target object is located, wherein the area where the target object is located is an area within a first preset distance from the detected target object position, such as toward the surrounding areas of the target object area. For example, Figure 2b A schematic diagram of a wind avoidance target is shown, which can be referred to Figure 2bAs shown in the figure, the dot represents the target object. When the wind avoids the target (for example, the wind avoids people), the air supply direction is toward the surroundings of the target object area. In this mode, direct wind is avoided, reducing the discomfort caused by it, such as headaches, dry eyes, colds, etc. Through indirect air supply, the indoor temperature is more uniform and the overall comfort is improved. It is especially suitable for people who need to stay in an air-conditioned environment for a long time, such as office workers, the elderly and children. It can reach a comfortable temperature relatively quickly and avoid ineffective cooling or heating in unmanned areas, saving energy.

[0201] Preferably, the air supply action area is determined according to the position of the target object in the determined environment, and the air supply direction of the more than two air supply units is controlled according to the determined air supply action area to achieve an air supply mode in which the wind follows the target (for example, the wind follows the person) or an air supply mode in which the wind avoids the target (for example, the wind avoids the person).

[0202] In a specific embodiment, the air supply action area is determined according to the detected target object position, including, in the wind-following air supply mode, determining the connected area formed by more than two target object positions as the air supply action area and / or the single area where a single target object position is located as the air supply action area according to the distance between the detected target object positions. In the wind-avoiding air supply mode, determining the area within a second preset distance around the connected area formed by more than two target object positions as the air supply action area and / or the area within a second preset distance around the single area formed by a single target object position as the air supply action area according to the distance between the detected target object positions. Wherein, the single area where the single target object position is located is the area within the first preset distance from the target object position.

[0203] Specifically, if the distance d between any two or more single target object positions is less than the set distance threshold T, then in the wind-following air supply mode, a connected area within a first preset distance from the two or more target object positions is determined as the air supply action area, or in the wind-avoiding air supply mode, an area within a second preset distance around the connected area within the first preset distance from the two or more target object positions is determined as the air supply action area; if the distance d between any single target object position and other single target object positions is greater than or equal to the set distance threshold T, then in the wind-following air supply mode, a single area within the first preset distance from the single target object position is determined as the air supply action area, or in the wind-avoiding air supply mode, an area within a second preset distance around the single area within the first preset distance from the single target object position is determined as the air supply action area. The distance between a connected area and a connected area or a single area is determined by the two target object positions with the smallest distance.

[0204] Figure 3The figure shows the schematic diagram of the determination rules of connected areas and single areas. Figure 3 As shown in the figure, the dot is the detected target object position (i.e., the human body area shown in the figure), and the surrounding shadow area (i.e., the wind blowing with the person area shown in the figure) is a single area where a single target object position is determined or a connected area formed by two or more target object positions. Set a distance threshold T. When the distance between the positions of two people is less than or equal to T, it can be considered as a connected area. When the distance between the positions of two people is greater than T, it is considered as a single area.

[0205] Figure 4a Figure 2 shows a schematic diagram of the air supply action area in the air supply mode of wind following the target. Figure 4a As shown in , in the wind-follows-the-target air supply mode, the air supply action area is a connected area formed by the positions of two or more target objects, or a single area where a single target object is located, that is, Figure 4a Area 1 (i.e. the area where the wind blows with people) is shown in . Figure 4b Figure 2 shows a schematic diagram of the air supply action area in the air supply mode of the wind avoidance target. Figure 4b As shown, in the air supply mode of the wind avoidance target, the air supply action area is the area within the second preset distance around the connected area formed by the positions of more than two target objects, or the area within the second preset distance around a single area where a single target object is located, that is, Figure 4b The area 2 shown in FIG. 1 (i.e., the area where people are sheltered from the wind).

[0206] In a specific embodiment, the air supply direction of the more than two air supply units is controlled according to the determined air supply action area, including: when there is only one air supply action area, each air supply unit supplies air to the partial area closest to the air supply action area; that is, when there is only one connected area and / or single area, each air supply unit supplies air to the partial area closest to the corresponding air supply action area of ​​the connected area and / or single area (the connected area or single area when the wind avoids the target, and the area within a second preset distance around the connected area or single area when the wind avoids the target).

[0207] When there are more than two air supply action areas, a corresponding number of air supply units are allocated to supply air to the two or more air supply action areas according to the area ratio of the two or more air supply action areas. That is, when there are more than two connected areas and / or single areas, a corresponding number of air supply units are allocated to supply air to the corresponding air supply action areas of the two or more connected areas and / or single areas (the two or more connected areas or single areas when the wind avoids the target, and the area within the second preset distance around the two or more connected areas or single areas when the wind avoids the target) according to the area ratio of the two or more connected areas and / or single areas.

[0208] For example, Figure 4a The figure shows the schematic diagram of the air supply mode when there is only one air supply action area. Figure 4b A schematic diagram showing the air supply pattern of the wind avoidance target when there is only one air supply action area. Figure 4a , Figure 4b As shown, assuming that there are four independent air supply units, when there is only one connected area and / or a single area, each air supply unit is responsible for a part of the air supply action area. When the action area is relatively concentrated, the wind direction of each air outlet is fixed. When the action area is relatively dispersed, in order to avoid the wind from different air outlets interfering with each other in the air, each air outlet is responsible for a part of the area.

[0209] When there are more than two action areas, a corresponding number of air supply units are allocated according to the area ratio of the air supply action areas. Figure 5a The figure shows the schematic diagram of the air supply mode when there are more than two air supply action areas. Figure 5b A schematic diagram showing the air supply pattern of a wind avoidance target when there are more than two air supply action areas.

[0210] Optionally, the acquisition unit 110 is further used to: re-capture the first image of the environment through the binocular camera at preset time intervals; the processing unit 120 is further used to: perform preset processing on the first image re-captured by the acquisition unit 110 to obtain a depth map of the first image; the determination unit 130 is further used to: re-determine the position of the target object in the environment according to the obtained depth map of the first image; the control unit 140 is further used to: adjust the air supply of the air conditioner according to the position of the target object in the environment re-determined by the determination unit 130.

[0211] Specifically, the image acquisition frame rate is set, and when the image frame acquisition at the latest moment is completed, stereo matching and character segmentation are repeated to obtain the number and position of people in the visible area. When at least one of the following conditions is met, the determination unit re-determines the position of the target object in the environment, and the control unit adjusts the air supply of the air conditioner according to the position of the target object in the environment re-determined by the determination unit:

[0212] The number of target objects (e.g., the number of people) at the old and new moments changes, with new people entering or existing people leaving; the number of target objects remains unchanged, and the position difference of any target object in the first images acquired twice is greater than a preset threshold; the air supply mode switches, that is, from wind following the target to wind avoiding the target or from wind avoiding the target to wind following the target.

[0213] The present invention also provides a storage medium corresponding to the air conditioner control method, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the aforementioned methods are implemented.

[0214] The present invention also provides an air conditioner corresponding to the air conditioner control method, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the aforementioned methods when executing the computer program.

[0215] The present invention also provides an air conditioner corresponding to the control device of the air conditioner, comprising any of the control devices of the air conditioner described above.

[0216] The present invention also provides a computer program product corresponding to the air conditioner control method, comprising a computer program, and when the computer program is executed by a processor, the steps of any of the aforementioned methods are implemented.

[0217] Accordingly, the solution provided by the present invention introduces binocular vision technology, which can obtain depth information in three-dimensional space, thereby realizing accurate positioning of the specific position of a person or pet in the room. Through this three-dimensional positioning capability, the air conditioner can adjust the wind direction according to the specific position of the user; through binocular vision technology, multiple targets can be identified at the same time, and positioned and tracked respectively, thereby realizing intelligent control of multiple targets; the air supply direction is automatically adjusted according to the position of the located target object, realizing all-round wind following the target and wind avoiding the target, improving the user's comfort, improving the energy efficiency and service life of the air conditioner, and having significant social and economic benefits.

[0218] The solution provided by the present invention solves the problem that the two-dimensional image recognition technology of the air conditioner can only obtain the planar position of the user, but cannot accurately judge the distance and height difference between the user and the air conditioner, thereby failing to achieve refined control; solves the problem that the existing air conditioner cannot recognize multiple targets at the same time, thereby dynamically adjusting the air conditioner's wind direction, temperature and other parameters; solves the problem that the existing air conditioner image recognition technology has low accuracy due to external factors such as lighting and angle, thereby resulting in low system reliability and stability.

[0219] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium. Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hard wiring, or a combination of any of these. In addition, each functional unit may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit.

[0220] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0221] The units described as separate components may or may not be physically separated, and the components of the control device may or may not be physical units, that is, they may be located in one place or distributed in multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0222] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0223] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the claims of the present invention.

Claims

1. A method for controlling an air conditioner, characterized in that: A binocular camera is provided in the environment where the air conditioner is located, for collecting images of the environment, and the control method includes: Collecting a first image of the environment by using the binocular camera; Performing preset processing on the collected first image to obtain a depth map of the first image; Determining a position of a target object in the environment according to the obtained depth map of the first image; The air supply of the air conditioner is controlled according to the determined position of the target object in the environment.

2. The method according to claim 1, characterized in that The binocular camera includes: an infrared binocular camera and a COMS binocular camera; collecting a first image of the environment by the binocular camera includes: Collecting a second image of the environment through a CMOS binocular camera at a preset time interval, and determining the current light intensity of the environment based on the collected second image; According to the determined current light intensity of the environment, an infrared binocular camera or a CMOS binocular camera is used to collect a first image of the environment.

3. The method according to claim 1, characterized in that: Performing preset processing on the collected first image to obtain a depth map of the first image includes: Get the relative rotation matrix and relative translation matrix of the left and right cameras of the binocular camera used; Performing stereo correction on the first image according to the acquired relative rotation matrix and relative translation matrix of the binocular camera used to obtain a stereo-corrected first image; Performing stereo matching on the obtained first image after stereo correction to obtain a depth map of the first image; and / or, Determining a position of a target object in an environment according to the obtained depth map of the first image includes: The first image is segmented into target object regions, and the position of the target object in the environment is obtained according to the segmented target object regions and the depth map of the first image.

4. The method according to claim 3, characterized in that The first image includes a left image taken by a left camera and a right image taken by a right camera of the binocular camera used; stereo matching is performed on the obtained stereo-rectified first image to obtain a depth map of the first image, including: Calculating the matching cost for each pair of corresponding pixels in the left image and the right image; Based on the calculated matching cost, aggregating the matching costs in the neighborhood of each pixel point to obtain a disparity value with a minimum matching cost for each pixel point, thereby obtaining a disparity map of the first image; A depth map of the first image is calculated based on the obtained disparity map of the first image.

5. The method according to claim 3, characterized in that: Segmenting the first image into a target object region, and obtaining a target object position in the environment according to the segmented target object region and a depth map of the first image, including: Performing target object region segmentation on the first image to obtain a target object region; Performing an AND operation on the target object region obtained by segmentation and the depth map of the first image to obtain a target object depth region; The target object is located according to the obtained target object depth area to obtain the target object position.

6. The method according to claim 5, characterized in that Performing target object positioning according to the obtained target object depth region to obtain the target object position includes: Calculate the average value of the depth values ​​of the pixels in the depth area of ​​the target object as the Z coordinate of the target object position; Convert the coordinates of each pixel in the depth area of ​​the target object into the coordinates of the camera coordinate system according to the Z coordinate of the position of the target object and the camera internal parameters of the binocular camera used; The coordinates of each pixel in the target object depth area are converted from the camera coordinate system to the world coordinate system to obtain the position of the target object in the environment.

7. The method according to any one of claims 1 to 6, characterized in that: The air conditioner includes two or more air supply units, and the air supply of the air conditioner is controlled according to the determined position of the target object in the environment, including: According to the determined position of the target object in the environment, the air supply direction of the two or more air supply units is controlled to realize a wind-following-target air supply mode or a wind-avoiding-target air supply mode, wherein: The wind-follows-target air supply mode includes: according to the determined position of the target object, controlling the air supply direction of the two or more air supply units toward the area where the target object is located; The wind-avoiding target air supply mode includes: according to the determined position of the target object, controlling the air supply direction of the two or more air supply units to avoid the air supply direction toward the area where the target object is located.

8. The method according to claim 7, characterized in that According to the determined position of the target object in the environment, controlling the air supply direction of the two or more air supply units includes: Determine an air supply action area according to the determined target object position of the environment, and control the air supply directions of the two or more air supply units according to the determined air supply action area; wherein determining the air supply action area according to the determined target object position of the environment includes: In the wind-follows-target air supply mode, according to the distance between the detected target object positions, a connected area formed by two or more target object positions is determined as the air supply action area and / or a single area where a single target object position is located is determined as the air supply action area; and / or, In the wind avoidance target air supply mode, based on the distance between the detected target object positions, the area within a second preset distance around the connected area formed by more than two target object positions is determined as the air supply action area and / or the area within a second preset distance around a single area formed by a single target object position is determined as the air supply action area.

9. The method according to any one of claims 1 to 6, characterized in that: Also includes: Re-collecting a first image of the environment through the binocular camera at every preset time interval, and performing preset processing on the collected first image to obtain a depth map of the first image; The position of the target object in the environment is re-determined according to the obtained depth map of the first image, and the air supply of the air conditioner is adjusted according to the re-determined position of the target object in the environment.

10. The method according to claim 9, characterized in that When at least one of the following conditions is met, the position of the target object in the environment is re-determined, and the air supply of the air conditioner is adjusted according to the re-determined position of the target object in the environment: The number of target objects changes; The position difference of any target object in the first images acquired twice is greater than a preset threshold; Switch the air supply mode.

11. A control device for an air conditioner, characterized in that: A binocular camera is provided in the environment where the air conditioner is located, for collecting images of the environment, and the control device comprises: A collection unit, used for collecting a first image of the environment through the binocular camera; a processing unit, configured to perform a preset process on the first image acquired by the acquisition unit to obtain a depth map of the first image; a determination unit, configured to determine a position of a target object in the environment according to the depth map of the first image obtained by the processing unit; A control unit is used to control the air supply of the air conditioner according to the position of the target object in the environment determined by the determination unit.

12. A storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the steps of any method described in claims 1-10 are implemented.

13. An air conditioner, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any method described in claims 1 to 10 when executing the program, or comprises the control device described in claim 11.

14. A computer program product, characterized in that The invention comprises a computer program, which implements the steps of any one of the methods of claims 1 to 10 when being executed by a processor.

Images