Air conditioning operation terminal, computer-readable recording medium, and air conditioning system

Abstract

Provided are an air conditioner operation terminal, a computer-readable recording medium, and an air conditioner system. An object detection unit (212) detects a plurality of blades in a captured image in which an air conditioner indoor unit is captured, using a learned model. A blade selection unit (214) selects a target blade from the plurality of blades in the captured image. An image display unit (215) displays an overlapping image in which a target recognition mark and an adjustment interface are overlapped. A designation reception unit (216) receives an adjustment content designated through an operation of the adjustment interface. An air conditioner setting unit (217) sets the received adjustment content to the air conditioner indoor unit.

Description

Technical Field

[0001] This disclosure relates to the operation of air conditioners. Background Technology

[0002] There are air conditioners with indoor units that have multiple blades. For example, ceiling-mounted indoor units often have multiple blades.

[0003] In the use of such air conditioners, if the airflow direction and volume are adjusted according to each blade, a more comfortable indoor environment can be achieved.

[0004] Patent document 1 discloses a technique for using a terminal device such as a smartphone to change the direction and volume of air.

[0005] In this technology, a virtual spatial image corresponding to the airflow blown from the blades of the indoor unit is displayed on the screen. The user then interacts with this screen to change the airflow direction and volume.

[0006] Existing technical documents

[0007] Patent documents

[0008] Patent Document 1: Japanese Patent Application Publication No. 2014-190686 Summary of the Invention

[0009] The problem the invention aims to solve

[0010] Patent Document 1 addresses the operation of an air conditioner with a single blade in the indoor unit. Furthermore, it does not disclose operations for determining the blade to be operated on and adjusting the airflow direction and volume from the determined blade, etc., for air conditioners with multiple blades in the indoor unit.

[0011] The purpose of this disclosure is to enable an air conditioner with an indoor unit having multiple blades to perform operations such as identifying the blade to be operated and adjusting the airflow direction and volume from the identified blade.

[0012] means for solving problems

[0013] The air conditioning operation terminal disclosed herein includes: an image acquisition unit that acquires a captured image of an indoor air conditioning unit having multiple blades; an object detection unit that uses a learned model to detect the multiple blades in the captured image, the learned model being generated by machine learning on training images of indoor air conditioning units of the same type as the captured image; a blade selection unit that selects an object blade from the multiple blades in the captured image, the object blade being one blade from which the airflow is to be adjusted; an image display unit that displays the captured image as an overlay image, the object identification mark being used to identify the object blade, and the adjustment interface being a graphical user interface for specifying the adjustment content of the airflow from the object blade; a specification acceptance unit that accepts the adjustment content specified through the operation of the adjustment interface; and an air conditioning setting unit that sets the accepted adjustment content on the indoor air conditioning unit.

[0014] The effects of the invention

[0015] According to this disclosure, for an air conditioner with multiple blades in the indoor unit, it is possible to perform operations such as identifying the blade to be operated on and adjusting the airflow direction and volume from the identified blade. Attached Figure Description

[0016] Figure 1 This is a structural diagram of the air conditioning system 100 in Implementation Method 1.

[0017] Figure 2 This is a structural diagram of the indoor air conditioning unit 110 in Implementation Method 1.

[0018] Figure 3 This is a structural diagram of the air conditioning operation terminal 200 in Implementation Method 1.

[0019] Figure 4 This is a structural diagram of the storage unit 290 in Embodiment 1.

[0020] Figure 5 This is an explanatory diagram of the learning completed model 291 in implementation method 1.

[0021] Figure 6 This is a diagram showing the blade identification data 292 in Embodiment 1.

[0022] Figure 7 This is a flowchart of the air conditioning operation method in Implementation Method 1.

[0023] Figure 8 This is a diagram showing the captured image 281 in Embodiment 1.

[0024] Figure 9 This is a flowchart of step S120 in implementation method 1.

[0025] Figure 10 This is an explanatory diagram of step S130 in Embodiment 1.

[0026] Figure 11 This is a diagram showing the object blade 113 in Embodiment 1.

[0027] Figure 12 This is a diagram showing the overlapping image 282 in Embodiment 1.

[0028] Figure 13 This is a diagram showing a structural example of the air conditioning system 100 in Embodiment 1.

[0029] Figure 14 This is a diagram showing an example of the structure of the air conditioner controller 120 in Embodiment 1.

[0030] Figure 15 This is a flowchart of the air conditioning operation method in Implementation Method 2.

[0031] Figure 16 This is a diagram showing the candidate blade group 114 in Embodiment 2.

[0032] Figure 17 This is a diagram showing the overlapping image 285 in Embodiment 2.

[0033] Figure 18 This is a diagram showing the overlapping image 282 in embodiment 2.

[0034] Figure 19 This is a structural diagram of the air conditioning operation terminal 200 in embodiment 3.

[0035] Figure 20 This is a flowchart of the air conditioning operation method in Implementation Method 3.

[0036] Figure 21 This is a diagram showing the terminal direction in Embodiment 3.

[0037] Figure 22 This is a flowchart of step S350 in implementation method 3.

[0038] Figure 23 This is a diagram illustrating the display steps of the status interface 287 in embodiment 3.

[0039] Figure 24 This is a diagram showing the overlapping image 282 in embodiment 3.

[0040] Figure 25 This is a diagram showing the overlapping image 282 in embodiment 4.

[0041] Figure 26This is a hardware structure diagram of the air conditioning operation terminal 200 in the implementation method. Detailed Implementation

[0042] In the embodiments and accompanying drawings, the same or corresponding elements are labeled with the same reference numerals. Descriptions of elements labeled with the same reference numerals as those already described are appropriately omitted or simplified. Arrows in the figures primarily indicate data flow or processing procedures.

[0043] Implementation method 1.

[0044] based on Figures 1 to 14 The air conditioning system 100 is described below.

[0045] ***Structure Explanation***

[0046] based on Figure 1 The structure of the air conditioning system 100 is described.

[0047] The air conditioning system 100 includes an air conditioner 101 and an air conditioning control terminal 200.

[0048] Air conditioner 101 includes an outdoor unit 102 and an indoor unit 110.

[0049] Air conditioner outdoor unit 102 is the outdoor unit of air conditioner 101.

[0050] Air conditioner indoor unit 110 is the indoor unit of air conditioner 101.

[0051] The air conditioning control terminal 200 is a terminal used in various operations for air conditioning. For example, a smartphone is used as the air conditioning control terminal 200.

[0052] The air conditioner control terminal 200 communicates with the air conditioner 101 wirelessly. Specifically, the air conditioner control terminal 200 communicates with the indoor unit 110 of the air conditioner.

[0053] based on Figure 2 The structure of the indoor unit 110 of the air conditioner is described.

[0054] The indoor unit 110 of the air conditioner has multiple blades 111, an extension component 112, and a communication device 119.

[0055] Blade 111 is the air outlet.

[0056] The extension component 112 is a component installed in the indoor unit 110 of the air conditioner. For example, the extension component 112 is a component equipped with a human sensor and a temperature sensor.

[0057] Communication device 119 is both a receiver and a transmitter. For example, communication device 119 is a communication chip or NIC. Communication device 119 is used for communication with the indoor unit 110 of the air conditioner.

[0058] NIC is short for Network Interface Card.

[0059] The indoor unit 110 of the air conditioner can adjust the airflow according to each blade 111. Specifically, the indoor unit 110 of the air conditioner adjusts the airflow direction and air volume according to each blade 111.

[0060] Items that allow for the adjustment of the blowing air (such as air direction and air volume) are called "adjustment items".

[0061] The specific details of the adjustment are called the "adjustment content". For example, the adjustment content refers to the wind direction (up, down, left, right) and the intensity of the wind.

[0062] based on Figure 3 The structure of the air conditioning control terminal 200 is described.

[0063] The air conditioning control terminal 200 is a computer equipped with hardware such as a processor 201, a memory 202, an auxiliary storage device 203, a communication device 204, a camera 205, and a display 206. These hardware components are interconnected via signal lines.

[0064] Processor 201 is an IC that performs computational processing and controls other hardware. For example, processor 201 is a CPU, DSP, or GPU.

[0065] IC is short for Integrated Circuit.

[0066] CPU is short for Central Processing Unit.

[0067] DSP is short for Digital Signal Processor.

[0068] GPU is short for Graphics Processing Unit.

[0069] Memory 202 is a volatile or non-volatile storage device. Memory 202 is also referred to as main storage device or main memory. For example, memory 202 is RAM. The data stored in memory 202 is stored in auxiliary storage device 203 as needed.

[0070] RAM is short for Random Access Memory.

[0071] Auxiliary storage device 203 is a non-volatile storage device. For example, auxiliary storage device 203 is ROM, HDD, flash memory, or a combination thereof. Data stored in auxiliary storage device 203 is loaded into memory 202 as needed.

[0072] ROM is short for Read Only Memory.

[0073] HDD is short for Hard Disk Drive.

[0074] The communication device 204 is both a receiver and a transmitter. For example, the communication device 204 is a communication chip or NIC. The communication device 204 is used for communication with the air conditioner operation terminal 200.

[0075] Camera 205 is a shooting device.

[0076] Display 206 is a display device. For example, display 206 is a touch panel display.

[0077] The air conditioning operation terminal 200 includes elements such as an image acquisition unit 211, an object detection unit 212, a blade recognition unit 213, a blade selection unit 214, an image display unit 215, a designated acceptance unit 216, and an air conditioning setting unit 217. These elements are implemented by software.

[0078] The auxiliary storage device 203 stores an air conditioning operation program that enables the computer to function as an image acquisition unit 211, an object detection unit 212, a blade recognition unit 213, a blade selection unit 214, an image display unit 215, a designated receiving unit 216, and an air conditioning setting unit 217. The air conditioning operation program is loaded into the memory 202 and executed by the processor 201.

[0079] The OS is also stored in the auxiliary storage device 203. At least a portion of the OS is loaded into the memory 202 and executed by the processor 201.

[0080] The processor 201 executes the OS and the air conditioning operation program simultaneously.

[0081] OS is short for Operating System.

[0082] The input and output data of the air conditioning operation program are stored in the storage unit 290.

[0083] The memory 202 functions as a storage unit 290. However, auxiliary storage devices such as the auxiliary storage device 203, registers within the processor 201, and cache memory within the processor 201 may also replace the memory 202 or function as a storage unit 290 together with the memory 202.

[0084] The air conditioning control terminal 200 may also have multiple processors to replace the processor 201.

[0085] Air conditioning operating procedures can be recorded (stored) in a computer-readable manner on non-volatile recording media such as optical discs or flash memory.

[0086] based on Figure 4 The structure of the storage unit 290 will be described.

[0087] The storage unit 290 stores data such as the learned model 291 and the blade recognition data 292.

[0088] based on Figure 5 The learning process for Model 291 will be explained.

[0089] The learned model 291 is a model used to detect each blade 111 and extension component 112 from the input image of the indoor unit 110 of the air conditioner.

[0090] Machine learning is performed using multiple training images as input to generate a fully learned model 291.

[0091] Training images are images that serve as training data. The training images depict air conditioner indoor units of the same type as air conditioner indoor unit 110. However, the air conditioner indoor unit depicted in the training images can be air conditioner indoor unit 110, or it can be a different model.

[0092] Ideally, there should be a large number of training images. To improve generalization performance, it is desirable to prepare training images that take into account variations in the indoor lighting environment. Specifically, it is desirable to prepare images with added noise and images with altered brightness as training images.

[0093] For example, the object detection unit 212 accepts one or more training images, generates different training images, and performs machine learning using the accepted training images and the generated training images.

[0094] In machine learning, learning models are used, such as convolutional neural networks, YOLO, SSD, or Faster R-CNN.

[0095] YOLO is short for You Only Look Once.

[0096] SSD is short for Single Shot Multibox Detector.

[0097] Faster R-CNN is short for Faster Region Convolutional Neural Network.

[0098] In machine learning, specifically, it involves learning bounding boxes and class classification.

[0099] A bounding box is a frame that surrounds an object such as leaf 111 or extension 112, representing the area where the object is located.

[0100] The category classification indicates the type of object, such as blade 111 or extended component 112.

[0101] based on Figure 6 Explanation of blade identification data 292.

[0102] The blade identification data 292 is data representing the positional relationship of each blade 111 relative to the extension component 112 and the identification symbol of each blade 111.

[0103] Specifically, the blade identification data 292 shows the position number and identification number corresponding to each blade 111.

[0104] The position number indicates the positional relationship of the blade 111 relative to the extension member 112. For example, the position number of the blade 111 on the right side of the extension member 112 is "1". Furthermore, starting from the blade 111 with position number "1", the position numbers of each blade 111 are increased sequentially in a clockwise order.

[0105] The identification number is the identification number for blade 111.

[0106] Furthermore, the installation location of the extension component 112 is determined when the air conditioner indoor unit 110 is installed. Moreover, in the case of multiple air conditioner indoor units 110, the installation location of the extension component 112 may differ for each air conditioner indoor unit 110. Therefore, the blade identification data 292 is managed for each air conditioner indoor unit 110.

[0107] The blade identification data 292 is pre-stored in the storage unit 290.

[0108] However, blade identification data 292 can also be automatically generated by the air conditioning operation terminal 200.

[0109] The method for automatically generating leaf identification data 292 will be described below.

[0110] For example, the air conditioning operation terminal 200 obtains positional relationship data from the indoor unit 110 via communication. The positional relationship data shows the positional relationship between the extension component 112 and each blade 111, as well as the identifier of each blade 111. Then, the air conditioning operation terminal 200 generates blade identification data 292 based on the position number determined by the learned model 291 and the obtained positional relationship data.

[0111] For example, the air conditioning operation terminal 200 identifies the position numbers of each blade 111 determined by the learned model 291 as temporary identification numbers. Then, the air conditioning operation terminal 200 uses the identified temporary identification numbers to operate the indoor unit 110. Next, the air conditioning operation terminal 200 generates blade identification data 292 based on the deviation between the actual operating blades 111 in response to the operation and the temporary identification numbers. Figure 6 In this context, the temporary identification number is the same as the location number. Therefore, the deviation between the actual operating blade 111 and the temporary identification number is equivalent to a number misalignment of 1. Furthermore, the actual operating blade 111 in response to operation can be automatically detected by the camera 205 capturing images of the indoor air conditioning unit 110, or it can be detected by the user specifying which blade 111 is operating.

[0112] ***Instructions for the Actions***

[0113] The steps of the operation of the air conditioner operation terminal 200 are equivalent to the air conditioner operation method. Furthermore, the steps of the operation of the air conditioner operation terminal 200 are equivalent to the processing steps based on the air conditioner operation procedure.

[0114] based on Figure 7 Instructions on how to operate the air conditioner.

[0115] In step S110, the user operates the camera 205 of the air conditioner operation terminal 200 to take pictures of the air conditioner indoor unit 110.

[0116] The camera 205 takes pictures of the indoor unit 110 of the air conditioner according to the user's operation and outputs the images. The images obtained by taking pictures are called "picture images 281".

[0117] The image acquisition unit 211 acquires the captured image 281 from the camera 205 and stores the captured image 281 in the storage unit 290.

[0118] based on Figure 8 A specific example of the captured image 281 will be explained.

[0119] The captured image 281 is displayed on the display 206 of the air conditioning operation terminal 200.

[0120] The indoor unit 110 of the air conditioner is captured in image 281. The indoor unit 110 of the air conditioner has four blades (111A to 111D). That is, the indoor unit 110 of the air conditioner with four blades (111A to 111D) is captured in image 281.

[0121] Return to Figure 7 The explanation continues from step S120.

[0122] In step S120, the object detection unit 212 uses the learned model 291 to detect multiple leaves 111 in the captured image 281.

[0123] Specifically, the object detection unit 212 uses the captured image 281 as input to learn the model 291. This allows it to detect multiple blades 111 and extension components 112.

[0124] based on Figure 9 The steps of step S120 will be explained.

[0125] In step S121, the object detection unit 212 uses the learned model 291 to estimate the bounding boxes of the blade 111 and the extension component 112 respectively.

[0126] In step S122, the object detection unit 212 determines the respective positions of the blade 111 and the extension component 112 based on each bounding box.

[0127] For example, the object detection unit 212 calculates the center of the bounding box. The calculated center is the determined position.

[0128] Return to Figure 7 The explanation continues from step S130.

[0129] In step S130, the blade recognition unit 213 determines the positional relationship of each blade 111 in the captured image 281 relative to the extension member 112, and identifies each blade 111 based on the determined positional relationship.

[0130] Specifically, the blade identification unit 213 uses the blade identification data 292 to determine the identifier of each blade 111.

[0131] based on Figure 10 The details of step S130 will be explained.

[0132] The vector that shows the reference direction with the extension member 112 as the base point is called the "reference vector". Specifically, the reference direction is the right direction when the extension member 112 is located in the upper right position.

[0133] The vector from the extension component 112 to each blade 111 is called the "relative position vector".

[0134] (1) First, the blade identification unit 213 calculates the relative angle of the relative position vector relative to the reference vector for each blade 111. The calculated relative angle is the rotation angle when the reference vector is rotated counterclockwise from the extension member 112 until it overlaps with the relative position vector.

[0135] (2) Next, the blade identification unit 213 determines the position number of each blade 111 based on the relative angle of each blade 111.

[0136] Specifically, the blade identification unit 213 selects the blade 111 with the smallest relative angle and assigns the position number "1" to the selected blade 111. Then, the blade identification unit 213 selects the remaining blades 111 in descending order of relative angle and assigns position numbers starting from "2" to each selected blade 111.

[0137] (3) Then, the blade identification unit 213 obtains the identification number corresponding to the position number from the blade identification data 292 for each blade 111.

[0138] Return to Figure 7 The explanation continues from step S140.

[0139] In step S140, the blade selection unit 214 selects one blade 111 from the plurality of blades 111 in the captured image 281. The selected blade 111 is referred to as the "object blade 113".

[0140] The object blade 113 is the blade 111 that adjusts the blowing air.

[0141] based on Figure 11 A specific example of the object blade 113 will be explained.

[0142] It is believed that the wind blowing from the blade 111 closest to the user has a stronger impact on the user.

[0143] It is believed that the uppermost leaf 111 in the captured image 281 is the leaf 111 closest to the user.

[0144] Therefore, the blade selection unit 214 selects the topmost blade 111 in the captured image 281 as the target blade 113.

[0145] Return to Figure 7 The explanation continues from step S150.

[0146] In step S150, the image display unit 215 generates an overlay image 282 using the captured image 281, and displays the overlay image 282 on the display 206.

[0147] based on Figure 12 The overlapping image 282 is explained.

[0148] The overlay image 282 is an image 281 that is captured by overlaying the object identification mark 283 and the adjustment interface 284.

[0149] Object identification mark 283 is a mark used to identify object blade 113. Object identification mark 283 overlaps with object blade 113.

[0150] The adjustment interface 284 is a graphical user interface (GUI) for specifying the adjustment content of the blown air from the object blade 113.

[0151] The adjustment interface 284 includes a GUI for each type of adjustment. The types of adjustments include up / down wind direction, left / right wind direction, air volume, and operating mode.

[0152] Use the icon in the adjustment interface 284 (see reference). Figure 12 GUI such as sliders.

[0153] Adjust interface 284 in Figure 12 The interface 284 overlaps the image below the overlapping image 282. However, the overlapping position of the interface 284 is not limited to... Figure 12 The location shown.

[0154] Return to Figure 7 The explanation continues from step S160.

[0155] In step S160, the user specifies the adjustment content for the target blade 113 by operating the adjustment interface 284.

[0156] Then, the designated acceptance department 216 accepts the adjustment content for the target blade 113.

[0157] In step S170, the air conditioning setting unit 217 communicates with the air conditioning indoor unit 110 to set adjustment content for the target blade 113.

[0158] Specifically, the air conditioning setting unit 217 sends an identifier representing the target blade 113 and a setting request for the adjustment content of the target blade 113 to the air conditioning indoor unit 110. The air conditioning indoor unit 110 receives the setting request. Then, the air conditioning indoor unit 110 sets the adjustment content shown in the setting request for the blade 111 identified by the identifier shown in the setting request.

[0159] Then, the indoor unit 110 of the air conditioner adjusts the airflow from the target blade 113 according to the set adjustment settings.

[0160] ***Description of the Implementation Examples***

[0161] based on Figure 13 and Figure 14 The embodiments are described below.

[0162] like Figure 13 As shown, the air conditioner 101 may also have an air conditioner controller 120.

[0163] The air conditioner controller 120 is a remote control used to control the indoor unit 110 of the air conditioner. The air conditioner controller 120 can be either a wired remote control or a wireless remote control.

[0164] The air conditioner controller 120 is connected to the indoor unit 110 via wired or wireless connection to control the indoor unit 110. When the air conditioner controller 120 is connected to the indoor unit 110 via wired connection, the communication device 119 of the indoor unit 110 is not required.

[0165] Figure 14 The structure of the air conditioner controller 120 is shown.

[0166] The air conditioner controller 120 includes hardware such as a processing circuit 121, a communication device 122, and a display 123. These hardware components are interconnected via signal lines.

[0167] The processing circuit 121 is the hardware that implements the air conditioning control unit 124.

[0168] The communication device 122 is both a receiver and a transmitter. For example, the communication device 122 is a communication chip or a NIC. The communication device 122 is used to communicate with the air conditioner controller 120.

[0169] Display 123 is a display device. For example, display 123 is a liquid crystal display or a touch panel display.

[0170] The details of the processing circuit 121 are explained below.

[0171] The processing circuit 121 can be dedicated hardware or a processor that executes the program stored in the memory.

[0172] When the processing circuit 121 is dedicated hardware, the processing circuit 121 may be, for example, a single circuit, a composite circuit, a programmable processor, a parallel programmable processor, an ASIC, an FPGA, or a combination thereof.

[0173] ASIC is short for Application Specific Integrated Circuit.

[0174] FPGA is short for Field Programmable Gate Array.

[0175] The air conditioner controller 120 may also have multiple processing circuits that can replace the processing circuit 121.

[0176] In the processing circuit 121, some functions may be implemented by dedicated hardware, while the remaining functions may be implemented by software or firmware.

[0177] When the air conditioner 101 is equipped with an air conditioner controller 120, step S170 (refer to...) Figure 7 ) Perform as follows.

[0178] In step S170, the air conditioning setting unit 217 sends data indicating the adjustment content for the target blade 113 to the air conditioning controller 120. The air conditioning control unit 124 receives the sent data.

[0179] Then, the air conditioning control unit 124 communicates with the air conditioning indoor unit 110 to set the adjustment content for the target blade 113.

[0180] Alternatively, the identification number of each blade 111 can be determined without using the blade identification data 292.

[0181] For example, the air conditioning operation terminal 200 obtains positional relationship data from the indoor unit 110 via communication. The positional relationship data shows the positional relationship between the extension component 112 and each blade 111, as well as the identifier of each blade 111. Then, the air conditioning operation terminal 200 determines the identification number of each blade 111 based on the position number determined by the learned model 291 and the obtained positional relationship data.

[0182] For example, the air conditioning operation terminal 200 identifies the position numbers of each blade 111 determined by the learned model 291 as temporary identification numbers. Then, the air conditioning operation terminal 200 operates the indoor unit 110 using the identified temporary identification numbers. Next, the air conditioning operation terminal 200 determines the identification number of the blade 111 based on the deviation between the blade 111 actually operating in response to the operation and the temporary identification number. Furthermore, the blade 111 actually operating in response to the operation can be automatically detected by the camera 205 capturing images of the indoor unit 110, or it can be detected by the user specifying which blade 111 to operate.

[0183] ***Effects of Implementation Method 1***

[0184] Embodiment 1 is for an air conditioner 101 with multiple blades 111 in the indoor unit, which can perform operations such as identifying the blades 111 that are to be operated and adjusting the air direction and air volume.

[0185] Specifically, when photographing the indoor unit 110 of the air conditioner, multiple blades 111 are identified from the captured image, and the target blade 113 is determined. Thus, for the air conditioner 101 with multiple blades 111 in the indoor unit, operations such as adjusting the airflow direction and airflow of the target blade 113 can be performed.

[0186] Implementation method 2.

[0187] The method of selecting one blade 111 from two or more candidates to become the target blade 113 is mainly based on... Figures 15 to 18 To illustrate the differences from implementation method 1.

[0188] ***Structure Explanation***

[0189] The structure of the air conditioning system 100 is the same as that in embodiment 1.

[0190] ***Instructions for the Actions***

[0191] based on Figure 15 Instructions on how to operate the air conditioner.

[0192] Steps S210 to S230 are the same as steps S110 to S130 in Embodiment 1.

[0193] After step S230, the process proceeds to step S241.

[0194] In step S241, the blade selection unit 214 selects two or more blades 111 from the plurality of blades 111 in the captured image 281. The selected blades 111 are referred to as "candidate blade group 114".

[0195] Candidate blade group 114 consists of two or more blades 111 that are candidates to become target blade 113.

[0196] based on Figure 16 Specific examples of candidate blade group 114 will be explained.

[0197] It is believed that the wind blown from the blade 111 near the user has a strong impact on the user.

[0198] It is believed that the higher the position of the leaf 111 in the captured image 281, the closer it is to the user.

[0199] Therefore, the blade selection unit 214 selects two blades 111 sequentially from above the captured image 281 as candidate blade groups 114.

[0200] Return to Figure 15 The explanation continues from step S242.

[0201] In step S242, the image display unit 215 generates an overlay image 285 using the captured image 281, and displays the overlay image 285 on the display 206.

[0202] based on Figure 17 The overlapping image 285 is explained.

[0203] Overlapping image 285 is an image 281 that overlaps with the candidate identification marker group.

[0204] The candidate identification mark group is two or more candidate identification marks 286 corresponding to two or more blades 111 that constitute the candidate blade group 114.

[0205] Candidate identification mark 286 is a mark used to identify each blade 111 of candidate blade group 114. The position of candidate identification mark 286 overlaps with that of each blade 111 of candidate blade group 114.

[0206] Return to Figure 15 The explanation continues from step S243.

[0207] In step S243, the user specifies the identifier of a blade 111 by selecting a candidate identification mark 286 of a blade 111 that is desired to be the target blade 113.

[0208] Designated Receiving Department 216 accepts the identifier of one blade 111.

[0209] The blade selection unit 214 selects one blade 111 from the candidate blade group 114 that is identified by the received identifier. The selected blade 111 becomes the target blade 113.

[0210] After step S243, the process proceeds to step S250.

[0211] In step S250, the image display unit 215 generates an overlay image 282 using the captured image 281, and displays the overlay image 282 on the display 206.

[0212] Step S250 is equivalent to step S150 in Implementation Method 1.

[0213] Figure 18 A specific example of overlapping image 282 is shown.

[0214] On the overlapping image 282, candidate identification markers 286 of leaves 111 that were not selected as object leaves 113 can also be overlapped.

[0215] Return to Figure 15 The explanation continues from step S260.

[0216] Steps S260 and S270 are the same as steps S160 and S170 in Embodiment 1.

[0217] ***Effects of Implementation Method 2***

[0218] In Embodiment 2, when the indoor unit 110 of the air conditioner is photographed, multiple blades 111 are identified from the captured image to determine a candidate blade group 114. Furthermore, the designation and acceptance unit 216 accepts the designation of the blades 111.

[0219] Therefore, for an air conditioner 101 with multiple blades 111 in the indoor unit, it is possible to perform operations such as specifying the target blade 113 from the candidate blade group 114 and adjusting the air direction and air volume of the target blade 113.

[0220] Implementation method 3.

[0221] The method for displaying the status of the airflow from object blade 113 is mainly based on Figures 19 to 24 To illustrate the differences from implementation method 1.

[0222] ***Structure Explanation***

[0223] The structure of the air conditioning system 100 is the same as that in embodiment 1.

[0224] However, the structure of the air conditioning operation terminal 200 is different from that in embodiment 1.

[0225] based on Figure 19 The structure of the air conditioning control terminal 200 is described.

[0226] The air conditioning control terminal 200 also has hardware such as a direction sensor 207.

[0227] The orientation sensor 207 is a sensor used to measure the orientation of the air conditioning control terminal 200. For example, the orientation sensor 207 is an accelerometer sensor and a gyroscope sensor.

[0228] The air conditioning control terminal 200 also has a direction acquisition unit 218.

[0229] The air conditioning operating procedure also enables the computer to function as the orientation acquisition unit 218.

[0230] ***Instructions for the Actions***

[0231] based on Figure 20 Instructions on how to operate the air conditioner.

[0232] In step S310, the image acquisition unit 211 acquires the captured image 281 from the camera 205. This process is the same as step S110 in Embodiment 1.

[0233] Furthermore, the direction acquisition unit 218 acquires the terminal direction from the direction sensor 207.

[0234] Terminal direction refers to the direction of the air conditioning operation terminal 200.

[0235] like Figure 21 As shown, the terminal direction is represented by the angle between the direction perpendicular to the plane of the air conditioning operation terminal 200 and the vertical direction.

[0236] Return to Figure 20 The explanation continues from step S320.

[0237] In step S320, the object detection unit 212 uses the learned model 291 to detect multiple blades 111 in the captured image 281.

[0238] Step S320 is the same as step S120 in Embodiment 1.

[0239] In step S320, the bounding boxes of each of the multiple blades 111 are estimated.

[0240] Steps S330 and S340 are the same as steps S130 and S140 in Embodiment 1.

[0241] In step S350, the image display unit 215 generates an overlay image 282 using the captured image 281, and displays the overlay image 282 on the display 206.

[0242] The overlay image 282 is a captured image 281 that overlays the status interface 287 and the adjustment interface 284.

[0243] Status interface 287 is a graphical user interface (GUI) that represents the status of the blowing air from object blade 113.

[0244] based on Figure 22 The steps of step S350 are explained.

[0245] In step S351, the image display unit 215 acquires data (object state) indicating the state of the blowing air from the object blade 113.

[0246] The state of an object is represented by its current value and instruction value.

[0247] The current value is data representing the current state of the blowing air from object blade 113.

[0248] The instruction value is data representing the adjusted state of the blowing air from the object blade 113.

[0249] Specifically, the image display unit 215 obtains the current value by communicating with the air conditioner indoor unit 110 (or air conditioner controller 120).

[0250] In addition, the image display unit 215 obtains the initial value of the instruction from the storage unit 290.

[0251] The instruction initial value is the initial instruction value. For example, the previous instruction value or the current value becomes the instruction initial value.

[0252] In step S352, the image display unit 215 calculates the object tilt based on the bounding box of the object blade 113.

[0253] The object tilt is the tilt of the object leaf 113 in the captured image 281.

[0254] For example, the image display unit 215 uses existing techniques such as the Hough transform to calculate the tilt of the object.

[0255] In step S353, the image display unit 215 calculates the overlap direction based on the terminal direction and the object tilt.

[0256] The overlap direction is the direction overlaid on the state interface 287 of the captured image 281. For example, the overlap direction is represented by a rotation matrix in the reference coordinate system of the state interface 287.

[0257] In step S354, the image display unit 215 causes the status interface 287 to overlap the position of the object leaf 113 in the captured image 281 in the overlapping direction, thereby generating an overlapping image 282, and displays the overlapping image 282 on the display 206.

[0258] Specifically, the image display unit 215 operates as follows.

[0259] First, the image display unit 215 generates a status interface 287 that represents the state of the blowing air from the object blade 113 based on the object state (current value and instruction value).

[0260] Next, the image display unit 215 rotates the status interface 287 in accordance with the overlapping direction.

[0261] Next, the image display unit 215 generates an overlaid image 282 by superimposing the rotated state interface 287 onto the position of the object leaf 113 in the captured image 281.

[0262] Then, the image display unit 215 displays the overlapping image 282 on the display 206.

[0263] In step S355, the image display unit 215 overlays the adjustment interface 284 onto the displayed overlay image 282.

[0264] Figure 23 This shows a summary of the display steps of status interface 287.

[0265] First, estimate the bounding boxes of multiple blades 111 (step S320).

[0266] Next, the bounding box of the object blade 113 is selected, and the tilt of the object blade 113 is calculated based on the bounding box of the object blade 113 (step S352).

[0267] Then, the status interface 287 is superimposed on the tilt angle of the object blade 113 at the position of the object blade 113, and the superimposed image 282 is displayed (step S354).

[0268] Return to Figure 20 The explanation continues from step S360.

[0269] In step S360, the user specifies the adjustment content for the target blade 113 through the operation adjustment interface 284.

[0270] Then, the designated acceptance department 216 accepts the adjustment content for the target blade 113.

[0271] Step S360 is equivalent to step S160 in Implementation Method 1.

[0272] Figure 24 A specific example of overlapping image 282 is shown.

[0273] In the overlay image 282, a state interface 287 and multiple adjustment interfaces (284A to 284C) are overlaid.

[0274] Status interface 287 uses arrows to indicate the direction and intensity of the blowing air from object blade 113.

[0275] The adjustment interface 284A is an icon.

[0276] The adjustment interfaces (284B and 284C) are sliders. Users move the black circle to operate them. Adjusting interface 284B allows specifying the horizontal wind direction. Adjusting interface 284C allows specifying the vertical wind direction.

[0277] The image display unit 215 may also display a mark indicating the current value and a mark indicating the command value on at least one of the status interface 287 and the adjustment interface (284B and 284C).

[0278] Return to Figure 20Step S370 will be explained below.

[0279] In step S370, the air conditioning setting unit 217 sets the adjustment content for the target blade 113 on the air conditioning indoor unit 110.

[0280] Step S370 is the same as step S170 in Embodiment 1.

[0281] ***Description of the Implementation Examples***

[0282] Implementation method 3 can also be applied to implementation method 2. That is, the target blade 113 can also be selected from the candidate blade group 114.

[0283] ***Effects of Implementation Method 3***

[0284] Implementation method 3 displays a status interface 287. Therefore, for an air conditioner 101 with multiple blades 111 in the indoor unit, operations such as adjusting the airflow direction and volume of the target blades 113 can be performed more intuitively.

[0285] Implementation method 4.

[0286] Regarding the method of using status interface 287 as adjustment interface 284, it is mainly based on Figure 25 To illustrate the differences from implementation method 3.

[0287] ***Structure Explanation***

[0288] The structure of the air conditioning system 100 is the same as that in embodiment 3.

[0289] ***Instructions for the Actions***

[0290] The steps for operating the air conditioner are the same as those in Implementation Method 3.

[0291] However, step S360 differs from the processing in embodiment 3 in the following aspects.

[0292] The status interface 287 also serves as the adjustment interface 284. That is, the status interface 287 is a GUI that shows the status of the airflow from the object blade 113, and it is also a GUI for specifying the adjustment content of the airflow from the object blade 113.

[0293] In step S360, the user specifies the adjustment content for the target blade 113 by operating the status interface 287.

[0294] Then, the designated acceptance department 216 accepts the adjustment content for the target blade 113.

[0295] Figure 25 A specific example of overlapping image 282 is shown.

[0296] The user operates the status interface 287 by extending or retracting the arrow indicating the desired wind direction. Through the operation of the status interface 287, the wind in each direction can be specified.

[0297] ***Effects of Implementation Method 4***

[0298] In implementation method 4, the status interface 287 is operated directly.

[0299] Therefore, for an air conditioner 101 with multiple blades 111 in the indoor unit, it is possible to more intuitively perform operations such as adjusting the airflow direction and air volume of the target blades 113.

[0300] ***Supplement to the Implementation Method***

[0301] based on Figure 26 The hardware structure of the air conditioner control terminal 200 is described.

[0302] The air conditioner control terminal 200 has a processing circuit 209.

[0303] The processing circuit 209 is the hardware that implements the image acquisition unit 211, the object detection unit 212, the blade recognition unit 213, the blade selection unit 214, the image display unit 215, the designated acceptance unit 216, the air conditioning setting unit 217, and the direction acquisition unit 218.

[0304] The processing circuit 209 can be dedicated hardware or a processor 201 that executes the program stored in the memory 202.

[0305] When the processing circuit 209 is dedicated hardware, the processing circuit 209 may be, for example, a single circuit, a composite circuit, a programmable processor, a parallel programmable processor, an ASIC, an FPGA, or a combination thereof.

[0306] The air conditioning control terminal 200 may also have multiple processing circuits that can replace the processing circuit 209.

[0307] In the processing circuit 209, some functions may be implemented by dedicated hardware, while the remaining functions may be implemented by software or firmware.

[0308] In this way, the functions of the air conditioning control terminal 200 can be implemented through hardware, software, firmware, or a combination thereof.

[0309] The various embodiments are examples of preferred embodiments and are not intended to limit the technical scope of this disclosure. Each embodiment may be implemented in part or in combination with other embodiments. The steps described using flowcharts, etc., may also be appropriately modified.

[0310] The "section" that is an element of the air conditioner operation terminal 200 may also be rewritten as "process", "procedure", "circuit", or "line".

[0311] Description of Reference Numerals

[0312] 100 Air conditioner system, 101 Air conditioner, 102 Outdoor unit of air conditioner, 110 Indoor unit of air conditioner, 111 Blade, 112 Expansion component, 113 Target blade, 114 Candidate blade group, 119 Communication device, 120 Air conditioner controller, 121 Processing circuit, 122 Communication device, 123 Display, 124 Air conditioner control section, 200 Air conditioner operation terminal, 201 Processor, 202 Memory, 203 Auxiliary storage device, 204 Communication device, 205 Camera, 206 Display, 207 Direction sensor, 209 Processing circuit, 211 Image acquisition section, 212 Object detection section, 213 Blade recognition section, 214 Blade selection section, 215 Image display section, 216 Designated acceptance section, 217 Air conditioner setting section, 218 Direction acquisition section, 281 Captured image, 282 Overlapped image, 283 Object recognition mark, 284 Adjustment interface, 285 Overlapped image, 286 Candidate recognition mark, 287 Status interface, 290 Storage section, 291 Learned complete model, 292 Blade recognition data.

Claims

1. An air conditioning control terminal, comprising: The image acquisition unit acquires images obtained by photographing an indoor air conditioning unit with multiple blades; The object detection unit uses a learned model to detect the multiple blades in the captured image. The learned model is generated by machine learning on training images of air conditioner indoor units of the same type as the captured air conditioner indoor unit. The blade selection unit selects a target blade from the plurality of blades in the captured image, the target blade being the one whose blowing air is to be adjusted; The image display unit displays the captured image as an overlay image, which overlaps an object identification mark and an adjustment interface. The object identification mark is used to identify the object blade, and the adjustment interface is a graphical user interface for specifying the adjustment content of the blowing air from the object blade. The designated acceptance department accepts the adjustment content specified through the operation of the adjustment interface; as well as The air conditioning setting unit is responsible for setting the adjustment content of the indoor unit of the air conditioner.

2. The air conditioning operation terminal according to claim 1, wherein, The indoor unit of the air conditioner has an extension component. The object detection unit detects the plurality of blades and the extended component in the captured image. The air conditioner operating terminal has a blade recognition unit, which determines the positional relationship of each blade in the captured image relative to the extended component, and identifies each blade based on the determined positional relationship.

3. The air conditioning operation terminal according to claim 2, wherein, The blade identification unit uses blade identification data, which shows the positional relationship of each blade relative to the extension member and the identification of each blade, to determine the identification of each blade.

4. The air conditioning operation terminal according to any one of claims 1 to 3, wherein, The object detection unit uses the learned model to estimate the bounding box of each blade, and determines the position of each blade based on the bounding box of each blade.

5. The air conditioning operation terminal according to any one of claims 1 to 4, wherein, The leaf selection unit selects two or more leaves from the plurality of leaves in the captured image as candidate leaf groups. The image display unit displays the captured image, which overlaps with two or more candidate identification marks corresponding to the two or more leaves constituting the candidate leaf group. The designated receiving department accepts the identifier of the one blade specified by selecting one candidate identifier from two or more candidate identifiers. The blade selection unit selects one blade identified by the accepted identifier from the candidate blade group as the target blade.

6. The air conditioning operation terminal according to any one of claims 1 to 5, wherein, The air conditioning operation terminal has a direction acquisition unit. The direction acquisition unit acquires the terminal direction representing the direction of the air conditioning operation terminal. The object detection unit uses the learned model to estimate the bounding boxes of each leaf in the captured image. The image display unit calculates the tilt angle of the object blade in the captured image as the object tilt angle based on the bounding box of the object blade, calculates the overlap direction based on the terminal direction and the object tilt angle, and makes the status interface overlap the position of the object blade in the captured image in the overlap direction, thereby generating the overlapping image, wherein the status interface is a graphical user interface representing the state of the blowing wind from the object blade.

7. The air conditioning operation terminal according to claim 6, wherein, The status interface is used as the adjustment interface. The designated receiving department accepts the adjustment content specified through the operation of the status interface.

8. A computer-readable recording medium recording an air conditioning operating procedure, said air conditioning operating procedure being used to enable a computer to function as the following components: The image acquisition unit acquires images obtained by photographing an indoor air conditioning unit with multiple blades; The object detection unit uses a learned model to detect the multiple blades in the captured image. The learned model is generated by machine learning on training images of air conditioner indoor units of the same type as the captured air conditioner indoor unit. The blade selection unit selects a target blade from the plurality of blades in the captured image, the target blade being the one whose blowing air is to be adjusted; The image display unit displays the captured image as an overlay image, which overlaps an object identification mark and an adjustment interface. The object identification mark is used to identify the object blade, and the adjustment interface is a graphical user interface for specifying the adjustment content of the blowing air from the object blade. The designated acceptance department accepts the adjustment content specified through the operation of the adjustment interface; as well as The air conditioning setting unit is responsible for setting the adjustment content of the indoor unit of the air conditioner.

9. An air conditioning system comprising an indoor unit with multiple blades and an air conditioning control terminal. The air conditioning operation terminal includes: The image acquisition unit acquires images obtained by photographing the indoor unit of the air conditioner. The object detection unit uses a learned model to detect the multiple blades in the captured image. The learned model is generated by machine learning on training images of air conditioner indoor units of the same type as the captured air conditioner indoor unit. The blade selection unit selects a target blade from the plurality of blades in the captured image, the target blade being the one whose blowing air is to be adjusted; The image display unit displays the captured image as an overlay image, which overlaps an object identification mark and an adjustment interface. The object identification mark is used to identify the object blade, and the adjustment interface is a graphical user interface for specifying the adjustment content of the blowing air from the object blade. The designated acceptance department accepts the adjustment content specified through the operation of the adjustment interface; as well as The air conditioning setting unit is responsible for setting the adjustment content of the indoor unit of the air conditioner.

10. An air conditioning control terminal, comprising: The image acquisition unit acquires images obtained by photographing an indoor air conditioning unit with multiple blades and extended components; An object detection unit uses a learned model to detect the plurality of blades in the captured image and to detect the position number of each blade in the captured image. This learned model is generated through machine learning on training images of air conditioner indoor units of the same type as the captured indoor unit. The blade identification unit uses blade identification data that shows the positional relationship of each blade relative to the extension component and the identification symbol of each blade, and identifies each blade based on the positional relationship of each blade relative to the extension component in the captured image. The blade identification data is generated based on the blades that are actually operating when the indoor unit of the air conditioner is operated according to the position number of each blade.

Images