Control device, air conditioner control device, and program

Abstract

In feedback control systems where there is a discrepancy between the target value and the set value, manual adjustment of the set value may be necessary. For example, in air conditioners where there is a discrepancy between the target temperature and the set temperature, the set temperature must be manually adjusted. Manually adjusting the set temperatures individually for numerous air-conditioned spaces in large facilities is not easy. [Solution] The control device related to the disclosed technology that solves the above problem controls a device that adjusts the output to achieve a set value (specified using a GUI) using first information acquired by a first sensor as feedback, and comprises a control amount determination unit and an RPA processing unit. The control amount determination unit acquires second information from a second sensor and calculates a control amount based on the target value and the second information. The RPA processing unit modifies the set value of the device by operating the GUI using RPA based on the control amount.

Description

Technical Field

[0001] The disclosed technology relates to a technique for automatically adjusting a set value (e.g., temperature) of a feedback control system (e.g., an air conditioner). In this specification, the "set temperature" refers to the temperature instructed to the air conditioner, and the "target temperature" refers to the temperature that the user truly wants to achieve.

Background Art

[0002] Air conditioners of a type (air conditioners with built-in temperature sensors) that measure the temperature of the air sucked in by the indoor unit (intake temperature) and regard it as the room temperature are widespread. This is because it is possible to save the labor of installing the temperature sensor separately from the indoor unit. Fig. 1 schematically shows the configuration of an air conditioner with a built-in temperature sensor. The air conditioner with a built-in temperature sensor is composed of an indoor unit 102, an outdoor unit 103, and a controller 104. The indoor unit 102 includes an intake thermometer 105. When the user 106 instructs the set temperature with the controller 104, the air conditioner measures the intake temperature and blows out air whose temperature and air volume are controlled so that the intake temperature approaches the set temperature into the air conditioning target space 101. For example, when the intake temperature is lower than the set temperature, air that warms the air conditioning target space is blown into the air conditioning target space. In the equilibrium state, the intake temperature becomes equal to the set temperature.

[0003] However, air conditioners are generally installed on or near the ceiling, and the temperature near the ceiling is higher than the temperature near the floor. For this reason, the temperature near the floor will be different from the temperature (target temperature) expected by the user, and the user adjusts the set temperature to achieve the target temperature. For example, during cooling operation, if the user still feels hot, the user sets the set temperature slightly lower than the target temperature (Non-Patent Document 1). That is, the air conditioner with a built-in temperature sensor can automatically set the intake temperature to the set temperature, but the user is forced to perform a manual operation to adjust the set temperature.

[0004] In large-scale facilities, numerous air conditioning units are sometimes centrally managed (Figure 2). The degree of discrepancy between the set temperature and the target temperature is influenced by factors such as the time of day (number of people) and the season / weather (temperature of the outside air). Therefore, it is not easy for a small number of managers to individually and manually adjust the set temperatures of numerous air conditioning units to achieve the target temperature for each air-conditioned space. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Panasonic, "Frequently Asked Questions: Air conditioner does not cool to the set temperature (there is a difference between room temperature and the set temperature)", [Accessed September 10, 2024], Internet <https: / / jpn.faq.panasonic.com / app / answers / detail / a_id / 77988 / ~ / %E3%82%A8%E3%82%A2%E3%82%B3%E3%83%B3%E3%81%AE%E5%86%B7%E6%88%BF%E9%81%8B%E8%BB%A2%E3%81%A7%E8%A8%AD%E5%AE%9A%E6%B8%A9%E5%BA%A6%E3%81%BE%E3%81%A7%E5%86%B7%E3%81%88%E3%81%AA%E3%81%84%EF%BC%88%E5%AE%A4%E6%B8%A9%E3%81%A8%E8%A8%AD%E5%AE%9A%E6%B8%A9%E5%BA%A6%E3%81%AB%E5%B7%AE%E3%81%8C%E3%81%82%E3%82%8B%EF%BC%89 / > . [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] As mentioned above, in air conditioners where there is a discrepancy between the target temperature and the set temperature, the set temperature must be manually adjusted. In large facilities, manually adjusting the set temperature of numerous air-conditioned spaces individually is not easy. Furthermore, although this has been explained using air conditioners as an example, similar challenges exist with manual adjustment in all feedback control systems where there is a discrepancy between the target value and the set value. [Means for solving the problem]

[0007] The air conditioner control device related to the disclosed technology, which solves the above problems, controls an air conditioner that adjusts the supply air volume and supply air temperature to achieve a set value (specified using a GUI) by using first temperature information acquired by a first temperature sensor (e.g., an intake air thermometer) as feedback, and comprises a control amount determination unit and an RPA processing unit. The control quantity determination unit acquires second temperature information from a second temperature sensor (for example, a thermometer installed near the user) and calculates a control quantity based on the target value and the second temperature information. Based on the control quantity, the RPA processing unit operates the GUI using RPA to modify the settings of the air conditioner.

[0008] Furthermore, a control device related to the disclosed technology that solves similar problems controls a device that adjusts the output to achieve a set value (specified using a GUI) using first information acquired by a first sensor as feedback, and comprises a control amount determination unit and an RPA processing unit. The control variable determination unit acquires second information from the second sensor and calculates the control variable based on the target value and the second information. The RPA processing unit then uses RPA to operate the GUI and modify the device settings based on the control variable. [Effects of the Invention]

[0009] According to the disclosed technology, in a feedback control system where there is a discrepancy between the set value and the target value, the set value can be continuously and automatically adjusted to maintain the target value. [Brief explanation of the drawing]

[0010] [Figure 1] A schematic diagram showing the functional configuration of an air conditioner with a built-in temperature sensor. [Figure 2] A diagram illustrating the centralized management of numerous air conditioning units in a large-scale facility. [Figure 3] A functional block diagram showing an example configuration of the air conditioning system 3 according to Example 1. [Figure 4]A flowchart for explaining an example of the operation of an air conditioner control device. [Figure 5] A diagram for explaining the air conditioner selection screen when centrally managing multiple air conditioners. [Figure 6] A diagram for explaining the detailed setting screen of the air conditioner in Conference Room 1. [Figure 7] A diagram for explaining the details of the air conditioner selection process and the air conditioner selection data. [Figure 8] A flowchart showing the operation of the RPA support department in the RPA area test mode. [Figure 9] A diagram showing the search area superimposed on the capture of the remote control screen using the air conditioner tab as an example. [Figure 10] A diagram for explaining the display screen in the RPA reliability test mode. [Figure 11] A diagram for explaining false detections on the display screen in the RPA reliability test mode. [Figure 12] A diagram for explaining non-detections on the display screen in the RPA reliability test mode. [Figure 13] A diagram for explaining Example 1 of the procedure for determining reliability in the RPA reliability test mode. [Figure 14] A diagram for explaining Example 2 of the procedure for determining reliability in the RPA reliability test mode. [Figure 15] A diagram for explaining Example 3 of the procedure for determining reliability in the RPA reliability test mode. [Figure 16] A flowchart showing the operation of the RPA support department in the RPA image search test mode. [Figure 17] A diagram for explaining the display screen in the RPA image search test mode. [Figure 18] A diagram for explaining a configuration example when the remote control function, room temperature acquisition function, control amount determination function, and RPA function are provided in separate devices. [Figure 19] A diagram showing a functional configuration example of a computer.

Modes for Carrying Out the Invention

[0011] <000009The embodiments of the disclosed technology will be described in detail below. Components with the same function will be numbered identically, and redundant explanations will be omitted.

[0012] In air conditioners with built-in temperature sensors, using the measurement value of an external temperature sensor instead of the intake air temperature sensor is not easy if the air conditioner is not designed for such use. Incidentally, recent air conditioners are equipped with a touch panel that allows users to change the set temperature and other settings, and this touch panel can be remotely controlled using a PC or similar device. The PC or similar device used for remote control can be operated using Robotics Process Automation (RPA). The set temperature value can be determined from the room temperature history and the target temperature. Therefore, by separately measuring the room temperature, determining the amount of change to the set temperature using a calculator, and then implementing the change using RPA, the set temperature can be continuously and automatically adjusted without modifying the existing air conditioner's hardware or software.

[0013] There are three methods of RPA: image recognition, object recognition, and coordinate recognition. Of these, object recognition is a method that analyzes the structure of the GUI being operated on and detects specific targets (objects) within it, and its advantage is its high recognition accuracy. This method is effective when the target GUI is a web page, but it is unsuitable for commercial and industrial control panels such as air conditioning touch panels, as it is often difficult to detect the structure from the target GUI.

[0014] On the other hand, image recognition is a highly versatile method that involves pre-registering button images to be pressed and then recognizing them. However, image recognition has disadvantages, such as difficulty in detecting certain button images and the risk of malfunctioning if the screen environment changes. Therefore, this specification also discloses techniques for minimizing the aforementioned disadvantages when using image recognition methods in RPA. In this specification, the GUI to be operated on in the image recognition method will be referred to as an "image element," and the GUI to be operated on in the object recognition method will be referred to as an "object." [Examples]

[0015] Figure 3 is a functional block diagram showing an example configuration of the air conditioning system 3 according to Example 1. The air conditioning system 3 consists of an existing air conditioner 301 and an air conditioner control device 303. The existing air conditioner 301 is equipped with a control panel 302 for the user to set the temperature and other parameters. The control panel 302 has a graphical user interface (GUI) and can be remotely operated using a PC or the like.

[0016] The air conditioner control device 303 related to the disclosed technology consists of a management device 309 and a room temperature sensor 308. The control device 309 obtains the room temperature from the room temperature sensor 308 using wired or wireless communication means. The management device 309 includes a remote control unit 304, an RPA processing unit 305, a control amount determination unit 306, a recording unit 307, an RPA support unit 310, and a room temperature sensor 308. The remote control unit 304 remotely operates the control panel 302 of the air conditioner. The room temperature sensor 308 is installed in an appropriate location within the air-conditioned space 101 (the location where the target temperature is to be achieved). The air conditioner control device 303 may control multiple air conditioners 301. In that case, the GUI processed by the remote control unit includes an air conditioner selection screen listing the air conditioners to be controlled (Figure 5). Figure 4 is a flowchart illustrating an example of the operation of the air conditioner control device 303. The following explanation will be given using Figures 3 and 4.

[0017] The air conditioner control device 303 periodically measures the room temperature using room temperature sensors 308 placed in each air-conditioned space and records the data as time-series data for each air-conditioned space in the recording unit 307. The recording unit 307 is assumed to have the target temperature already recorded in it. The air conditioner control device 303 first selects the air conditioner to be processed (step S401). The RPA processing unit operates the air conditioner selection screen (Figure 5) to select the desired air conditioner (step S402). After selection, the remote control screen transitions to the detailed settings screen for the air conditioner in conference room 1 (for example, the contents of Figure 6).

[0018] The control variable determination unit 306 acquires the room temperature history data and target temperature of conference room 1 from the recording unit 307 (step S403). Next, the control variable determination unit 306 determines the control variable based on the room temperature history data and the target temperature (step S404). The control variable is the amount by which the current set temperature is raised or lowered in order to achieve the target temperature. Next, the control variable determination unit 306 obtains the current set temperature of conference room 1 and calculates "current set temperature + control variable" to set the updated set temperature (step S405). The RPA processing unit 305 operates the remote control screen, selects the air conditioner tab in Figure 6, and updates the set temperature of the air conditioner in conference room 1 to the new set temperature (step S406). After that, the air conditioner control device 303 returns to step S401 and processes the next air conditioner.

[0019] The above is a description of the basic operation of the air conditioner control device 303 according to Example 1.

[0020] [Air conditioner selection process] Let's explain the RPA process for selecting an air conditioner (step S402) in more detail. Figure 7(a) shows a detailed example of the flow of step S402. The RPA processing unit 305 acquires data (air conditioner selection data) from the recording unit 307 for selecting the air conditioner selected in step S401 (step S701). The air conditioner selection data consists of button image data, area information within the remote control screen, and processing details. A specific example is shown in Figure 7(b). p1, p2, p3, and p4 are areas that include the Conference Room 1 button, as shown in Figure 5. We will refer to this as area p.

[0021] The RPA processing unit 305 then uses image recognition to search for the conference room 1 button within region p and identifies the region P of the conference room 1 button (step S702). The RPA processing unit 305 then clicks within area P of the air conditioner selection screen (step S703). The click location can be, for example, the center of area P. As a result of the above, the remote control screen 5 transitions from the air conditioner selection screen (Figure 5) to the detailed settings screen for the air conditioner in conference room 1 (Figure 6).

[0022] The above is a detailed explanation of the air conditioner selection process.

[0023] [Temperature setting change process] The RPA process for changing the set temperature (step S406) will be explained in detail. The RPA processing unit 305 searches for the "set temperature box" on the detailed settings screen 6 of the air conditioner in conference room 1 using the image information and search area information of the "set temperature box," identifies area P2 related to the "set temperature box," moves the input cursor to area P2, and inputs the updated set temperature. Next, the RPA processing unit 305 searches for the "change button" using the image information and search area information of the "change button," identifies area P3 related to the "change button," and clicks within area P3. As a result of the above, the set temperature of conference room 1 is updated to the "updated set temperature" calculated in step S405. The above is a detailed explanation of the [Temperature Setting Change Process].

[0024] Next, we will explain the support processes for setting the above process in the RPA processing unit 305. Specifically, we will explain the "RPA area test mode" which assists in setting the search area in button image search, the "RPA reliability test mode" which assists in pre-checking the reliability of button image search, and the "RPA image search test mode" which verifies whether the predetermined RPA process is executed as expected. Each test mode is mainly executed by the RPA support unit 310.

[0025] [RPA domain test mode] The RPA area test mode is a mode that assists in setting the search area in button image search. Figure 8 is a flowchart showing the operation of the RPA support unit 310 in the RPA area test mode. We will explain using Figure 8. The user registers the button images to be searched for, along with their IDs, in the recording unit 307 of the management device 309 beforehand. The user enters the button ID to be processed into the management device 309 (step S801). Based on the button ID, the management device 309 presents the user with a capture of the remote control screen including the image of the button to be processed (step S802). The user specifies the area in which the button image should be searched to the management device 309 (step S803).

[0026] The RPA support unit 310 overlays the search area onto the capture of the remote control screen (step S804). Figure 9 shows an example of the overlay display when q1, q2, q3, and q4 are specified as the search area in the air conditioner tab. The user checks whether the button image search area is appropriate for the button image based on the overlay display. If it is not appropriate (No in step S805), the user re-specifies the button image search area. If it is appropriate (Yes in step S805), the RPA support unit 310 records the button image search area in the recording unit 307 in association with the button ID (step S806).

[0027] The above is an explanation of the [RPA domain test mode].

[0028] [RPA Reliability Test Mode (Reliability Simulation)] The RPA reliability test mode is a mode that helps to pre-verify the accuracy (reliability) of button image search. "Confidence level" is a threshold (0-1) that enables fuzzy matching of button images. Increasing the confidence level (approaching 1) results in a stricter, less ambiguous judgment, increasing the likelihood of "not detected" (no search results). Lowering the confidence level increases the probability of obtaining some search result, but the judgment becomes looser and more ambiguous, increasing the likelihood of "false positives." Ideally, the confidence level should be set near the median between confidence level 'a' when gradually increasing from 0 and changing from "false positive" to "detected," and confidence level 'b' when gradually decreasing from 1 and changing from "not detected" to "detected." However, this value varies depending on the button image and screen conditions (for example, monochrome or grayscale button images have a narrow gap between 'a' and 'b' and are difficult to detect. Button images with color or clear outlines have a wider gap between 'a' and 'b' and are easier to detect. Screen conditions will be discussed later), so prior verification is necessary. First, let's explain the display screen for the RPA reliability test mode.

[0029] The RPA confidence test mode attempts to search for button images within the search area defined in the RPA domain test mode, varying the confidence level each time. Figure 10 shows a capture of the remote control screen with the search area 1001, detection area 1002, and confidence level display area 1005 overlaid. In principle, the detection area 1002 is displayed only for the number of searches performed (the number of confidence levels). In Figure 10, the search area is made to almost perfectly match the search target button image, so the detection area 1002 is displayed overlapping with it. The screen status (button image position) within the remote control screen may fluctuate depending on the state and environment of the PC used for the management device 309 (OS version, browser version, display resolution). According to the inventors' experience, an OS update resulted in a change of approximately 10% in the button image position. Therefore, fixed coordinates cannot be used for the button image position; the button image must be searched for and the target button position identified each time an operation is performed by the RPA. For this reason, the search area needs to be expanded to a degree that anticipates variations in the button image position.

[0030] Figure 11 illustrates the false detection of button images that can occur when the search area 1101 is set to a wide area. The "Air Conditioner Tab" is the button image to be searched, and 1102, 1103, 1104, and 1105 are the detection areas. In the actual confidence display area 1005, the relationship between the detection area and the confidence level used for the search is made immediately clear. For example, the colors of the confidence markers 1010 to 1020 are displayed to match the color of the frame indicating the detection area.

[0031] In Figure 11, the confidence levels corresponding to detection region 1102 are assumed to be 0.4 (1014) to 1.0 (1020), those corresponding to detection region 1104 are 0.3 (1013) and 0.2 (1012), those corresponding to detection region 1103 are 0.1 (1011), and those corresponding to detection region 1105 are 0 (1010). Detection regions 1103, 1104, and 1105 are "false detections," where strings different from "air conditioner" are incorrectly identified as "detected" in areas with low confidence (search that tolerates ambiguity). If the search area 1101 were to be used directly in the RPA operation, the user would understand from the displayed screen that the confidence level would need to be 0.4 or higher. Alternatively, if, for example, the user requires that the "Air Conditioner Tab" be successfully detected with a confidence level of 0.3, the user will understand from the display screen that the search area 1101 needs to be reviewed.

[0032] Figure 12 shows a search area 1201 outside the button image (air conditioner tab) to illustrate a button image search failure (hereinafter referred to as "not detected"). The "air conditioner tab" is the button image to be searched, and 1202, 1203, and 1204 are the detection areas. In Figure 12, the confidence levels corresponding to detection area 1203 are assumed to be 0.4 (1014) and 0.3 (1013), the confidence level corresponding to detection area 1202 is 0.2, and the confidence levels corresponding to detection area 1204 are assumed to be 0.1 (1011) and 0 (1010). Markers 1015 to 1020 represent the confidence levels where the button image was not detected. In the actual RPA confidence test mode display screen, it is displayed in a way that makes it immediately clear that something was not detected. For example, the confidence levels where something was not detected can be indicated with a marker that does not have a color.

[0033] Figure 12 shows the search area 1201 without including the "air conditioner tab" for illustrative purposes. However, even if the search area 1201 includes the "air conditioner tab," if the referenced button image (image obtained from the recording unit 307) is unclear, "not detected" may occur in areas with high confidence (search that does not tolerate ambiguity).

[0034] Next, we will explain three examples of procedures for determining reliability using the RPA reliability test mode.

[0035] <Example Procedure 1> Figure 13(a) is a detailed functional block diagram of the RPA support unit 310 related to Procedure Example 1. Figure 13(b) is a flowchart of Procedure Example 1. The user inputs the simulation target button ID using the input unit 1301 (step S1301). The search unit 1302 obtains the button image search area from the recording unit 307 based on the button ID (step S1302). The search unit 1302 repeatedly searches for button images using a predetermined set of confidence levels (for example, from 0 to 1 in increments of 0.01) (step S1303).

[0036] The reliability support unit 1304 displays on the display unit 1303 an image (similar to Figures 10, 11, and 12) which overlays the search area, detection area, and reliability marker of the button image onto a capture of the remote control screen (step S1304). As described above, confidence markers corresponding to detection areas that do not contain the target button image represent "false detections," while confidence markers without color represent "not detected." We will refer to confidence markers other than "false detections" and "not detected," and their corresponding detection areas, as "normal detections." The user reads the minimum value a and maximum value b of the confidence level for normal detection from the display unit 1303 and inputs them into the input unit 1301 (step S1305). The confidence support unit 1304 calculates c = (a + b) / 2 and records c in the recording unit 307 as the confidence level to be used in searching for the button image (step S1306).

[0037] The above is an explanation of <Procedure Example 1>.

[0038] <Procedure Example 2> Figure 14(a) is a detailed functional block diagram of the RPA support unit 310 related to procedure example 2. It differs from Figure 13(a) in that it does not have a display unit 1303. Figure 14(b) is a flowchart of Procedure Example 2. It differs from Figure 13(b) in that step S1304 is omitted, step S1302 is replaced by step S1402, and step S1305 is replaced by step S1405.

[0039] The user inputs the simulation target button ID using the input unit 1301 (step S1301). The search unit 1302 obtains the button image search area and button click coordinates from the recording unit 307 based on the button ID (step S1402). The button click coordinates are recorded at the same time as setting the button image search area in RPA area determination test mode. For example, the user can be asked to click the center of the button image on a capture of the remote control screen, and its coordinates can be obtained. The search unit 1302 repeatedly searches for button images using a predetermined set of confidence levels (for example, from 0 to 1 in increments of 0.01) (step S1303).

[0040] Next, the confidence support unit 1304 determines the minimum value a and maximum value b of the confidence level for normal detection (step S1405). Specifically, the maximum value of the confidence markers that are not undetected is set to b. In addition, the detection region containing the button image (correct detection region) is extracted from among the multiple detection regions, and the minimum value of the confidence markers corresponding to the correct detection region is set to a. Finally, the confidence support unit 1304 calculates c = (a + b) / 2 and records c in the recording unit 307 as the confidence level to be used in searching for the button image (step S1306).

[0041] The above is an explanation of <Procedure Example 2>. While this specification assumes changes in screen status depending on the PC's state and environment, in the RPA reliability test scenario, the environment of the management device 309 (PC) is considered to be unchanged from when the RPA area determination test was performed. Therefore, the button click coordinates obtained in the RPA area determination test can be used as is. Furthermore, while the above explanation describes obtaining the button click coordinates by having the user click the center of the button image during the RPA area determination test, a button image cropping tool can also be used. The user can specify the button image to be cropped with the mouse, and the cropping tool will automatically crop the button image, obtaining the coordinates of the center of the button image as the button click coordinates.

[0042] <Example Procedure 3> Figure 15(a) is a detailed functional block diagram of the RPA support unit 310 related to procedure example 3. It differs from Figure 13(a) in that it also includes a search area support unit 1501. Figure 15(b) is a flowchart of Procedure Example 3.

[0043] Step S1301 is the same as in Procedure Example 1. The exploration unit 1302 obtains, from the recording unit 307, the button image search area and the target minimum value a and target maximum value b of the reliability of normal detection (step S1502). a and b may be determined for each button image or may be common to all button images. For example, they may be determined as a = 0.7 and b = 0.95. Steps S1303 and S1304 are the same as those in Procedure Example 1.

[0044] The user checks the minimum value A and maximum value B of the reliability of normal detection (detection area including the button image) from the display screen and confirms that "A ≤ a and b ≤ B". If "A ≤ a and b ≤ B" does not hold (No in step S1501), the search area support unit 1501 performs a button image search area re - setting process (step S1502). The button image search area re - setting process is the same as the processes from step S802 to S806 in FIG. 8. When the button image search area is re - set, the process returns to step S1303.

[0045] As a result of the search area re - setting, if "A ≤ a and b ≤ B" is satisfied, the reliability support unit 1304 calculates c = (a + b) / 2 and records c in the recording unit 307 as the reliability to be used in the search for the button image (step S1306).

[0046] The above is the description of Procedure Example 3. In the "button image search area re - setting process", the user designates the area according to the following guidelines. "a < A" means that false detections have occurred more than expected. This is likely because the search area is too wide, so the search area should be set narrower. "B < b" means that undetected cases have occurred more than expected. This is likely because the setting of the search area for the target button image is inappropriate (such as misidentifying the target button, the search area crossing the button image, etc.), so the positional relationship between the search area and the button image should be reviewed.

[0047] The above is the description of the [RPA reliability test mode].

[0048] [RPA Image Search Test Mode] The RPA image search test mode is used to verify whether a predetermined RPA process is executed as expected. This will be explained using the example of specifying the [Air Conditioner] tab in Figure 9. Figure 16(a) shows an example of the RPA image search test mode flow. The RPA support unit 310 obtains data (air conditioner tab test data) from the recording unit 307 to test the selection of the air conditioner tab (step 1601). The air conditioner tab test data consists of the ID of the button image on the air conditioner tab, the coordinates of the button search area, the processing content, and the confidence level.

[0049] Figure 16(b) shows a specific example of air conditioner tab test data. q1, q2, q3, and q4 are regions containing the button images of the air conditioner tab, as shown in Figure 9. We will refer to this as the search region q. The RPA support unit 310 then uses image recognition to search for button images of the air conditioner tab within the search area q and detects the area Q of the button images (step S1602). The RPA support unit 310 then clicks within the detection area Q (step S1603). The click location can be, for example, the center of the detection area Q. At this stage, the user verifies whether the RPA process was executed as expected.

[0050] Finally, the RPA support unit 310 overlays the search area q (1701 in Figure 17) and the detection area Q (1702 in Figure 17) onto the capture of the remote control screen 6, and further displays the confidence level used in the search in the confidence level display area 1703 (step S1604). The user checks the positional relationship between the target button image, the search area, and the detection area, regardless of whether the RPA process was executed correctly or not.

[0051] If any changes occur to the target GUI, the RPA needs to be adjusted. By using the RPA image search test mode, the areas that need adjustment can be quickly identified. The above is a description of the [RPA Image Search Test Mode]. In the above explanation of the RPA support unit 310, the user specifies the search area for the button image. However, it is also possible to use AI technology to automatically analyze the positional relationship between the target button image, the search area, and the detection area, and automatically set the optimal search area.

[0052] The above is a description of Example 1. [Examples]

[0053] In Example 1, the control device 309 that remotely controls the air conditioner control panel also performed control amount determination and RPA processing. The functions for remotely controlling the air conditioner control panel, acquiring room temperature, determining control parameters, and RPA (Robotic Process Automation) may be handled by separate devices. Figure 18(a) shows an example where the central control unit 1801 manages multiple air conditioners. The central control unit 1801 is equipped with a remote control function. The control unit 1802 is equipped with a room temperature acquisition function, a control amount determination function, and an RPA function. As a result, the set temperature of each air conditioner can be continuously and automatically adjusted without making any modifications to the existing air conditioners or the central control unit.

[0054] Furthermore, the functions for remotely controlling the air conditioner control panel, acquiring room temperature, determining control amounts, and RPA functions may be located in a different location from the air conditioner itself. Figure 18(b) again shows an example where the centralized control device 1803 manages multiple air conditioners, and the control device 1804 communicates with the centralized control device 1803 via the communication network 1805. The centralized control device 1803 is equipped with, for example, a remote control function and a room temperature acquisition function. The control device 1804 is equipped with, for example, a control amount determination function and an RPA function. As a result, a large number of air conditioners can be managed remotely without making any modifications to the existing air conditioners.

[0055] The above is a description of Example 2.

[0056] [Example 1] In Example 1, the control variable was determined by the room temperature history. This constitutes feedback control. When the outside temperature rises or the room becomes crowded, a rise in room temperature is expected. Therefore, instead of a room temperature sensor, an outside temperature sensor or a crowd detection sensor may be used to predict the rise in room temperature and determine the control amount.

[0057] [supplement] Although the above explanation used air conditioners as an example, the disclosed technology can be applied to all systems where settings are specified via a GUI (for example, factory automation on production lines in factories, temperature control of aquarium tanks, and liquid level control of water storage tanks in industrial plants). Furthermore, although we have explained using an air conditioner with a built-in temperature sensor, which is prone to discrepancies between the set temperature and the target temperature, even in the case of an air conditioner with an external temperature sensor installed outside the indoor unit near the user (an external temperature sensor type air conditioner), if a discrepancy occurs between the set temperature and the target temperature, forcing manual adjustment of the set temperature, the present invention can be applied to automate the adjustment.

[0058] Furthermore, while the above embodiment described the detection and operation of the air conditioner selection button, temperature setting box, and change execution button on the GUI as examples, the GUI image elements that the disclosed technology can detect are not limited to these. All elements that can be operated via the GUI, such as GUI elements that realize screen transitions, GUI elements that determine the control target, and GUI elements that realize the setting or change of numerical values, can be the target of detection by the disclosed technology. Furthermore, for example, in a GUI screen equipped with machine monitoring video and an operation panel, predetermined images appearing in the monitoring video can also be targets (elements) for detection.

[0059] [Processor, program, recording medium] The functions realized by the components described herein may be implemented in a circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor includes transistors and other circuits and is considered a circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in memory.

[0060] In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0061] If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0062] The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 19, and then causing the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc. to operate.

[0063] The program describing this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory.

[0064] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0065] A computer executing such a program may, for example, first store the program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. Then, when processing is to be executed, the computer reads the program stored on its own storage medium and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the processing may be executed by a so-called ASP (Application Service Provider) type service, where the processing function is realized only by execution instructions and result acquisition, without transferring the program from the server computer to this computer. Furthermore, the processing may be executed using a so-called SaaS (Software as a Service) type service, where a part of the server computer is made available to the user along with the program. In this form, the program includes information used for processing by an electronic computer that is equivalent to a program (data that is not a direct instruction to the computer but has the property of defining the computer's processing).

[0066] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware. [Explanation of Symbols]

[0067] 102 Indoor unit 103 Outdoor unit 104 Controller 105 Intake air temperature gauge 3. Air conditioning management system 301 Air conditioner 302 Control Panel 303 Air Conditioner Control Unit 304 Remote Control Unit 305 RPA Processing Unit 306 Control variable determination unit 307 Records Department 308 Room Temperature Sensor 309 Management device 310 RPA Support Department 5 Remote control screen 6. Air Conditioner Detailed Settings Screen 1001,1101,1201,1701 Search area 1002, 1102, 1103, 1104, 1105 Detection area 1202, 1203, 1204, 1702 Detection area 1005,1703 Confidence level display area 1501 Search area support department 1801,1803 Central control device 1802, 1804 Control device 1805 Communication Network 2000 Computer 2010 Control Unit 2020 Records Department 2030 Input Section 2040 Output Section 2050 Display section

Claims

1. A device for controlling a first device that adjusts the output to achieve a set value using first information acquired by a first sensor as feedback, The aforementioned settings are specified using a GUI. A control variable determination unit that acquires second information from a second sensor and calculates a control variable based on the target value and the second information, The system includes an RPA processing unit that, based on the control amount, operates the GUI using RPA to change the setting value of the first device to a modified setting value, The aforementioned modified setting value is used by the first device to adjust the output to achieve the modified setting value, using the first information acquired by the first sensor as feedback. Control device.

2. A device for controlling an air conditioner that adjusts the supply air volume and supply air temperature to achieve a set value by using first temperature information acquired by a first temperature sensor as feedback, The aforementioned settings are specified using a GUI. A control variable determination unit that acquires second temperature information from a second temperature sensor and calculates a control variable based on the target value and the second temperature information, The system includes an RPA processing unit that operates the GUI using RPA based on the control amount to change the setting value of the air conditioner to a modified setting value. The modified setting value is used by the air conditioner to adjust the supply air volume and supply air temperature to achieve the modified setting value, using the first temperature information acquired by the first temperature sensor as feedback. Air conditioner control unit.

3. A program for causing a computer to function as the control device described in claim 1, or as the air conditioner control device described in claim 2.

Images