Information processing device, control system, information processing method, and recording medium

Abstract

The information processing device (100) includes an application control unit (111), an image analysis unit (112), and a communication control unit (113). The application control unit (111) executes an application for displaying an image on a display (170). The image analysis unit (112) analyzes the image displayed on the display (170) by the application. The communication control unit (113) converts data based on the image displayed on the display by the application into data recognizable by a control device based on the analysis result obtained by the image analysis unit (112), and transmits the converted data to the control device.

Description

Technical Field

[0001] The present invention relates to an information processing device, a control system, an information processing method and a program. Background Art

[0002] In manufacturing plants, information devices, such as tablet computers, are sometimes used to manage the entry and exit of workers, while control devices, such as PLCs (Programmable Logic Controllers), operate equipment. In these cases, the information devices manage the entry and exit of workers through a general-purpose application. When a worker enters, this status is displayed on the display. Furthermore, when a worker enters, the worker operates the control device to operate the equipment. If data indicating the entry of a worker can be sent to the control device via a general-purpose application to activate the equipment, the worker's workload can be reduced, leading to improved operational efficiency.

[0003] However, when the information device and the control device are to be linked, a dedicated application having a linking function with the control device is required, which leads to a problem of increased development costs.

[0004] In contrast, Patent Document 1 discloses a technique for creating a dedicated application that enables collaboration between information equipment and external equipment by introducing a command conversion library that converts control commands in a format recognizable by a dedicated application into control commands in a format recognizable by external equipment.

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2015-162214 Summary of the Invention

[0006] However, the technology disclosed in Patent Document 1 enables collaboration between information devices and external devices by introducing a command conversion library. Therefore, without knowing the control commands in a format recognizable by the external device, it is difficult to create a dedicated application. Furthermore, when enabling information devices to collaborate with control devices, including FA (Factory Automation) equipment, if the communication protocol or interface specifications are not publicly available, it is difficult for third parties to create dedicated applications that collaborate with the control devices.

[0007] The present invention has been made in view of the above-mentioned circumstances, and an object of the present invention is to provide an information processing device, a control system, an information processing method, and a program that enable a control device to cooperate with an application program that does not support the control device.

[0008] In order to achieve the above-mentioned purpose, the information processing device related to the present invention has: an application control unit, which executes an application for displaying an image on a display; an image analysis unit, which analyzes the image displayed on the display by the application; and a communication control unit, which converts data based on the image displayed on the display by the application into data recognizable by a control instrument based on the analysis result obtained by the image analysis unit, and sends the converted data to the control instrument.

[0009] Effects of the Invention

[0010] According to the present invention, by converting data based on an image displayed by an application on a display into data recognizable by a control device and sending the converted data to the control device, the control device can cooperate with an application that does not support the control device. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a diagram showing a control system according to an embodiment of the invention.

[0012] Figure 2 This is a diagram showing an information processing device according to an embodiment of the invention.

[0013] Figure 3 This is a block diagram showing an information processing device according to an embodiment of the invention.

[0014] Figure 4A It is a diagram showing the display on the display of the information processing device according to the embodiment of the invention.

[0015] Figure 4B This is a diagram showing a first table according to an embodiment of the invention.

[0016] Figure 4C This is a diagram showing a second table according to an embodiment of the invention.

[0017] Figure 5 It is a diagram showing a control device according to an embodiment of the invention.

[0018] Figure 6 This is a block diagram showing a control device according to an embodiment of the invention.

[0019] Figure 7 This is a flowchart showing a setting process according to an embodiment of the invention.

[0020] Figure 8 This is a diagram for explaining the setting process according to the embodiment of the invention.

[0021] Figure 9 This is a diagram for explaining the setting process according to the embodiment of the invention.

[0022] Figure 10 This is a flowchart showing a control process according to an embodiment of the invention.

[0023] Figure 11 This is a diagram for explaining the control process according to the embodiment of the invention. DETAILED DESCRIPTION

[0024] Below, with reference to the attached Figure 1 An information processing device and a control system according to an embodiment of the present invention will be described.

[0025] The control system 1 of this embodiment is a system that makes application software (hereinafter referred to as application) for labor management including entry and exit management of workers cooperate with control equipment. Figure 1 As shown, the control system 1 includes an information processing device 100 having a display 170, and a control unit 200 that communicates with the information processing device 100 to control the robot 300a. With this configuration, the control system 1 can automatically power on the robot 300a upon the user's entry and automatically power off the robot 300a upon the user's departure.

[0026] The robot 300a and the measuring instrument 300b are connected to the control device 200. The control system 1 controls the robot 300a, which can reduce the workload of inputting the measurement data obtained from the measuring instrument 300b into the FA (Factory Automation) system or the workload of outputting the data from the control device 200.

[0027] The workload input to the general application program The information processing device 100 is connected to the control device 200 using a wired method such as a USB (Universal Serial Bus) cable or a wireless method implemented by Wi-Fi (registered trademark).

[0028] The information processing device 100 includes a smartphone or a tablet PC (Personal Computer).

[0029] Outside, such as Figure 2 As shown, the information processing apparatus 100 includes a control unit 110 that executes a program, a communication unit 120 that communicates with a control device 200 , a display 170 that displays an image, and an operation unit 180 that receives a user's operation.

[0030] The control unit 110 includes a processor 130 that executes processing for controlling the communication unit 120, the display 170, and the operation unit 180; a main storage unit 140 that is used as a work area for the processor 130; and an auxiliary storage unit 150 that stores various data and programs used by the processor 130. Both the main storage unit 140 and the auxiliary storage unit 150 are connected to the processor 130 via a bus 160.

[0031] The processor 130 includes an MPU (Micro Processing Unit). The processor 130 executes

[0032] The various functions of the information processing device 100 are implemented using programs stored in the auxiliary storage unit 150. The main storage unit 140 includes RAM (Random Access Memory). Programs are loaded from the auxiliary storage unit 150 into the main storage unit 140. Furthermore, the main storage unit 140 is used as a work area for the processor 130 and stores data sent to the control device 200.

[0033] The auxiliary storage unit 150 includes nonvolatile memory, typically EEPROM (Electrically Erasable Programmable Read-Only Memory). In addition to storing programs, the auxiliary storage unit 150 also stores general-purpose application programs and various data used by the processor 130. Following instructions from the processor 130, the auxiliary storage unit 150 supplies data used by the processor 130 and stores data supplied by the processor 130.

[0034] The communication unit 120 transmits a signal indicating data output from the processor 130 to the control device 200 . In addition, the communication unit 120 receives a signal from the control device 200 and outputs the data indicated by the signal to the processor 130 .

[0035] The display 170 displays images required for operation, and includes an LCD (liquid crystal display) and an organic EL (electro-luminescence) display.

[0036] The operation unit 180 includes a GUI (Graphical User Interface) and receives instructions based on user input. In addition, the operation unit 180 and the display 170 constitute a touch screen device. As an example, Figure 4AAs shown, if a user touches a contact point 10 displayed on display 170 by a general-purpose application with a finger, operation unit 180 inputs data indicating the touch into the general-purpose application. Furthermore, if text or numbers are input into input box 20 displayed on display 170 by a general-purpose application, operation unit 180 inputs data indicating the input text or numbers into the general-purpose application.

[0037] The control unit 110 executes the program stored in the auxiliary storage unit 150, thereby Figure 3 As shown, the following functional units function: an application control unit 111 that executes an application; an image analysis unit 112 that analyzes an image displayed by the application; and a communication control unit 113 that controls communication with the control device 200.

[0038] The application control unit 111 executes a general application stored in the auxiliary storage unit 150. The general application does not support the control device 200, and includes existing applications. As an example, it includes an application for performing labor management including the entry and exit management of workers. Figure 4A As shown, an image is displayed on the display 170. In this image, a bar graph 30 indicates whether a worker has entered the site. Figure 4A In the example, a short bar graph 30 indicates an "exit," while a long bar graph 30 indicates an "entry." Furthermore, the bar graph 30 need not display the entry and exit events themselves; it only needs to display data linked to the entry and exit events. The application control unit 111 can also be implemented by the operating system (OS) of the information processing device 100. Furthermore, the application control unit 111 activates general-purpose applications stored in the auxiliary storage unit 150 based on data transmitted from the control device 200.

[0039] The image analyzing unit 112 analyzes the image displayed by the general application, and extracts the data to be sent to the control device 200 from the image displayed at a predetermined position in the display area of ​​the display 170. In addition, the image analyzing unit 112 determines a data input area located at a predetermined position from among a plurality of data input areas displayed by the general application, displays the data received from the control device 200 by overlapping the data input area, performs a predetermined input action, and inputs data to the application. As an example, the image analyzing unit 112 sets a read mark RM1 at a position specified by the user on the display 170, sets the position coordinates of the position where the read mark RM1 is set, the action to be performed by the device of the control device 200, and the execution conditions for determining whether the action is to be performed, and saves the set data in Figure 4BThe first table 112a is stored in the auxiliary storage unit 150. Figure 4A As for the color of the position where the read mark RM1 is set, when the bar graph 30 is extended, it becomes the color of the bar graph 30, that is, the specified color, or when the bar graph 30 is not extended, it becomes a color other than the specified color. The image analysis unit 112 determines the color of the position where the read mark RM1 is set, thereby obtaining data for determining whether the operator has entered. In addition, the image analysis unit 112 sets the input mark WM1 at the position specified by the user in the image displayed by the general application, sets the coordinates of the position where the input mark WM1 is set, the data sent from the device of the control device 200, and the input conditions for determining whether the sent data is input to the general application, and saves the set data in Figure 4C The second table 112b is shown. The input mark WM1 is set in the area of ​​the display 170 on which the image is displayed by the general application and which can be input to the general application by the user's operation. The second table 112b is stored in the auxiliary storage unit 150. In addition, the image analysis unit 112 reads the data shown at the position where the read mark RM1 is set. In detail, the image analysis unit 112 reads the color of the position where the read mark RM1 is set. In addition, the image analysis unit 112 performs an operation of inputting the data sent from the control device 200 into the data input area at the position where the input mark WM1 is set, and inputs the data into the application. In detail, the image analysis unit 112 performs a single-touch, double-touch, pull-down operation, and an operation of inputting a numerical value or text into the input box at the position where the input mark WM1 is set on the data sent from the control device 200 through a program, and inputs the data into the general application.

[0040] The communication control unit 113 controls the communication unit 120 to communicate with the control device 200. The communication control unit 113 converts the image data displayed on the display 170 based on a general-purpose application into data recognizable by the control device 200 and transmits the converted data to the control device 200. For example, the communication control unit 113 converts the data displayed at a position where the read mark RM1 is set into data recognizable by the control device 200 and transmits the converted data to the control device 200. As described above, this data includes the device of the control device 200 corresponding to the position where the read mark RM1 is set, the action to be executed by the control device 200, and the execution conditions of the action. Furthermore, the communication control unit 113 receives data transmitted from the control device 200 via the communication unit 120.

[0041] The control device 200 includes a PLC (Programmable Logic Controller) and is communicatively connected to the information processing device 100 via a wired or wireless connection. It receives data sent from the information processing device 100 and sends data to the information processing device 100. The control device 200 is connected to the robot 300a and the measuring device 300b. The robot 300a is controlled by the data sent from the control device 200, and the measuring device 300b sends the measured data to the control device 200. The control device 200 is connected to the robot 300a and the measuring device 300b. Figure 5 The figure shows a control unit 210 and a communication unit 220 .

[0042] The control unit 210 includes a processor 230 that executes processing for controlling the robot 300a; a main storage unit 240 that serves as a work area for the processor 230; and an auxiliary storage unit 250 that stores various data and programs used by the processor 230. Both the main storage unit 240 and the auxiliary storage unit 250 are connected to the processor 230 via a bus 260. The processor 230 includes an MPU. The processor 230 implements the various functions of the control device 200 by executing the programs stored in the auxiliary storage unit 250.

[0043] The main storage unit 240 includes a RAM. Programs are loaded from the auxiliary storage unit 250 into the main storage unit 240. The main storage unit 240 is used as a work area for the processor 230.

[0044] The auxiliary storage unit 250 includes a nonvolatile memory represented by an EEPROM. In addition to storing programs, the auxiliary storage unit 250 also stores various data used by the processor 230. The auxiliary storage unit 250 supplies data used by the processor 230 to the processor 230 in accordance with instructions from the processor 230 and stores data supplied from the processor 230. The communication unit 220 includes a network interface circuit for communicating with the information processing device 100. The communication unit 220 receives signals from the information processing device 100 and the measuring device 300b and outputs the data represented by these signals to the processor 230. Furthermore, the communication unit 220 transmits signals representing data output from the processor 230 to the robot 300a or external equipment.

[0045] The control unit 210 executes the program stored in the auxiliary storage unit 250, thereby Figure 6 As shown, it functions as the communication control unit 211.

[0046] The communication control unit 211 communicates with the information processing device 100 via the communication unit 220. The communication control unit 211 controls the robot 300a based on data transmitted from the information processing device 100. The communication control unit 211 transmits data transmitted from the measuring device 300b to the information processing device 100.

[0047] Next, the configuration and control processes performed by the information processing device 100 having the above configuration will be described. The configuration process is used to configure the general-purpose application and the control device 200 for collaboration. The control process is used to send data from the general-purpose application to the control device 200 and to control the robot 300a based on the data sent. Furthermore, the control process inputs data measured by the measuring device 300b and sent from the control device 200 into the general-purpose application.

[0048] In response to an instruction from the user to start processing, the information processing apparatus 100 starts Figure 7 The user starts the control device 200. The following describes the setting process executed by the information processing device 100 using a flowchart.

[0049] When the setting process starts, the application control unit 111 starts the general application stored in the auxiliary storage unit 150 (step S101). When the general application is started, the display 170 displays Figure 4A The image shown.

[0050] Next, the communication control unit 113 executes communication setting with the control device 200 in response to the user's operation (step S102 ).

[0051] Next, the user Figure 4A The bar graph 30 shown in FIG. 1 determines whether the operator has entered the site, and specifies the position of the reading mark RM1 at a position where the expansion and contraction of the bar graph 30 can be determined. Thus, the image analysis unit 112 sets the position of the reading mark RM1 on the display 170 that displays an image through a general application (step S103), and stores the position coordinates of the reading mark RM1 in the Figure 4B The first table 112a is shown.

[0052] The image analysis unit 112 is as follows Figure 8 As shown, the device X1 of the control device 200 corresponding to the read mark RM1, the action to be performed by the device X1, and the execution condition for determining whether to perform the action are set by the user (step S104), and these data are saved in Figure 4BThe first table 112a shown in FIG. The action executed by device X1 is to control the main power supply of robot 300a on and off based on the transmitted data. The execution condition is whether the color of the position where the read mark RM1 is set is the specified color or a color other than the specified color. If the color is the specified color, data indicating ON is transmitted; if the color is other than the specified color, data indicating OFF is transmitted.

[0053] Next, the image analysis unit 112 determines whether a read flag setting end instruction has been input (step S105). If no end instruction has been input (step S105; No), the process returns to step S103 and repeats steps S103 to S105. This allows multiple read flags to be set.

[0054] If it is determined that the end instruction has been input (step S105; Yes), the image analysis unit 112 guides the user to designate a data input area in the displayed image as the position of the input mark. Figure 4A Input mark WM1 is set at a position designated by the user in the display area of ​​display 170 of the image shown (step S106 ).

[0055] The image analysis unit 112 is as follows Figure 9 As shown, the device Y1 of the control device 200 corresponding to the input mark WM1, the input action, and the input condition for determining whether to input data are set by the user (step S107), and these data are saved in Figure 4C The second table 112b is shown. Thus, the input tag WM1 is associated with the device Y1. The input action here is the action of inputting data into a general-purpose application program using a single touch. The image analysis unit 112 executes the user's single-touch input action through a program, inputting data into the general-purpose application program.

[0056] Next, the image analysis unit 112 determines whether an end instruction for input mark setting has been input (step S108). If no end instruction has been input (step S108: No), the process returns to step S106 and repeats steps S106 to S108. This allows multiple input marks to be set.

[0057] If it is determined that an end instruction has been input (step S108: Yes), the setup process ends. As described above, once the setup process is executed, data can be sent from the general application to the control device 200, and the robot 300a can be controlled based on the sent data. Next, the control process for controlling the robot 300a will be described.

[0058] In response to an instruction from the user to start processing, the information processing apparatus 100 starts Figure 10 The control process shown.

[0059] When the control process starts, the communication control unit 113 performs a communication test with the control device 200 (step S201). Furthermore, the user executes a cooperation program for the control device 200. When the cooperation program for the control device 200 is executed, a trigger signal is sent from the control device 200 to the information processing apparatus 100, and the communication control unit 113 performs communication setup with the control device 200 (step S202).

[0060] When the communication setting is executed, data for starting a general-purpose application is transmitted from the control device 200 , and the application control unit 111 starts the general-purpose application stored in the auxiliary storage unit 150 based on the data transmitted from the control device 200 (step S203 ).

[0061] Next, the image analysis unit 112 reads the image data generated by the general application in order to determine whether the operator has entered the site. Figure 4A The data shown at the position where the reading mark RM1 is set in the image displayed on the display 170 shown in FIG. Figure 4B The first table 112a shown reads data indicated at the position in the image where the read mark RM1 is set, based on the position coordinates of the read mark RM1 stored in the first table 112a. The image analysis unit 112 detects the intensities of the red, blue, and green components of the image at the position where the read mark RM1 is set, based on a screenshot of the image displayed on the display 170, and determines whether the image is the designated color or a color other than the designated color.

[0062] Next, the communication control unit 113 transmits the data indicated at the position where the read mark RM1 is set read in step S204 to the control device 200 (step S205) in order to transmit data indicating whether the operator has entered the site. Figure 4B The first table 112a shown in the figure lists the device X1 of the control device 200 corresponding to the position where the read mark RM1 is set, the action to be executed by the device X1 of the control device 200, and the execution conditions of the action. Therefore, if the color of the position where the read mark RM1 is set is the specified color, data indicating ON is sent to the device X1 of the control device 200. If the color is not the specified color, data indicating OFF is sent to the device X1 of the control device 200. Thus, the control device 200 controls the robot 300a based on the ON or OFF data sent to the device X1.

[0063] Next, the communication control unit 113 receives data transmitted from the device included in the control apparatus 200 via the communication unit 120 (step S206 ).

[0064] Next, since the data sent from the device Y1 includes the ID (IDentification) of the device Y1 as the source, the image analysis unit 112 refers to the ID in the Figure 4C The data stored in the second table 112b shown in FIG2 corresponds to the coordinate position where the input mark WM1 is set. Therefore, by performing an input operation on the data sent from the device Y1 of the control device 200 at the coordinate position where the input mark WM1 is set, the data is input to the general application (step S207). Figure 7 As shown in step S107, when data indicating ON is transmitted from device Y1, the data input action is to input data to the general application program by a single touch. When data indicating OFF is transmitted from device Y1, no data input action is performed. Image analysis unit 112 executes the same operation as a user's touch operation on display 170 through a program.

[0065] Next, the image analysis unit 112 determines whether an end instruction has been input (step S208). If it is determined that an end instruction has not been input (step S208: No), the process returns to step S204 and repeats steps S204 to S208. If it is determined that an end instruction has been input (step S208: Yes), the control process ends.

[0066] Next, regarding the setting process and receiving process performed by the information processing device 100, Figure 11 As shown, the following example is described, that is, reading marks RM1, RM2 and input marks WM1, WM2 are set in the display area of ​​the display 170 on which an image is displayed by executing a general application for labor management including entry and exit management of operators, and the control device 200 is controlled.

[0067] In response to an instruction from the user to start processing, the information processing apparatus 100 starts Figure 7 The user then starts the control device 200 .

[0068] When the setting process starts, the application control unit 111 starts up a general application stored in the auxiliary storage unit 150 (step S101; Figure 7). Thus, an image is displayed by the general application on the display 170. The image includes a contact point 10 that allows the user to input into the general application by touching it with a finger, an input box 20 for inputting text or numbers, and a bar graph 30 indicating whether the operator has entered.

[0069] Next, the image analyzing unit 112 sets a read mark RM1 at a position designated by the user in the display area of ​​the display 170 on which an image is displayed by a general-purpose application (step S103; Figure 7 Here, the reading mark RM1 is set at a position where it can be determined whether the worker has entered the site by the extension of the bar graph 30 .

[0070] The image analysis unit 112 sets the device X1 of the control device 200 corresponding to the position where the read mark RM1 is set, the action to be executed by the device X1 of the control device 200, and the execution condition for determining whether to execute the action (step S104; Figure 7 The execution action involves turning the main power of robot 300a on and off via device X1. The execution condition is determined by whether the color of the position where read mark RM1 is set is the specified color. If the color of the position where read mark RM1 is set is the specified color due to the extension of bar graph 30, it is determined that the operator has entered the site and data indicating ON is transmitted. If the color is not the specified color, data indicating OFF is transmitted. This data is stored in first table 112a.

[0071] Next, if it is determined that the end instruction has not been input (step S105; No; Figure 7 ), the image analysis unit 112 sets a read mark RM2 at a position designated by the user in the display area of ​​the display 170 on which an image is displayed by a general-purpose application (step S103; Figure 7 In this example, the reading mark RM2 is set in a region of the display area of ​​the display 170 where an image of a measuring device indicating air cleanliness is displayed.

[0072] Next, the image analysis unit 112 sets the device X2 of the control device 200 corresponding to the position where the read mark RM2 is set, the action to be executed by the device X2 of the control device 200, and the execution condition for determining whether to execute the action (step S104; Figure 7 The action is to analyze the image of the area marked with the read mark RM2, perform a numerical conversion, and send data D2 representing the numerical value obtained by the numerical conversion to device X2. The action execution condition is none. Data D2 is data representing the output of the motor of the fan of the air purifier equipped by robot 300a.

[0073] If it is determined that the end instruction is input (step S105; Yes; Figure 7 ), the image analyzing unit 112 sets the input mark WM1 at the position designated by the user in the display area of ​​the display 170 where the image is displayed by the general application (step S106; Figure 7 ).

[0074] Next, the image analysis unit 112 sets a correspondence between the data sent from the control device 200 corresponding to the position where the input mark WM1 is set and the input conditions for inputting the data into the general application (step S107; Figure 7 The data sent from the control device 200 is data D3 sent from the device Y1 included in the control device 200. The input action is a single touch, and the input condition is that the data D3 indicates ON.

[0075] Next, if it is determined that the end instruction has not been input (step S108; No; Figure 7 ), the image analysis unit 112 sets the input mark WM2 at the position specified by the user in the display area of ​​the display 170 where the image is displayed by the general application (step S106; Figure 7 Input mark WM2 is set in a region of the display area of ​​display 170 where a frame for inputting a numerical value is displayed.

[0076] The image analysis unit 112 sets the data sent from the control device 200 and the input conditions of the data corresponding to the position where the input mark WM2 is set (step S107; Figure 7 The data sent from the control device 200 is data D4 sent from the device Y2 included in the control device 200. The input operation is an operation of inputting the numerical value indicated by the data D4. The input condition is none.

[0077] If it is determined that the end instruction is input (step S108; Yes; Figure 7 ), the setting process ends.

[0078] After the setting process is completed, when the user controls the control device 200 through the information processing device 100, the user starts the information processing device 100. Figure 10 The operation of the control process shown is as follows. In addition, the user starts the control device 200.

[0079] When the control process is started, the communication control unit 113 performs a communication test with the control device 200 (step S201; Figure 10Next, when the user executes the program for cooperating with the control device 200, the communication control unit 113 executes the cooperating function with the control device 200 (step S202; Figure 10 If the cooperation function is executed, data for starting a general application is sent from the control device 200, and the application control unit 111 starts the general application stored in the auxiliary storage unit 150 based on the data sent from the control device 200 (step S203; Figure 10 ). Next, the image analysis unit 112 reads the image data generated by the general application. Figure 11 The data shown at the positions where the reading marks RM1 and RM2 are set in the image displayed on the display 170 (step S204; Figure 10 If the color of the position where the read mark RM1 is set is the specified color, data D1 indicating ON is obtained. If the color is not the specified color, data D1 indicating OFF is obtained. Data D2 is obtained by analyzing the image of the area where the read mark RM2 is set and performing numerical conversion.

[0080] Next, the communication control unit 113 transmits the data D1 and D2 indicated at the positions where the read marks RM1 and RM2 are set, which are read in step S204, to the control device 200 via the communication unit 120 (step S205; Figure 10 ). Data D1 is data indicating ON when the color of the position where the read mark RM1 is set is a specified color, and is data indicating OFF when it is a color other than the specified color, and this data D1 is sent to the device X1. Data D2 is data indicating the numerical value obtained by analyzing the image of the area where the read mark RM2 is set and performing numerical conversion, and this data D2 is sent to the device X2. If the data D1 and D2 are sent to the devices X1 and X2, the control device 200 controls the robot 300a based on the data D1 and D2 sent to the devices X1 and X2. As an example, the data D1 is data for controlling the main power supply of the robot 300a on and off, and the data D2 is data indicating the output of the motor of the fan of the air purifier possessed by the robot 300a.

[0081] Next, the communication control unit 113 receives the data transmitted from the control device 200 (step S206; Figure 10). Here, an example of receiving data D3 and D4 sent from the control instrument 200 is described. Data D3 and D4 are sent from the measuring device 300b to the devices Y1 and Y2 of the control instrument 200, respectively, and are stored in the devices Y1 and Y2. The devices Y1 and Y2 of the control instrument 200 are associated through the second table 112b and correspond to the input tags WM1 and WM2, respectively. As an example, data D3 is data obtained by measuring the temperature of the operator's workplace by the measuring device 300b. When the measured temperature is greater than or equal to the reference temperature, it is data indicating ON, and when it is less than the reference temperature, it is data indicating OFF. Data D4 is numerical data indicating the number of products produced by the operator.

[0082] Next, the image analysis unit 112 performs an operation of inputting the data D3 and D4 sent from the control device 200 to the positions where the input marks WM1 and WM2 are set, respectively, and inputs the data D3 and D4 to the general application (step S207; Figure 10 ). Data D3 is sent from device Y1, and based on the second table 112b, it is determined that the input action is an action of inputting data to a general application through a single-touch operation, and the input condition is that data D3 indicates ON. When the measured temperature is greater than or equal to the reference temperature, data D3 is data indicating ON, and a single-touch input operation is performed at the position of the input mark WM1 through the program, thereby inputting data D3 to the general application. In addition, data D4 sent from the control instrument 200 is data sent from device Y2, and through the second table 112b, it is determined that the input action is an action of inputting the numerical value shown in data D4, and the input condition is none. Data D4 is numerical data indicating the number of products produced by the operator, and the operation of inputting to the position where the input mark WM2 is set is performed by the program, and data D4 is input to the general application. Therefore, when the temperature of the place where the operator works is greater than or equal to the reference temperature, the data is recorded in the general application. In addition, the number of products produced by the operator is input to the general application. These data are used for labor management.

[0083] Next, the image analysis unit 112 determines whether an end instruction has been input (step S208; Figure 10 If it is determined that no end instruction has been input (step S208; No; Figure 10 ), then return to step S204, and repeat steps S204 to S208. If it is determined that the end instruction is input (step S208; Yes; Figure 10 ), then the control process ends.

[0084] The information processing device 100 with the above-described structure can coordinate general-purpose applications with the control device 200 based on data obtained from images displayed on the display 170 by the general-purpose applications. This makes it easy to coordinate various existing applications running on the information processing device 100 with the control device. This eliminates the need to develop dedicated applications or functions for each coordinated application, reducing development costs. Furthermore, users can determine the coordination method by operating the information processing device 100, allowing even users without programming expertise to build a coordinated system. While the above example describes coordination between a general-purpose application for managing the entry and exit of workers and the control device 200, the information processing device 100 can coordinate with any general-purpose application. By enabling general-purpose applications to control devices, including the coordinated control device 200, the scope of system construction is expanded. Furthermore, by automating the operation of the general-purpose application, the control device 200 can reduce the input effort and errors associated with user operation of the information processing device 100.

[0085] (Variation)

[0086] Alternatively, the image analysis unit 112 of the information processing device 100 can read data from an image displayed by a general-purpose application or input data to a general-purpose application without setting the read flags RM1 and RM2 and the input flags WM1 and WM2. In this case, the image analysis unit 112 analyzes the image displayed by the general-purpose application on the display 170 using one or a combination of artificial intelligence, machine learning, or pattern recognition. In this case, the image analysis unit 112 extracts data representing user-specified items from the image displayed on the display 170, using one or a combination of artificial intelligence, machine learning, or pattern recognition, to transmit data to the control device 200. As an example, when determining whether a worker has entered a room, machine learning is performed using training data including images displayed by the general-purpose application when the worker has entered the room and images displayed by the general-purpose application when the worker has not entered the room, creating a determination model for determining whether the worker has entered the room based on the image displayed by the general-purpose application. The image analysis unit 112 uses this determination model to determine whether the worker has entered the room based on the image displayed by the general-purpose application. Specifically, image analysis unit 112 acquires an image displayed on display 170 using a screenshot function or the like and analyzes the image data. Various analysis methods can be applied, including color recognition based on image pixel values ​​or character recognition (OCR: Optical Character Recognition).

[0087] Furthermore, in the above embodiment, the image analysis unit 112 of the information processing device 100 reads an image displayed by a general-purpose application and inputs data into the general-purpose application. However, the image analysis unit 112 may only perform one of reading an image displayed by the general-purpose application and inputting data into the general-purpose application. This allows the device to handle only data transmission from the general-purpose application to the control device 200 or only data transmission from the control device 200 to the general-purpose application.

[0088] The information processing device 100 may also include a camera. In this case, a general-purpose application may display images captured by the camera, and the image analysis unit 112 may obtain and analyze the images captured by the camera. In this case, the image analysis unit 112 may also convert the image data into text data through character recognition. As another example, a camera may be used to capture images of a workplace, and the image analysis unit 112 may determine whether a worker is present by performing image recognition on the captured images. If a worker is present, the image analysis unit 112 transmits data to the control device 200 to turn on the devices controlled by the control device 200. Alternatively, a camera may be used to capture images of the display of a measuring instrument not linked to the control device 200, and the image analysis unit 112 may convert the image displayed on the display in the captured image into numerical data through character recognition. This allows data measured by a measuring instrument not linked to the control device 200 to be transmitted to the control device 200.

[0089] Furthermore, in the above embodiment, the application control unit 111 of the information processing device 100 is described as executing a single general-purpose application. However, the application control unit 111 may also execute multiple general-purpose applications and simultaneously display images of the multiple general-purpose applications on the display 170. When the information processing device 100 executes a first application and a second application, the image analysis unit 112 of the information processing device 100 may read data from an image displayed by the first application, transmit the obtained data to the control device 200, and input the data transmitted from the control device 200 into the second application. This makes it possible to automate the process of a user inputting data obtained from an image displayed in the first application into the control device 200 and inputting data obtained from the control device 200 into the second application.

[0090] In the above embodiment, an example in which the operating unit 180 and display 170 of the information processing device 100 constitute a touchscreen device has been described. The operating unit 180 only needs to include a GUI (Graphical User Interface), and can also receive input via a mouse or touchpad. Even in this case, based on the analysis results of images displayed on display 170 by the general-purpose application, it is possible to easily coordinate with the control device 200.

[0091] In the above embodiment, an example in which the control system 1 is used as a control system for an FA system is described. However, as long as the control device 200 is controlled based on the image displayed on the display 170 of the information processing device 100, the control system 1 can be used for any device or instrument, and can also be used as a control system for controlling elevators or platform doors.

[0092] In the above embodiment, the control unit 110 of the information processing device 100 is shown as having a single processor 130. However, the above functions may be executed by a plurality of processors 130 in cooperation. Furthermore, the control unit 110 may include a plurality of main storage units 140 and auxiliary storage units 150. The above hardware configuration including the control device 200 is merely an example and can be arbitrarily modified or altered.

[0093] The control system 1, information processing device 100, and control instrument 200 are not limited to being implemented using a dedicated system, but can also be implemented using a conventional computer system. For example, a computer program for executing the above-mentioned operations can be stored on a computer-readable recording medium (such as a floppy disk, CD-ROM (Compact Disc Read-Only Memory), DVD-ROM (Digital Versatile Disc Read-Only Memory), etc.) and distributed, and the control system 1 that executes the above-mentioned processing can be constructed by installing the computer program on a computer. Alternatively, the computer program can be pre-stored in a storage device of a server device on a communication network, and downloaded by a conventional computer system to construct the control system 1, information processing device 100, and control instrument 200.

[0094] Furthermore, when the functions of the control system 1 , information processing device 100 , and control instrument 200 are realized by sharing or cooperating between the OS and the application, only the application portion may be stored in a recording medium or storage device.

[0095] Alternatively, the computer program can be superimposed on a carrier wave and distributed via a communication network. For example, the computer program can be posted on a bulletin board (BBS) on a communication network and distributed via the communication network. Furthermore, the computer program can be activated and executed under the control of the OS in the same manner as other application programs to perform the above-described processing.

[0096] The present invention is capable of various embodiments and variations without departing from the broad spirit and scope of the present invention. Furthermore, the above-described embodiments are intended to illustrate the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is not indicated by the embodiments but by the claims. Furthermore, various variations implemented within the scope of the claims and within the meaning of inventions equivalent thereto are deemed to be within the scope of the present invention.

[0097] Description of the label

[0098] 1···Control system, 100···Information processing device, 110, 210···Control unit, 111···Application program control unit, 112···Image analysis unit, 113, 211···Communication control unit, 120, 220···Communication unit, 130, 230···Processor, 140, 240···Main storage unit, 150, 250···Auxiliary storage unit, 160, 260···Bus, 170···Display, 180···Operation unit, 200···Control instrument, 300a···Robot, 300b···Measurement device

Claims

1. An information processing device comprising: an application control unit that executes an application for displaying an image on the display; an image analyzing unit that analyzes an image displayed on the display by the application; and The communication control unit converts data based on the image displayed on the display by the application into data recognizable by a control device based on an analysis result obtained by the image analysis unit, and transmits the converted data to the control device.

2. The information processing device according to claim 1, wherein The image analysis unit specifies a data input area in a display area of ​​the display on which an image is displayed by the application, and inputs data transmitted from the control device into the application based on the data input area.

3. The information processing device according to claim 1 or 2, wherein: The image analysis unit sets a position of a read mark indicating a position at which data is to be read in a display area of ​​the display on which an image is displayed by the application program, and reads data displayed at the position where the read mark is set.

4. The information processing device according to claim 2, wherein: The image analysis unit sets an input mark indicating a position for inputting data in a display area of ​​the display where an image is displayed by the application, and inputs the data into the application based on the position where the input mark is set when data is sent from the control device.

5. The information processing device according to claim 1 or 2, wherein: The image analysis unit analyzes the image displayed on the display by the application using one of artificial intelligence, machine learning, and pattern recognition, or a combination thereof.

6. The information processing device according to claim 1 or 2, wherein: The application displays the image captured by the camera on the display. The image analysis unit analyzes the image displayed on the display.

7. A control system comprising: The information processing device according to claim 1 or 2; and The control device communicates with the information processing device.

8. An information processing method using an information processing device having an application program for displaying an image on a display, the information processing method comprising the following steps: an image analyzing step of analyzing the image displayed on the display by the application; and The communication control step converts the data of the image displayed by the application on the display into data recognizable by a control device based on the analysis result obtained by analyzing the image in the image analysis step, and sends the converted data to the control device.

9. A computer-readable recording medium storing a program for causing a computer having an application program for displaying an image on a display to function as: an image analyzing unit that analyzes an image displayed on the display by the application; and The communication control unit converts data based on the image displayed on the display by the application into data recognizable by a control device based on an analysis result obtained by the image analysis unit, and transmits the converted data to the control device.

Images