Operation system, processing system, method for constructing a processing system, computer, operation method, program, and storage medium

Abstract

An operating system, a processing system, a method for constructing a processing system, a computer, an operating method, a program, and a storage medium that can improve convenience are provided. [Solution] An operation system according to an embodiment acquires a video signal from a first device to be operated. The operation system displays a screen based on the video signal on a display device. In response to an input operation from a user, the operation system generates an operation command corresponding to the input operation. The operation system generates an operation signal corresponding to the first device based on the operation command and transmits the operation signal to the first device.

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to an operation system, a processing system, a method for constructing a processing system, a computer, an operation method, a program, and a storage medium. [Background technology]

[0002] There are systems in which one device is operated by another device, and there is a demand for further improvements in the convenience of these systems. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-110489 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide an operation system, a processing system, a method for constructing a processing system, a computer, an operation method, a program, and a storage medium that can improve convenience. [Means for solving the problem]

[0005] An operation system according to an embodiment acquires a video signal from a first device to be operated. The operation system displays a screen based on the video signal on a display device. In response to an input operation from a user, the operation system generates an operation command corresponding to the input operation. The operation system generates an operation signal corresponding to the first device based on the operation command and transmits the operation signal to the first device. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram illustrating an operation system according to an embodiment. [Figure 2]FIG. 2 is a schematic diagram illustrating a screen of a display device. [Figure 3] FIG. 2 is a schematic diagram illustrating a screen of a display device. [Figure 4] FIG. 2 is a schematic diagram for explaining coordinate transformation in the operation system according to the embodiment. [Figure 5] 4 is a flowchart showing the operation of the operation system according to the embodiment. [Figure 6] FIG. 10 is a schematic diagram showing an operation system according to a first modified example of the embodiment. [Figure 7] FIG. 2 is a schematic diagram illustrating a screen of a display device. [Figure 8] 1A is a schematic diagram illustrating a portion of a first file, and FIG. 1B is a schematic diagram illustrating a portion of a second file. [Figure 9] 10 is a flowchart showing the procedure of an automatic operation method using an operation system according to a first modified example of the embodiment. [Figure 10] 10 is a flowchart illustrating a part of an automatic operation mode trial by the operation system according to the embodiment. [Figure 11] FIG. 10 is a schematic diagram showing an operation system according to a second modified example of the embodiment. [Figure 12] FIG. 10 is a schematic diagram showing an operation system according to a third modified example of the embodiment. [Figure 13] FIG. 10 is a schematic diagram showing an operation system according to a fourth modified example of the embodiment. [Figure 14] FIG. 10 is a schematic diagram showing an operation system according to a fifth modified example of the embodiment. [Figure 15] FIG. 10 is a schematic diagram showing a processing system according to a sixth modified example of the embodiment. [Figure 16] FIG. 13 is a schematic diagram showing an operation system according to a seventh modified example of the embodiment. [Figure 17] FIG. 13 is a schematic diagram showing an operation system according to an eighth modified example of the embodiment. [Figure 18] FIG. 2 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present specification and the drawings, elements similar to those already described are designated by the same reference numerals, and detailed descriptions thereof will be omitted where appropriate.

[0008] FIG. 1 is a schematic diagram showing an operation system according to an embodiment. 1, the processing system 1 includes an operation system 10 and a processing device 21 (first device). The operation system 10 includes an acquisition device 11, a calculation device 12, a generation device 13, a storage device 14, an input device 15, and a display device 16. The operation system 10 is used to operate the processing device 21.

[0009] The processing device 21 processes a workpiece. The processing is, for example, at least one selected from transport, processing, cleaning, heating, cooling, and drying. The processing is, for example, at least one selected from film formation, etching, polishing, lithography, and bonding.

[0010] The processing device 21 includes a control unit 21a. Each component of the processing device 21 operates based on instructions sent from the control unit 21a to process work. The control unit 21a includes, for example, a central processing unit (CPU), a read-only memory (ROM), a random access memory (RAM), a storage device, an input interface, an output interface, a communication interface, and a bus connecting these. The control unit 21a may be a computer dedicated to the processing device 21, or may be a general-purpose computer. The functions of the control unit 21a may be realized by cooperation between multiple computers.

[0011] The input device 22 is used by a user to input data to the processing device 21. The user is, for example, an operator in charge of the processing executed by the processing device 21. The input device 22 includes, for example, at least one selected from a keyboard, a mouse, a touchpad, and a microphone (voice input).

[0012] The display device 23 displays the video signal output from the processing device 21 so that the user can view it. The display device 23 includes a monitor.

[0013] The acquisition device 11 acquires a video signal output from the processing device 21 to the display device 23 and transmits the video signal to the display device 23. The acquisition device 11 includes, for example, a video signal distributor and a capture unit. The acquisition device 11 transmits the acquired video signal to the calculation device 12.

[0014] The arithmetic device 12 displays a screen based on the received video signal on the display device 16. The arithmetic device 12 accepts an operation (input operation) input by a user using the input device 15. In response to the input operation, the arithmetic device 12 generates an operation command corresponding to the input operation. The operation command is a command to execute the same operation content as the input operation. The arithmetic device 12 transmits the operation command to the generation device 13.

[0015] Input operations include pointer movement, selection, character input, etc. For example, a selection operation results in icon selection, menu display, menu selection, focus on an input field, etc. A combination of selection and pointer movement results in range selection, selection of multiple icons, etc.

[0016] When a selection operation is performed, the arithmetic unit 12 performs coordinate conversion. Specifically, when the content displayed on the screen of the display device 23 is reduced or enlarged depending on the resolution of the screen of the display device 16, the arithmetic unit 12 performs coordinate conversion. In coordinate conversion, the arithmetic unit 12 acquires the coordinates selected on the screen of the display device 16. The arithmetic unit 12 converts the coordinates into coordinates on the screen of the display device 23. This is because, in general, when different types of display devices are used, the resolution, coordinates of the origin, coordinate systems, etc. differ between those display devices.

[0017] The computing device 12 includes, for example, a CPU, a ROM, a RAM, a storage device, an input interface, an output interface, a communication interface, and a bus connecting these. The computing device 12 is a dedicated or general-purpose computer. The functions of the computing device 12 may be realized by cooperation between multiple computers. The input device 15 includes, for example, at least one selected from a keyboard, a mouse, a touchpad, and a microphone. The display device 16 includes a monitor.

[0018] The generating device 13 generates an operation signal corresponding to the processing device 21 based on the operation command. The generating device 13 inputs the generated operation signal to the processing device 21 (control unit 21a), which is the target of operation. For example, if the input device 22 includes a mouse or a keyboard, the operation signal corresponds to a signal transmitted from the mouse or keyboard to the control unit 21a. If the input device 22 includes a touchpad, the operation signal corresponds to a signal transmitted from the touchpad to the control unit 21a. The generating device 13 includes, for example, a microcomputer programmed to generate an operation signal corresponding to a signal transmitted from the input device 22.

[0019] When the processing device 21 receives an operation signal, it moves a pointer, selects an icon, inputs characters, and so on, in accordance with the operation signal.

[0020] The storage device 14 stores various data necessary for the operation of the processing device 21. For example, the storage device 14 stores the resolution, coordinates of the origin, coordinate system, etc. of each of the display devices 16 and 23. The storage device 14 includes a hard disk drive (HDD), a solid state drive (SSD), a network attached hard disk (NAS), or the like.

[0021] 2(a), 2(b), 3(a), and 3(b) are schematic diagrams illustrating examples of screens of display devices. 2(a) and 3(a) are examples of screens displayed on the display device 23. FIGS. 2(b) and 3(b) are examples of screens displayed on the display device 16. In this example, the resolution of the screen of the display device 23 is higher than the resolution of the screen of the display device 16. The entire screen of the display device 23 is displayed in a reduced size on the screen of the display device 16. If the resolution of the screen of the display device 23 is lower than the resolution of the screen of the display device 16, the entire screen of the display device 23 is displayed on the screen of the display device 16 at the same size as the screen of the display device 23. The content displayed on the screen of the display device 23 may be enlarged in accordance with the resolution of the screen of the display device 16.

[0022] The screen displayed on the display device 16 and the screen displayed on the display device 23 are based on the same video signal. Therefore, as shown in Figures 2(a) and 2(b), the screen A1 of the display device 16 displays the same content as the screen a1 of the display device 23. In the illustrated example, the screen a1 displays icons b1 to b7 and information c1 related to the workpiece being processed by the processing device 21. The screen A1 displays icons B1 to B7 that are the same as the icons b1 to b7, and information C1 that is the same as the information c1.

[0023] While viewing screen A1 of display device 16, the user inputs an operation to arithmetic device 12 using input device 15. For example, as shown in FIGS. 2(b) and 3(b), the user moves pointer P to icon B3 on screen A1 and selects icon B3 by clicking. In this case, arithmetic device 12 acquires the coordinates of the clicked point. The arithmetic device 12 accesses storage device 14 and refers to the coordinates of the origin of display device 16, the coordinate system of display device 16, the coordinates of the origin of display device 23, and the coordinate system of display device 23.

[0024] 4(a) and 4(b) are schematic diagrams for explaining coordinate transformation in the operation system according to the embodiment. 4(a) and 4(b), the resolution in the X direction of display device 16 is Rx1, the X coordinate of the origin in the coordinate system of display device 16 is Ox1, the resolution in the X direction of display device 23 is Rx2, and the X coordinate of the origin in the coordinate system of display device 23 is Ox2. Arithmetic device 12 can convert the X coordinate x1 of the point clicked on the screen of display device 16 into the X coordinate x2 of the point clicked on the screen of display device 23 using the following formula. x2=(x1-Ox1)×(Rx2-Ox2) / (Rx1-Ox1)+Ox2

[0025] The resolution in the Y direction of display device 16 is Ry1, the Y coordinate of the origin in the coordinate system of display device 16 is Oy1, the resolution in the Y direction of display device 23 is Ry2, and the Y coordinate of the origin in the coordinate system of display device 23 is Oy2. Using the following formula, calculation device 12 can convert the Y coordinate y1 of the point clicked on the screen of display device 16 into the Y coordinate y2 of the point clicked on the screen of display device 23. y2=(y1-Oy1)×(Ry2-Oy2) / (Ry1-Oy1)+Oy2

[0026] The calculation unit 12 generates an operation command to move the pointer to the converted coordinates (x, y) = (x2, y2) and an operation command to select the coordinates (x, y) = (x2, y2). The generation unit 13 generates an operation signal corresponding to the processing unit 21 based on the operation command.

[0027] 3(b), the processing device 21 moves the pointer p to the icon b3 in accordance with the operation signal and selects the icon b3. By including the converted coordinates in the operation command, the operation system 10 can appropriately operate the processing device 21 regardless of the difference between the specifications of the display device 16 and the display device 23.

[0028] As an example, the resolution of the display device 16 is 2000 x 1000. The origin coordinates in the coordinate system of the display device 16 are (x, y) = (10, 20). The resolution of the display device 23 is 640 x 480. The origin coordinates in the coordinate system of the display device 23 are (x, y) = (0, 0). A point with coordinates (x, y) = (460, 400) is clicked on the screen a1 of the display device 16. The calculation device 12 converts the coordinates clicked on the screen A1 into the coordinates clicked on the screen a1 using the above-mentioned formulas as follows: x2=460×(2000-10) / 640+10 y2=400×(1000-20) / 480+20 As a result, the coordinates (x, y)=(1440, 837) to be clicked on the screen a1 are calculated.

[0029] FIG. 5 is a flowchart showing the operation of the operation system according to the embodiment. The acquisition device 11 acquires a video signal (step S1). The calculation device 12 displays a screen based on the video signal on the display device 16 (step S2). The calculation device 12 accepts an input operation using the input device 15 (step S3). In response to the acceptance of the input operation, the calculation device 12 generates an operation command corresponding to the input operation (step S4). The generation device 13 generates an operation signal corresponding to the processing device 21 based on the operation command (step S5). The generation device 13 transmits the generated operation signal to the processing device 21 (step S6). The processing device 21 is operated in accordance with the received operation signal.

[0030] The advantages of the embodiment will be described. The operation system 10 acquires a video signal from the processing device 21 and transmits an operation signal corresponding to the processing device 21 to the processing device 21. The operation system 10 transmits an operation signal corresponding to an input operation from a user to the processing device 21. By using the operation system 10, the user can operate the processing device 21 without using the input device 22. By applying the operation system 10 to an existing processing device 21, input device 22, and display device 23, the processing device 21 can be easily remotely controlled without requiring modification of the processing device 21, communication settings for remote control, or preparation of special communication requirements. According to the operation system 10, the user can directly operate the processing device 21 using the input device 15 without storing input operations or executing the stored input operations. According to the embodiment, the convenience of the operation system 10 can be improved.

[0031] When the operation system 10 operates the processing device 21, the operation system 10 does not need to access data, etc., of the processing device 21. According to the embodiment, the processing device 21 can be remotely operated even when the processing device 21 is connected only to a limited network, thereby improving the security of data related to the processing device 21.

[0032] Here, "remote control" means that the processing device 21 is controlled by an operation system 10 separate from the processing device 21, using electrical signals or the like. The operation system 10 does not need to be installed in a location remote from the processing device 21. The operation system 10 may be installed in a location remote from the processing device 21, or may be installed adjacent to the processing device 21.

[0033] The arithmetic unit 12 reduces or enlarges the screen based on the video signal in accordance with the resolution of the display device 16. This allows the entire screen of the display device 23 to be displayed on the display device 16, facilitating operations such as icon selection, menu display, menu selection, and focusing on an input field, thereby improving the usability of the processing system 1 and the operation system 10.

[0034] When the screen is enlarged or reduced, a deviation occurs between the coordinates selected on the screen of display device 16 and the coordinates to be selected on the screen of display device 23. In order to correct the deviation, calculation device 12 performs the coordinate conversion described above. This makes it possible to select an appropriate position on the screen of display device 23 even when the screen is enlarged or reduced.

[0035] (First Modification) FIG. 6 is a schematic diagram showing an operation system 10a according to a first modified example of the embodiment. The operation system 10a according to the first modification can execute an automatic operation mode for automatically operating the processing device 21. The processing system 1a includes the operation system 10a and the processing device 21. The storage device 14 stores an operation file 14a that is referenced in the automatic operation mode. The operation file 14a describes operations to be performed on the processing device 21.

[0036] A user operates the processing device 21 using the input device 15 or 22. When an operation signal from the generation device 13 or an input operation from the input device 22 is input to the processing device 21, the processing device 21 is operated and the screen of the display device 23 changes. The calculation device 12 determines the operation that has been performed from the input from the input device 15 and the change in the screen, and writes it in an operation file. In the automatic operation mode, the calculation device 12 generates an operation command based on the operation file. As a result, the operation performed by the user is automatically repeated by the operation system 10a.

[0037] The operation file includes a first file and a second file. The first file describes a generation operation for generating an operation command. For example, the first file is a program file describing an operation sequence for generating an operation command and transmitting it to the generating device 13. The first file is written in a general-purpose programming language such as C.

[0038] The second file is referenced when the generation operation is executed. The second file contains information used in identifying the output signal and generating the operation signal (described later). For example, the second file contains information in text format, such as coordinates selected by a pointing device and data referenced during image recognition.

[0039] The arithmetic unit 12 executes the generating operation described in the first file while referring to the information contained in the second file. The arithmetic unit 12 performs image recognition processing on the screen based on the acquired video information. The image recognition determines changes on the screen, operations performed, etc. The image recognition involves performing at least one selected from template matching, character recognition, and color recognition. The operation file and reference data required for image recognition are prepared in advance by the user or the operation system 10a.

[0040] For a pointing device such as a mouse, the arithmetic unit 12 writes the selected coordinates in a second file and stores them in the storage device 14. For example, the user clicks using the input device 15 while looking at the screen of the display device 16. The arithmetic unit 12 converts the clicked coordinates into coordinates on the screen of the display device 23 and writes them in the second file.

[0041] Regarding the keyboard, the acquisition device 11 acquires a video signal from the processing device 21 to the display device 23 when characters are input via the keyboard. The calculation device 12 displays a screen based on the video signal on the display device 16. On the displayed screen, the user points the pointer to a text box in which characters are to be input and selects the coordinates. This brings the text box into focus. The user inputs characters into the text box using the input device 15. The calculation device 12 writes the coordinates on the screen of the display device 23 and one or more characters to be input into the text box in a second file.

[0042] After an operation signal is sent to the processing device 21, the arithmetic device 12 performs image recognition to determine whether the processing device 21 has been operated in response to the operation signal. If it is determined that the processing device 21 has not been operated, the arithmetic device 12 may send a notice to the display device 16 or another terminal device that the processing device 21 cannot be operated. The file name of the template image used for image recognition, the range used for character recognition, the image and range used for color recognition, etc. are written in the second file.

[0043] In preparing a template image, the acquisition device 11 acquires a video signal from the processing device 21 to the display device 23 when performing image recognition. The calculation device 12 displays a screen based on the video signal on the display device 16. The user uses the input device 15 to select a portion of the displayed screen. The selected portion is stored in the storage device 14 as a template image. The user writes the file name of the template image to be referenced in a second file. For example, template images are prepared for all screens on which automatic operations are performed by the operation system 10a. When the processing device 21 is operated, a change occurs on the screen. For example, the displayed screen transitions to another screen, or a window is displayed. A partial image that is not included in the screen before the change but is included in the screen after the change is used as the template image.

[0044] In preparing the character recognition range, the acquisition device 11 acquires a video signal from the processing device 21 to the display device 23 when a character needs to be recognized. The calculation device 12 displays a screen based on the video signal on the display device 16. The user uses the input device 15 to confirm the range in which character recognition will be performed on the displayed screen. The user writes the range in a second file. Alternatively, the user may select two diagonal points of the range in which character recognition will be performed. The calculation device 12 converts the coordinates of the selected two diagonal points into coordinates on the screen of the display device 23. The calculation device 12 writes the converted coordinates in a second file.

[0045] In preparation for color recognition, the acquisition device 11 acquires a video signal from the processing device 21 to the display device 23 when color recognition is required. The calculation device 12 displays a screen based on the video signal on the display device 16. The user uses the input device 15 to select an area on the displayed screen where color recognition is to be performed. An image of the selected area is stored in the storage device 14. The user writes the area in a second file. Alternatively, the user may select two diagonal points of the area where color recognition is to be performed. The calculation device 12 converts the coordinates of the selected two diagonal points into coordinates on the screen of the display device 23. The calculation device 12 writes the converted coordinates in a second file.

[0046] In image recognition using a template image, the arithmetic unit 12 compares the screen based on the video signal with the template image and determines whether the screen contains an image similar to the template image. In character recognition, the arithmetic unit 12 cuts out an image of a pre-stored range. The arithmetic unit 12 performs character recognition processing such as Optical Character Recognition (OCR) on the image. In color recognition, the arithmetic unit 12 cuts out an image of a pre-stored range. The arithmetic unit 12 determines whether the colors of each part match between the pre-stored image and the cut-out image. By recognizing the image through these processes, the arithmetic unit 12 determines whether the operation to the processing unit 21 by the operation signal has been completed.

[0047] Figures 7(a) and 7(b) are schematic diagrams illustrating screens of a display device, Figure 8(a) is a schematic diagram illustrating a portion of a first file, and Figure 8(b) is a schematic diagram illustrating a portion of a second file. Creation of an operation file by the operation system 10a will be described with reference to a specific example. During part of the processing by the processing device 21, the screens shown in Fig. 7(a) and Fig. 7(b) are displayed on the display device 16 based on the video signal output from the processing device 21. Screen A1 shown in Fig. 7(a) displays icons B1 to B7 and information C1. Screen A2 shown in Fig. 7(b) displays icons B1 to B4, icons B11 to B14, and a schematic diagram D of the processing device 21.

[0048] The user moves the pointer P to the icon B3 on the screen A1 shown in FIG. 7(a) and selects the icon B3. The arithmetic unit 12 converts the selected coordinates into coordinates on the screen of the display device 23. The arithmetic unit 12 writes the operation of moving the pointer to the converted coordinates and the operation of selecting the converted coordinates in a first file. The arithmetic unit 12 writes the converted coordinates in a second file.

[0049] Screen A2 shown in FIG. 7(b) is displayed on display device 16. The user cuts out a template image from screen A2 for determining the transition from screen A1 to screen A2. For example, schematic diagram D or any of icons B11 to B14 is used as the template image. The user stores the cut-out template image as image data in storage device 14. The user writes in a second file the file name of the template image to be referenced when determining the transition from screen A1 to screen A2.

[0050] The user moves pointer P to icon B14 on screen A2 and selects icon B14. Arithmetic unit 12 converts the selected coordinates into coordinates on the screen of display device 23. Arithmetic unit 12 writes the action of moving the pointer to the converted coordinates and the action of selecting the coordinates in a first file. Arithmetic unit 12 writes the converted coordinates in a second file. Thereafter, the process of storing the template image and writing the file name in the second file is repeated in the same manner.

[0051] For example, as shown in Figures 8(a) and 8(b), a first file and a second file are prepared for operation by the operation system 10a. The first file describes operations to be executed by the arithmetic unit 12, such as generating an operation command for selecting a specific icon and referencing a template image, as described above. The second file describes the coordinates to be selected, the file name of the template image to be referenced, and so on.

[0052] The trigger for starting an operation by the operation system 10a is arbitrary. For example, the operation may be started based on a signal input to the processing device 21 or a signal sent from another sensor, measuring instrument, or the like. The operation may be started in response to a signal transmitted from another operation system 10a or a signal transmitted from another processing device 21 operated by another operation system 10a. The signal may be transmitted during operation by the other operation system 10a, or may be transmitted at a time other than during operation by the other operation system 10a. The operation may be started when a specific screen is displayed on the display device 23. The operation may be started when a user inputs a specific command to the arithmetic device 12.

[0053] FIG. 9 is a flowchart showing the procedure of an automatic operation method using an operation system according to a first modified example of the embodiment. The operation system 10a executes the operations of the flowchart shown in FIG. 5 so that the user can operate the processing device 21 using the input device 15. The user operates the processing device 21 so that the processing device 21 automatically executes the desired process. During the operation, a first file, reference data, and a second file are created (step S11). For example, the calculation device 12 writes an operation sequence corresponding to the user's operation in the first file. The calculation device 12 writes selected coordinates in the second file. The user stores the reference data in the storage device 14. The user writes the file name of the reference data, etc., in the second file.

[0054] The operation system 10a executes the automatic operation mode of the processing device 21 using the created first file, reference data, and second file (step S12).

[0055] According to the operation system 10a of the first modified example, the automatic operation mode can automatically operate the processing device 21. For example, the operation of the processing device 21 can be automated without modifying the processing device 21 or rewriting the program.

[0056] In the operation system 10a, an operation file is referenced when operating the processing device 21. The operation file includes a first file and a second file. The first file describes a generation operation for generating an operation command. The second file is created separately from the first file and is referenced when executing a generation operation based on the description in the first file.

[0057] In a manufacturing site, multiple processing devices 21 of the same type may be used. The multiple processing devices 21 may be automatically operated by multiple operation systems 10a. In this case, a common operation file and common reference data may be used. For example, when an operation file and reference data for operating one processing device 21 are prepared, the operation file and reference data are duplicated and used in operations by each operation system 10a.

[0058] Or, even between the same type of processing devices 21, the displayed screen may differ slightly. For example, the position of the same icon on the same screen may differ slightly between the processing devices 21. As a result, when selecting the coordinates described in the second file with a pointing device, the desired icon may not be selected. Even between the same type of processing devices 21, the appearance of the template image may differ due to the influence of noise, aging deterioration of the video signal output section of the processing device 21, etc. As a result, the accuracy of image recognition may decrease.

[0059] In conventional operation systems, when such differences exist between the processors 21, it is necessary to create a new program file that corrects for the differences. That is, even though the basic operations executed by the arithmetic unit 12, the reference data used, the characters entered, etc. are almost the same between the processors 21, it is necessary to recreate the program file for generating operation commands.

[0060] In the operation system 10a, information used when generating an operation command is written in a second file separate from the first file. For example, if there is a deviation in the coordinates of an icon, the description of the coordinates can be changed to correct the deviation. If the appearance of a template image differs, a template image with a different appearance can be prepared, and the description of the second file can be changed to reference that image data. According to the embodiment, machine differences can be easily corrected, reducing the burden on the user required to prepare the operation system 10a.

[0061] The second file is, for example, a text file, so that even a user without programming knowledge can write the second file in text format and easily modify the second file.

[0062] The reference data may be automatically generated by the operation system 10a. For example, when any coordinate is selected, the arithmetic device 12 saves a screen before selection and a screen after selection. The arithmetic device 12 detects changes in the feature amounts of the screen after selection relative to the feature amounts of the screen before selection. The arithmetic device 12 extracts an area where the feature amounts change significantly from the screen after selection. The arithmetic device 12 stores the extracted area as a template image in the storage device 14. The arithmetic device 12 writes the file name of the template image in a second file. Automatic generation of reference data reduces the burden on the user of preparing reference data.

[0063] Before starting the production line, the user may try out the automatic operation mode to check whether there are any problems with the operation system 10a. For example, in the automatic operation mode trial, the user checks whether icons are selected appropriately, whether images are recognized correctly, etc. If a problem with the automatic operation mode is confirmed, the user corrects the description of the second file as appropriate.

[0064] The arithmetic device 12 may automatically correct the coordinates described in the second file. For example, in a trial of the automatic operation mode, the arithmetic device 12 generates a selection operation command for selecting the coordinates described in the second file and transmits it to the generation device 13. The generation device 13 generates an operation signal based on the selection operation command and transmits it to the processing device 21. The arithmetic device 12 recognizes the image acquired by the acquisition device 11 after the operation signal is transmitted and determines whether a prepared template image is displayed. If the template image is not displayed, the arithmetic device 12 generates another selection operation command for selecting another coordinate different from the coordinates described in the second file. For example, the other coordinate is randomly determined from among coordinates near the coordinates described in the second file. The arithmetic device 12 transmits another selection operation command to the generation device 13. The generation device 13 generates another operation signal based on the other selection operation command and transmits it to the processing device 21. The generation of another selection operation command is repeated, for example, until a prepared template image is displayed or the number of times a selection operation command has been generated reaches a predetermined value. The arithmetic unit 12 generates another selection operation command to select another coordinate different from the previous coordinate.

[0065] FIG. 10 is a flowchart showing a part of the automatic operation mode trial by the operation system according to the embodiment. For example, in the flowchart shown in Fig. 9, between steps S1 and S2, an automatic operation mode using the first file, the second file, and the reference data is tried. In part of the trial, the arithmetic device 12 generates a selection operation command for selecting a part of the screen (step S21). The generation device 13 generates an operation signal based on the selection operation command and transmits it to the processing device 21. Based on the screen recognition result, the arithmetic device 12 determines whether the operation on the processing device 21 by the operation signal has been completed (step S22).

[0066] If the operation is not completed, the arithmetic device 12 determines whether the number of attempts of the selection operation is less than a specified number (step S23). If the number of attempts is less than the specified number, the arithmetic device 12 generates another selection operation command for selecting another coordinate (step S24). The generation device 13 generates an operation signal based on the selection operation command and transmits it to the processing device 21. If the number of attempts of the selection operation reaches the specified number, the arithmetic device 12 notifies the user (step S25). For example, the display device 16 displays information informing the user that the operation cannot be performed appropriately.

[0067] When it is determined in step S22 that the operation according to the selection operation command has been completed, the arithmetic unit 12 corrects the coordinates described in the second file to the coordinates used in the selection operation command that successfully executed the selection operation (step S26). Note that if the selection operation was successfully executed using the original coordinates described in the second file, the second file is not corrected.

[0068] When it is necessary to correct the coordinates written in the second file, the coordinates are automatically corrected by the process according to the flowchart shown in Fig. 10. This improves the convenience of the operation system 10a.

[0069] It should be noted that when an operation signal is input to the processing device 21, the response from the processing device 21 may be delayed. When it is determined whether the operation corresponding to the selection operation command has been completed, the screen may not have changed due to the delay, and the screen may change normally after the determination. For this reason, for a processing device 21 that may experience a delay in displaying the screen on the display device 23, the interval for transmitting the operation signal may be set longer, or the operation signal based on the selection operation command may be transmitted multiple times. For example, in the flowchart shown in FIG. 10, if an operation is not completed after a certain operation signal is input to the processing device 21, the same operation signal may be input again from the generating device 13 to the processing device 21.

[0070] (Second Modification) FIG. 11 is a schematic diagram showing an operation system according to a second modified example of the embodiment. An input display device having functions as an input device and a display device may be connected to the processing device 21. For example, as shown in FIG. 11 , a touch panel 24 is provided as the input display device. An operation system 10b according to a second modified example further includes a switching device 17. The processing system 1b includes the operation system 10b and the processing device 21.

[0071] The generating device 13 generates an operation signal corresponding to a signal transmitted from the touch panel 24. The switching device 17 switches the signal input to the processing device 21. Specifically, the signal transmitted from the touch panel 24 and the operation signal transmitted from the generating device 13 are input to the switching device 17. The switching device 17 switches between a state in which the signal transmitted from the touch panel 24 is input to the processing device 21 and a state in which the operation signal transmitted from the generating device 13 is input to the processing device 21. During operation by the operation system 10b, the switching device 17 inputs the operation signal to the processing device 21.

[0072] The switching device 17 can prevent the user from operating the processing device 21 via the touch panel 24 while the operation system 10b is in operation. For example, it can reduce the possibility that the automatic operation mode by the operation system 10b will be interrupted due to a transition to another screen caused by a user operation.

[0073] The switching device 17 includes, for example, a relay circuit or a switching circuit. By providing the switching device 17, the processing device 21 in which the touch panel 24 is used can be operated by the operation system 10b.

[0074] Even if the touch panel 24 is the same type of device, there is a large discrepancy in the coordinates of the generated operation signal. This is because the positional relationship between the coordinates of the sensor that detects the touch and the display is adjusted individually. Therefore, even if multiple touch panels 24 of the same type are used for multiple processing devices 21, the coordinates selected or referenced during automatic operation may need to be corrected for each processing device 21.

[0075] In this case, by correcting the coordinates described in the second file, the machine difference regarding the deviation of the coordinates can be easily corrected. For this reason, the invention according to the embodiment is particularly suitable for a processing device 21 that uses a touch panel 24. According to the second modification, the burden on the user for preparing the automatic operation mode of the processing device 21 connected to the touch panel 24 can be reduced.

[0076] (Third Modification) FIG. 12 is a schematic diagram showing an operation system according to a third modified example of the embodiment. The operation system 10c according to the third modified example further includes a switching device 17, as compared with the operation system 10. A signal transmitted from the generation device 13 or the input device 22 to the processing device 21 is input to the processing device 21 via the switching device 17. The switching device 17 switches between a state in which a signal transmitted from the input device 22 is input to the processing device 21 and a state in which an operation signal transmitted from the generation device 13 is input to the processing device 21.

[0077] During operation by the operation system 10c, the switching device 17 inputs an operation signal to the processing device 21. This prevents the user from operating the processing device 21 using the input device 22 during operation by the operation system 10c. For example, it is possible to reduce the possibility that the automatic operation mode will be interrupted by a user operation. According to the third modification, it is possible to improve the convenience of the processing system 1c including the operation system 10c and the processing device 21.

[0078] (Fourth Modification) FIG. 13 is a schematic diagram showing an operation system according to a fourth modified example of the embodiment. As shown in FIG. 13 , an operation system 10d according to the fourth modification further includes reading devices 18a and 18b compared to the operation system 10. The reading devices 18a and 18b read identification information. As the identification information, a one-dimensional barcode, a two-dimensional barcode, or a radio frequency identifier (RFID) can be used. The reading devices 18a and 18b are barcode readers or RFID readers. For example, the reading device 18a reads the identification information of an operator in charge of the processing device 21. The reading device 18b reads the identification information of a workpiece to be processed by the processing device 21.

[0079] For example, when executing processing by the processing device 21, the user (operator) places a workpiece in a location where the reading device 18b can read the workpiece's identification information. The user then has the reading device 18a read his / her own identification information. The arithmetic device 12 starts operating the processing device 21 in response to the reading of the identification information by the reading device 18a. For example, the arithmetic device 12 has the reading device 18b read the workpiece's identification information, and then starts generating an operation command.

[0080] The operation system 10d may generate an operation signal according to the read identification information and input it to the processing device 21. Alternatively, the identification information may be directly input from the arithmetic device 12 to the processing device 21 via a communication port by serial communication or the like.

[0081] As in the fourth modified example, the operation system 10d may include other devices related to the processing performed by the processing device 21. The operation of the processing device 21 may be started in response to the operation of the other devices. This eliminates the need for the user to operate the input device 15 or 22 to start the operation. According to the fourth modified example, the convenience of the operation system 10d can be improved. The convenience of the processing system 1d including the operation system 10d and the processing device 21 can be improved.

[0082] (Fifth Modification) FIG. 14 is a schematic diagram showing an operation system according to a fifth modified example of the embodiment. 14, an operation system 10e according to the fifth modified example is connected to a management device 30 (second device). The processing system 1e includes the operation system 10e, a processing device 21, and the management device 30. For example, the arithmetic device 12 is connected to the management device 30 via a network, wired communication, or wireless communication. The management device 30 is a higher-level device that transmits commands to the processing device 21.

[0083] When the management device 30 transmits a command to the processing device 21, the operation system 10e receives the command. The operation system 10e executes an operation in accordance with the command. For example, the calculation device 12 generates an operation command in accordance with the command and transmits it to the generation device 13.

[0084] For example, a plurality of operation files are stored in the storage device 14. The arithmetic device 12 selects one of the plurality of operation files in response to a command sent from the management device 30. The arithmetic device 12 executes a program included in the selected operation file to generate an operation command. Alternatively, the command sent from the management device 30 may be used as a trigger to start an operation.

[0085] The operation system 10e may input parameters related to the processing to be performed by the processing device 21 to the processing device 21. The parameters include the number of workpieces to be processed, conditions for performing the processing, etc. The conditions include, for example, temperature, pressure, gas or liquid flow rate, current, voltage, and processing time. The operation system 10e generates an operation signal for inputting the numerical values ​​of the parameters and transmits it to the processing device 21. As a result, the numerical values ​​are input into text boxes, etc. displayed on the display device 23.

[0086] The management device 30 may transmit parameters input from the operation system 10e to the processing device 21 to the operation system 10e. A command transmitted from the management device 30 to the processing device 21 includes the parameters. The operation system 10e receives the numerical value input as the parameter and generates an operation signal corresponding to the numerical value. As a result, the numerical value transmitted from the management device 30 is input to the processing device 21.

[0087] The management device 30 may transmit information about the work to be processed to the operation system 10e. A command transmitted from the management device 30 to the processing device 21 includes information about the work to be processed. For example, the management device 30 transmits identification information (ID) of the work to the operation system 10e. The operation system 10e inputs the identification information to the processing device 21 via a communication port. Alternatively, the operation system 10e may generate an operation signal according to the identification information and input it to the processing device 21.

[0088] At manufacturing sites, a higher-level device may be installed to manage multiple processing devices. However, due to incompatibility of the communication interface between the processing device 21 and the management device 30, differences in communication standards, and the like, direct command transmission from the management device 30 to the processing device 21 may not be possible. For example, if the processing device 21 is old, the processing device 21 may not be able to accept commands transmitted from the management device 30. In this case, use of the operation system 10e allows the management device 30 to manage the processing device 21. For example, there is no need to modify the processing device 21, and production efficiency can be more easily improved. Use of the operation system 10e allows the operating status of one or more processing devices 21 to be transmitted to the management device 30. This allows the management device 30 to easily obtain information regarding the operating rate or operating status of one or more processing systems 1e.

[0089] (Sixth Modification) FIG. 15 is a schematic diagram showing a processing system according to a sixth modified example of the embodiment. In a processing system 1f according to the sixth modified example, a management device 30 is connected to a plurality of operation systems 10e. The plurality of operation systems 10e are each connected to a plurality of processing devices 21. The management device 30 transmits commands to the plurality of processing devices 21. The plurality of operation systems 10e operate the plurality of processing devices 21 in accordance with the commands transmitted from the management device 30. The management device 30 may select one or more of the plurality of processing devices 21 and transmit a command to the selected processing devices 21.

[0090] According to the processing system 1f, the management device 30 can acquire information on the operation rates or operation states of the plurality of processing devices 21, and the production efficiency of the plurality of processing devices 21 can be improved.

[0091] The operation of the processing device 21 by any one of the operation systems 10e may be initiated in response to a signal transmitted from another operation system 10e or a signal transmitted from another processing device 21 operated by another operation system 10e. The signal may be transmitted during operation by another operation system 10e, or may be transmitted at a time other than during operation by another operation system 10e. By transmitting and receiving signals for initiating operation among three or more of the multiple operation systems 10e, the multiple processing devices 21, and the management device 30, for example, the multiple processing devices 21 can be operated efficiently.

[0092] (Seventh Modification) FIG. 16 is a schematic diagram showing an operation system according to a seventh modified example of the embodiment. In a processing system 1g according to the seventh modification, an operation system 10g is connected to a terminal device 35. The operation system 10g is set to be remotely operable by the terminal device 35. The terminal device 35 can remotely operate the arithmetic device 12 of the operation system 10g.

[0093] A user can transmit an input operation to the arithmetic device 12 by operating the terminal device 35. The operation system 10 operates the processing device 21 according to the input operation transmitted from the terminal device 35. By operating the terminal device 35, the user can prepare an operation file, execute an automatic operation mode by the operation system 10g, and the like.

[0094] Even when the operation system 10g is provided adjacent to the processing device 21, the operation system 10g can be operated from a location distant from the operation system 10g using the terminal device 35. According to the seventh modification, the convenience of the processing system 1g can be improved.

[0095] (Eighth Modification) FIG. 17 is a schematic diagram showing an operation system according to an eighth modified example of the embodiment. In a processing system 1h according to the eighth modification, a plurality of operation systems 10g are connected to a terminal device 35. The terminal device 35 can remotely control the arithmetic devices 12 of the plurality of operation systems 10g.

[0096] According to the eighth modification, a plurality of operation systems 10g can be operated by a single terminal device 35. Settings such as an automatic operation mode for operating a plurality of processing devices 21 can be performed at a location away from the plurality of operation systems 10g. According to the eighth modification, the convenience of the processing system 1h can be improved.

[0097] FIG. 18 is a schematic diagram showing a hardware configuration. Each of the arithmetic device 12, the control unit 21a, the management device 30, and the terminal device 35 includes, for example, the configuration of a computer 90 shown in Fig. 18. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0098] The ROM 92 stores a program that controls the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to perform each of the above-described processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0099] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a work memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During program execution, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0100] The storage device 94 stores data necessary for executing the program and data obtained by executing the program. The storage device 94 may be used as the storage device 14.

[0101] An input interface (I / F) 95 connects the computer 90 and an input device 95a. The input I / F 95 is, for example, a serial bus interface such as USB. The CPU 91 can read various data from the input device 95a via the input I / F 95. The input device 95a is used as the input device 15 or 22.

[0102] The output interface (I / F) 96 connects the computer 90 and a display device 96a. The output I / F 96 is, for example, a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HDMI (registered trademark)). The CPU 91 can transmit data to the display device 96a via the output I / F 96 and display a screen on the display device 96a. The display device 96a is used as the display device 16 or 23.

[0103] The communication interface (I / F) 97 connects the computer 90 to a server 97a external to the computer 90. The communication I / F 97 is, for example, a network card such as a LAN card. The CPU 91 can read various data from the server 97a via the communication I / F 97.

[0104] Two or more computers 90 may cooperate to function as the arithmetic device 12, the control unit 21a, the management device 30, or the terminal device 35. The generation device 13 may be incorporated into a computer that functions as the arithmetic device 12, and the computer may function as both the arithmetic device 12 and the generation device 13. A program having the functions of the arithmetic device 12 and a program having the functions of the generation device 13 may be implemented in one computer.

[0105] The various data processing operations described above may be recorded as a computer-executable program on a magnetic disk (such as a flexible disk or hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, or DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0106] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, a computer reads a program from the recording medium and causes a CPU to execute instructions written in the program based on the program. The computer may acquire (or read) the program via a network.

[0107] According to the operation system, processing system, or operation method described above, the processing device 21 can be easily remotely controlled, improving the convenience of the processing device 21. The same effect can be obtained by using a program that causes a computer to execute the operation method. According to the construction method of a processing system that constructs an operation system and connects it to a first device, a processing system that allows easy operation of the processing device 21 can be provided.

[0108] Implementations of the invention may include the following features. (Feature 1) Acquire a video signal from a first device that is an operation target; displaying a screen based on the video signal on a display device; In response to an input operation from a user, generate an operation command corresponding to the input operation; an operation system that generates an operation signal corresponding to the first device based on the operation command and transmits the operation signal to the first device; (Feature 2) 2. The operating system according to feature 1, wherein the screen based on the video signal is enlarged or reduced in size according to the resolution of the display device. (Feature 3) the input operation includes a selection on the screen, The operation system according to feature 2, wherein the coordinates selected on the screen are converted into coordinates on a screen of a display device connected to the first device and included in the operation command. (Feature 4) 4. The operation system according to any one of features 1 to 3, wherein the input operation is described in an operation file including operation content for the first device. (Feature 5) 5. The operating system according to claim 4, wherein the operating system is capable of executing an automatic operating mode in which the operating instructions are generated based on the operation file. (Feature 6) 6. The operation system according to feature 5, wherein in the automatic operation mode, it is determined whether the operation to the first device by the operation signal has been completed based on the screen recognition result. (Feature 7) generating a template image based on the change in the feature amount on the screen; 6. The operation system according to feature 5, wherein in the automatic operation mode, it is determined whether the operation on the first device by the operation signal has been completed based on a result of comparing the screen with the template image. (Feature 8) receiving a command from a second device that transmits the command intended for the first device; 8. The operation system according to any one of features 5 to 7, wherein the automatic operation mode is executed in response to the command. (Feature 9) 9. The operating system according to feature 8, wherein in the automatic operating mode in response to the command, the operating command corresponding to the command is generated. (Feature 10) selecting one of the plurality of operation files in the automatic operation mode in response to the command; 10. The operating system according to feature 8 or 9, wherein the operating command is generated based on the selected operation file. (Feature 11) 11. The operation system according to any one of features 8 to 10, wherein the command includes information on a parameter to be input to the first device or information on a workpiece to be processed by the first device. (Feature 12) 12. The operation system according to any one of features 5 to 11, wherein the automatic operation mode is executed in response to reading of identification information by a reading device. (Feature 13) An operation system according to any one of features 1 to 12; The first device performs processing of a workpiece; A processing system comprising: (Feature 14) A plurality of the first devices; a plurality of operation systems that transmit the operation signals to the plurality of first devices, respectively; a terminal device that transmits the input operation to the plurality of operation systems; 14. The processing system according to claim 13, comprising: (Feature 15) an operating system coupled to a first device, Acquiring a video signal from the first device; displaying a screen based on the video signal; In response to an input operation from a user, generate an operation command corresponding to the input operation; generating an operation signal corresponding to the first device based on the operation command and transmitting the operation signal to the first device; constructing the operating system; A method for constructing a processing system, wherein the operation system is set so as to be remotely operable from a terminal device. (Feature 16) In response to an input operation from a user, generate an operation command corresponding to the input operation; generating an operation signal corresponding to a first device that is an operation target based on the operation command; A computer that transmits the operation signal to the first device. (Feature 17) 17. The computer according to claim 16, wherein the input operation is described in an operation file including operation content for the first device. (Feature 18) 20. The computer of claim 17, wherein the computer is capable of executing an automatic operation mode for generating the operation instructions based on the operation file. (Feature 19) Acquire a video signal from a first device that is an operation target; displaying a screen based on the video signal on a display device; In response to an input operation from a user, generate an operation command corresponding to the input operation; an operation method for generating an operation signal corresponding to the first device based on the operation command and transmitting the operation signal to the first device; (Feature 20) the input operation includes a selection on the screen, The screen based on the video signal is enlarged or reduced in accordance with the resolution of the display device, and displayed; 20. The operating method according to feature 19, further comprising converting the coordinates selected on the screen into coordinates on a screen of a display device connected to the first device and including the coordinates in the operating command. (Feature 21) The operating method according to feature 19 or 20, further comprising determining an operation performed on the screen, and writing the operation to an operation file including details of the operation on the first device. (Feature 22) 22. The operating method according to feature 21, further comprising generating the operating instructions based on the operation file. (Feature 23) A program that causes a computer to execute the operating method according to any one of features 19 to 22. (Feature 24) A storage medium storing a program for causing a computer to execute the operating method according to any one of Features 19 to 22.

[0109] Although several embodiments of the present invention have been described above, these embodiments are presented by way of example only and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as set forth in the claims. Furthermore, the above-described embodiments can be implemented in combination with each other.

Claims

1. The video signal is acquired from the first device, which is the target of the operation. The screen based on the aforementioned video signal is displayed on the display device. In response to user input, the system generates an operation command corresponding to the input operation. An operating system that generates an operating signal corresponding to the first device based on the aforementioned operating command and transmits it to the first device, An operating system comprising a switching device capable of receiving a signal transmitted from an input device to the first device and the operation signal, and configured to transmit only one of the signal and the operation signal to the first device.

2. The switching device is capable of switching between a first state in which the signal is input to the first device and a second state in which the operation signal is input to the first device. The operating system according to claim 1, wherein the system is set to the second state when the aforementioned operating signal is generated.

3. The operating system according to claim 2, wherein the switching device includes a switching circuit for switching between the first state and the second state.

4. The operating system according to claim 3, wherein the switching circuit includes at least one of a relay circuit and a switching circuit.

5. The acquisition device includes an acquisition device that acquires the video signal and transmits the video signal to the display device, The operating system according to any one of claims 1 to 4, wherein the acquisition device includes a distributor and a capture unit.

6. comprising a generating device for generating the operation signal, The operating system according to any one of claims 1 to 4, wherein the generating device includes a microcontroller.

7. The operating system according to any one of claims 1 to 4, wherein the first device is configured to process a workpiece.

8. The operating system according to claim 7, wherein the process is at least one selected from conveying, processing, washing, heating, cooling, and drying.

9. The operating system according to any one of claims 1 to 4, wherein the screen based on the video signal is enlarged or reduced in size according to the resolution of the display device.

10. The input operation includes a selection on the screen, The operation system according to claim 9, wherein the coordinates selected on the screen are converted into coordinates on the screen of a display device connected to the first device and included in the operation command.

11. The operation system according to any one of claims 1 to 4, wherein the input operation is described in an operation file that includes the content of the operation to the first device.

12. The operating system according to claim 11, which is capable of executing an automatic operation mode that generates operation commands based on the operation file.

13. The operation system according to claim 12, wherein in the automatic operation mode, it is determined whether the operation on the first device by the operation signal has been completed based on the recognition result of the screen.

14. A template image is generated based on the changes in the feature quantities in the screen, The operation system according to claim 12, wherein in the automatic operation mode, it is determined whether the operation on the first device by the operation signal has been completed based on the result of comparing the screen with the template image.

15. A second device that transmits a command to the first device receives the command, The operating system according to claim 12, which executes the automatic operation mode in response to the aforementioned command.

16. The operation system according to claim 15, which generates the operation command corresponding to the command in the automatic operation mode corresponding to the command.

17. In the automatic operation mode corresponding to the command, one operation file is selected from a plurality of operation files, The operating system according to claim 15, which generates the operation command based on the selected operation file.

18. The operating system according to claim 15, wherein the instruction includes parameters input to the first device or information relating to a workpiece processed by the first device.

19. The operating system according to claim 12, wherein the automatic operation mode is executed in response to the reading of identification information by the reading device.

20. An operating system according to any one of claims 1 to 4, The first apparatus that performs workpiece processing, A processing system equipped with [the following features].

21. A plurality of the first devices, A plurality of operating systems that transmit the operation signals to the plurality of first devices, A terminal device that transmits the input operation to the plurality of operating systems, The processing system according to claim 20, comprising:

22. Acquire a video signal from the first device which is the target of the operation, The screen based on the aforementioned video signal is displayed on the display device. In response to user input, the system generates an operation command corresponding to the input operation. An operating method comprising generating an operating signal corresponding to the first device based on the aforementioned operating command and transmitting it to the first device, An operating method for transmitting the operation signal to the first device using a switching device that is capable of receiving a signal transmitted from an input device to the first device and the operation signal, and is configured to transmit only one of the signal and the operation signal to the first device.

23. The input operation includes a selection on the screen, The screen based on the video signal is displayed by enlarging or reducing it according to the resolution of the display device. The operation method according to claim 22, wherein the coordinates selected on the screen are converted into coordinates on the screen of a display device connected to the first device and included in the operation command.

24. The operation method according to claim 22 or 23, wherein the operation performed on the screen is determined and the operation is described in an operation file that includes the details of the operation to the first device.

25. The operation method according to claim 24, wherein the operation command is generated based on the operation file.

26. A program that causes a computer to execute the operation method described in claim 22 or 23.

27. ​​A storage medium storing a program that causes a computer to execute the operation method described in Claim 22 or 23.

Images