Remote monitoring system, remote monitoring method, data processing apparatus, data processing method, terminal apparatus, method for controlling terminal apparatus, and program

Abstract

To inhibit delays in the acquisition of monitoring information by a terminal apparatus and to enable easy remote control according to the status of a monitoring target.SOLUTION: A data processing apparatus includes: a generation unit that generates streaming data using image data captured by an image capturing device; a distribution unit that distributes the streaming data; and a robot command unit that issues at least one of an operation instruction to a robot or an operation instruction to a power supply, on the basis of an instruction from a terminal apparatus. The terminal apparatus includes: a display control unit that receives the streaming data distributed from the data processing apparatus and displays the received streaming data on a monitoring screen; and an instruction receiving unit that receives the instruction for the robot on the monitoring screen and transmits the received instruction to the data processing apparatus.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a remote monitoring system, a remote monitoring method, a data processing device, a data processing method, a terminal device and a control method therefor, and a program. [Background technology]

[0002] In recent years, remote monitoring using image data acquired by visual sensors has been adopted at production sites in various industries. This type of remote monitoring allows workers to grasp the situation at the production site while performing other tasks, and to check the production status in places where people cannot enter, and is therefore used to improve worker efficiency.

[0003] One example of a field where remote monitoring is used is the welding field, and there is a technology for monitoring the welding status as disclosed in Patent Document 1. Patent Document 1 discloses a configuration for providing feedback on the welding work to the welding worker, with the aim of enabling the welding worker to perform welding work as if he or she were in the same location as the welding robot. As a result, the configuration of Patent Document 1 makes it possible to perform various types of welding work remotely. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-505391 Summary of the Invention [Problem to be solved by the invention]

[0005] The method of Patent Document 1 is a technology for remotely performing various types of welding work. However, it does not take into consideration what specific information can be monitored and to what extent. In welding work, there is a lot of information to be monitored, and the greater the amount of information, the more likely it is that the information monitored on the terminal side will be delayed. Therefore, the difficulty of real-time monitoring remains an issue. Furthermore, it does not take into consideration remote operation based on the status of the monitored object. In order to further improve work efficiency, a function that can easily perform remote operation according to the status of the monitored object is desired.

[0006] In view of the above-mentioned problems, the present invention aims to suppress delays in the acquisition and presentation of monitoring information on terminal devices and to enable easy remote operation according to the status of the monitored object. [Means for solving the problem]

[0007] In order to solve the above problems, the present invention has the following configuration: Robots and an imaging device that captures an image of the operating area of ​​the robot; a power supply that controls the output of a current or a voltage; a data processing device that processes data related to the operation of the robot or the power source; a terminal device configured to be able to communicate with the data processing device via a network; A remote monitoring system comprising: The data processing device includes: a generating unit that generates streaming data using image data captured by the imaging device; a distribution unit that distributes the streaming data; a robot command unit that issues at least one of an operation instruction to the robot or an operation instruction to the power source based on at least an instruction from the terminal device; and The terminal device a display control unit that receives streaming data distributed from the data processing device and displays it on a monitoring screen; an instruction receiving unit that receives an instruction to the robot or the power source on the monitoring screen and transmits the instruction to the data processing device; It has.

[0008] Another aspect of the present invention has the following configuration: Robots and an imaging device that captures an image of the operating area of ​​the robot; a power supply that controls the output of a current or a voltage; a data processing device that processes data related to the operation of the robot or the power source; a terminal device configured to be able to communicate with the data processing device via a network; A method for remotely monitoring a system, comprising: In the data processing device, a generating step of generating streaming data using image data captured by the imaging device; a distribution step of distributing the streaming data; a robot command step of issuing at least one of an operation instruction to the robot or an operation instruction to the power source based on at least an instruction from the terminal device; and In the terminal device, a display control step of receiving streaming data distributed from the data processing device and displaying the data on a monitoring screen; an instruction receiving step of receiving an instruction for the robot or the power source on the monitoring screen and transmitting the instruction to the data processing device; It has.

[0009] Another aspect of the present invention has the following configuration: A data processing device that processes data related to the operation of a robot, a generating unit that generates streaming data using image data of the robot's motion area captured by an imaging device; a distribution unit that distributes the streaming data to a terminal device via a network; a robot command unit that issues at least one of an operation instruction to the robot and an operation instruction to a power source based on an instruction from the terminal device received at least via the network; It has.

[0010] Another aspect of the present invention has the following configuration: A terminal device configured to be able to communicate with a data processing device that processes data related to the operation of a robot or a power source via a network, a display control unit that receives streaming data distributed from the data processing device and displays it on a monitoring screen; an instruction receiving unit that receives an instruction to the robot or the power source on the monitoring screen and transmits the instruction to the data processing device; It has.

[0011] Another aspect of the present invention has the following configuration: A data processing method for processing data related to the operation of a robot, comprising: a generation step of generating streaming data using image data of the robot's motion area captured by an imaging device; a distribution step of distributing the streaming data to a terminal device via a network; a robot command step of issuing at least one of an operation instruction to the robot and an operation instruction to a power source based on an instruction from the terminal device received at least via the network; It has.

[0012] Another aspect of the present invention has the following configuration: A control method for a terminal device configured to be able to communicate with a data processing device that processes data related to the operation of a robot or a power source via a network, comprising: a display control step of receiving streaming data distributed from the data processing device and displaying the data on a monitoring screen; an instruction receiving step of receiving an instruction for the robot or the power source on the monitoring screen and transmitting the instruction to the data processing device; It has.

[0013] Another aspect of the present invention has the following configuration: A program for a computer, a generating unit that generates streaming data using image data of the robot's motion area captured by an imaging device; a distribution unit that distributes the streaming data to a terminal device via a network; a robot command unit that issues operation instructions to the robot or a power source based on instructions received from the terminal device via at least the network; Execute the following.

[0014] Another aspect of the present invention has the following configuration: A program for a computer configured to be able to communicate with a data processing device or a power source via a network, the computer processing data relating to the operation of a robot, a display control unit that receives streaming data distributed from the data processing device and displays it on a monitoring screen; an instruction receiving unit that receives an instruction to the robot or the power source on the monitoring screen and transmits the instruction to the data processing device; Execute the following. [Effects of the Invention]

[0015] According to the present invention, delays in the acquisition and presentation of monitoring information in a terminal device can be reduced, and remote control can be easily performed according to the status of the monitored object. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a schematic diagram showing an example of a system configuration according to an embodiment of the present invention; [Figure 2] FIG. 1 is a block diagram showing an example of the functional configuration of a system according to an embodiment of the present invention. [Figure 3] 10 is a flowchart of a monitoring process on the information processing device side according to an embodiment of the present invention. [Figure 4] 10 is a flowchart of a monitoring process on the terminal device side according to an embodiment of the present invention. [Figure 5] FIG. 2 is a schematic diagram showing an example of the configuration of a UI screen of a terminal device according to an embodiment of the present invention. [Figure 6A] FIG. 2 is a schematic diagram showing an example of the configuration of a UI screen of a terminal device according to an embodiment of the present invention. [Figure 6B] FIG. 2 is a schematic diagram showing an example of the configuration of a UI screen of a terminal device according to an embodiment of the present invention. [Figure 6C] FIG. 2 is a schematic diagram showing an example of the configuration of a UI screen of a terminal device according to an embodiment of the present invention. [Figure 6D] FIG. 2 is a schematic diagram showing an example of the configuration of a UI screen of a terminal device according to an embodiment of the present invention. [Figure 6E] FIG. 2 is a schematic diagram showing an example of the configuration of a UI screen of a terminal device according to an embodiment of the present invention. [Figure 6F] FIG. 2 is a schematic diagram showing an example of the configuration of a UI screen of a terminal device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. Note that the embodiment described below is one embodiment for explaining the present invention and is not intended to be interpreted as limiting the present invention. Furthermore, not all of the configurations described in each embodiment are necessarily essential configurations for solving the problems of the present invention. Furthermore, in each drawing, the same components are assigned the same reference numerals to indicate corresponding relationships.

[0018] First Embodiment An embodiment of the present invention will be described below with reference to the drawings. In this embodiment, an example of a welding system including a welding robot capable of remote monitoring and remote operation will be described. However, the remote monitoring method according to this embodiment is not limited to a configuration applied to a welding robot, but can be widely applied to devices that use robots, such as slag removal devices, cutting devices, gouging devices, non-destructive testing devices, object gripping devices, and transport devices. Furthermore, the configuration of the welding system shown below is an example and is not limited to this.

[0019] FIG. 1 is a schematic diagram showing an example of the configuration of a welding system according to this embodiment. As shown in FIG. 1, the welding system 50 includes a welding robot 100, a feeder 300, a welding power source 400, a shielding gas supply source 500, a robot control device 600, an imaging device 700, a data processing device 800, and a communication device 900. When the features of this embodiment are applied to a six-axis welding robot, additional components may be included in accordance with the configuration. Furthermore, the components constituting the welding system 50 are communicatively connected via various wired or wireless communication methods. The communication method used here is not limited to one, and multiple communication methods may be combined for connection.

[0020] Welding system 50 is configured to be able to communicate with terminal device 1000 via network 1100. Welding system 50 and terminal device 1000 may be installed at different locations within the same base, or may be installed at different bases.

[0021] (Robot control device) Robot controller 600 is connected to portable welding robot 100 via robot control cable 610, and to welding power source 400 via power supply control cable 620. Robot controller 600 has a data storage unit 601 that stores teaching data that predetermines the operation pattern, welding start position, welding end position, operation conditions, welding conditions, etc. of welding robot 100. Robot controller 600 sends instruction information as commands to welding robot 100 and welding power source 400 based on this teaching data, and controls the operation and welding conditions of welding robot 100 and welding power source 400.

[0022] The robot control device 600 also includes a control unit 602. The control unit 602 includes, for example, a groove shape information calculation unit 603, a welding condition acquisition unit 604, a communication control unit 605, and a control signal calculation unit 606. The groove shape information calculation unit 603 calculates groove shape information from detection data obtained by performing sensing such as touch sensing on the groove shape before welding. The welding condition acquisition unit 604 corrects and acquires the welding conditions of the teaching data based on the groove shape information. The communication control unit 605 and the control signal calculation unit 606 will be described later using FIG. 2.

[0023] After starting welding under the welding conditions of the teaching data corrected by sensing, the data processing device 800 extracts feature information (hereinafter also referred to as feature quantities) based on the image data acquired from the imaging device 700 during welding. The robot control device 600 then receives correction signals for various processes as command information from the feature quantities. While the feature quantities are not particularly limited, the present embodiment will be described using coordinate data of feature points indicated by the image data as an example. The robot control device 600 sequentially performs control operations such as gap adjustment, rod operation adjustment, tracking / amplitude adjustment, and speed adjustment. In this manner, the robot control device 600 performs each control operation in accordance with the correction signals received during welding, and then outputs status update information for various control conditions to the welding robot 100, the welding power source 400, the terminal device 1000, the data processing device 800, and the like. The robot control device 600 then repeats this process until the desired welding is completed. Note that the control signals based on the feature quantities extracted from the image data may be generated by either the data processing device 800 or the robot control device 600. Alternatively, one may generate the signal and then the other may correct it.

[0024] Furthermore, the robot control device 600 shown in FIG. 1 has a controller (hereinafter also referred to as a "teaching pendant") for teaching and manually operating the welding robot 100, and a controller with other control functions. These controllers may be formed integrally. From the viewpoint of usability at a welding site, it is preferable to separate the controllers into two: one for teaching and manually operating the portable welding robot 100, and one with other control functions. In this embodiment, signals are sent using the robot control cable 610 and the power supply control cable 620, but this is not limiting and signals may be sent wirelessly.

[0025] In this embodiment, the terminal device 1000 used for remote monitoring also functions as a controller. An operation example will be described later, but the operator of the terminal device 1000 issues desired instructions while referring to video image data (hereinafter also referred to as "streaming data") distributed by the data processing device 800. The robot control device 600 receives the instructions issued by the terminal device 1000 directly or via the data processing device 800, and performs control related to welding by issuing commands to, for example, the welding robot 100 or the welding power source 400 based on the instructions.

[0026] 1 shows one welding robot 100 as the monitoring target, but this is not limiting. For example, the welding status of each of the welding robots 100 may be captured by each of the multiple imaging devices 700 and monitored by one data processing device 800 or one terminal device 1000. In this case, the terminal device 1000 may switch the streaming data to be displayed on the remote monitoring screen in response to a user operation.

[0027] (welding power source) The welding power source 400 is a power source for supplying current and voltage related to the welding operation. In response to a command from the robot control device 600, the welding power source 400 supplies power to the welding wire 211, which is a consumable electrode, and the workpiece Wo, thereby generating an arc between the welding wire 211 and the workpiece Wo. The power from the welding power source 400 is sent to the wire feeder 300 via a power cable 410, and from the wire feeder 300 to the welding torch (hereinafter referred to as the "torch") 200 via a conduit tube 420. The power from the welding power source 400 is then supplied to the welding wire 211 via a contact tip at the tip of the torch 200. The current used during welding may be direct current or alternating current, and its waveform is not particularly important. Therefore, the current may be a pulse such as a square wave or a triangular wave.

[0028] Furthermore, welding power supply 400 has, for example, power cable 410 connected to torch 200 as a positive electrode, and power cable 430 connected to workpiece Wo as a negative electrode. Note that this is the case when welding is performed with reverse polarity; when welding with positive polarity, the power cable of the positive electrode is connected to workpiece Wo, and the power cable of the negative electrode is connected to torch 200.

[0029] (Shielding gas supply source) The shielding gas supply source 500 is composed of a container filled with shielding gas, a valve, and other auxiliary components. The shielding gas is sent from the shielding gas supply source 500 to the feeder 300 via a gas tube 510. The shielding gas sent to the feeder 300 is sent to the torch 200 via a conduit tube 420. The shielding gas sent to the torch 200 flows inside the torch 200, is guided by the nozzle 210, and is ejected from the tip side of the torch 200. For example, argon (Ar), carbon dioxide (CO2), or a mixture of these can be used as the shielding gas.

[0030] Conduit tube 420 has a conductive path formed on the outer skin side of the tube to function as a power cable, a protective tube for protecting welding wire 211 disposed inside the tube, and a flow path for shielding gas formed therein. However, conduit tube 420 is not limited to this, and for example, a power supply cable and a hose for supplying shielding gas can be bundled around a protective tube for feeding welding wire 211 to torch 200. Also, for example, the tubes for feeding welding wire 211 and shielding gas and the power cable can be installed separately.

[0031] (Feeding device) The feeder 300 pays out the welding wire 211 and feeds it to the torch 200. The welding wire 211 fed by the feeder 300 is not particularly limited and is selected depending on the properties of the workpiece Wo, the welding form, etc., and for example, a solid wire or a flux-cored wire is used. Furthermore, the diameter of the welding wire is not particularly limited.

[0032] Conduit tube 420 has a conductive path formed on the outer skin side of the tube to function as a power cable, a protective tube for protecting welding wire 211 disposed inside the tube, and a flow path for shielding gas formed therein. However, conduit tube 420 is not limited to this, and for example, a power supply cable and a hose for supplying shielding gas can be bundled around a protective tube for feeding welding wire 211 to torch 200. Also, for example, the tubes for feeding welding wire 211 and shielding gas and the power cable can be installed separately.

[0033] Furthermore, a touch sensor is used as the detection means, which senses the surface of the groove 10 by applying a voltage between the workpiece Wo and the welding wire 211 and utilizing the voltage drop phenomenon that occurs when the welding wire 211 comes into contact with the workpiece Wo. The detection means is not limited to a touch sensor, and an image sensor, a laser sensor, or a combination of these detection means may also be used, but it is preferable to use a touch sensor because of the simplicity of the device configuration.

[0034] (imaging device) The imaging device 700 (hereinafter also referred to as "camera") is configured, for example, by a camera equipped with a CMOS (Complementary Metal Oxide Semiconductor) as a visual sensor. The placement position of the imaging device 700 is not particularly limited, and the imaging device 700 may be attached directly to the welding robot 100, or may be fixed to a specific location in the vicinity as a surveillance camera. When the imaging device 700 is attached directly to the welding robot 100, the imaging device 700 moves in accordance with the operation of the welding robot 100 so as to capture images of the area around the tip of the torch 200. The imaging device 700 may include multiple cameras. For example, the imaging device 700 may be configured using multiple cameras with different functions and installation positions.

[0035] The imaging direction of the imaging device 700 is not particularly limited. For example, if the welding proceeds in the forward direction, the imaging device 700 may be positioned to capture images of the front side, the side side, or the rear side. Therefore, the imaging range of the imaging device 700 may be determined appropriately. To suppress interference with the torch 200, it is preferable to capture images from the front side. In this embodiment, imaging is performed from the front side. The captured image information is transmitted to the data processing device 800 and used by the data processing device 800. At this time, the data processing device 800 may capture any image from the captured image information, for example, at predetermined intervals, and use the image for processing, which will be described later. The capture method and capture settings here may be switched depending on, for example, the configuration and function of the imaging device 700 and the performance of the data processing device 800.

[0036] In this embodiment, imaging device 700 is directly attached and fixed to welding robot 100, and moving images are captured so that the imaging range includes at least workpiece Wo, welding wire 211, and the arc as objects (subjects) to be included in the image data. Note that various imaging settings may be specified in advance or may be switched depending on the operating conditions of welding system 50. Examples of imaging settings include frame rate, number of pixels of image, resolution, shutter speed, etc.

[0037] The data processing device 800 is an information processing device constituted by, for example, a computer. The computer has a processing unit 810 and a storage unit 820, which will be described later. The processing unit 810 is constituted by, for example, a CPU (Central Processing Unit). The storage unit 820 is constituted by, for example, a volatile or non-volatile memory such as a HDD (Hard Disk Drive), a ROM (Read Only Memory), or a RAM (Random Access Memory). The processing unit 810 executes computer programs for realizing various functions, which will be described later, stored in the storage unit 820, thereby transmitting various commands to the robot control device 600 and executing monitoring processing, which will be described later.

[0038] (Communication equipment) The communication device 900 includes an antenna (not shown), is connected to the data processing device 800 and the robot control device 600, and communicates with external devices via a network 1100. An example of the external device here is the terminal device 1000. The communication method and communication standard of the network 1100 are not particularly limited, and it is sufficient that the network 1100 is configured to enable data transmission and reception between the communication device 900 and the terminal device 1000.

[0039] (Terminal Device) The terminal device 1000 may be configured as, for example, a tablet terminal, a mobile PC, a POS terminal, a dedicated terminal, a smart watch, smart glasses, etc. The terminal device 1000 executes a monitoring process described below, receives instructions from a user, and provides various information to the user.

[0040] (Pre-trained model) In this embodiment, feature points extracted from image data and a trained model for extracting feature points will be described. The trained model used in this embodiment is configured by a convolutional neural network and includes multiple convolutional layers and multiple pooling layers. Note that the configuration of the convolutional neural network is not limited to the above, and other configurations and numbers of layers may be used. Therefore, a known method may be used for the training process, and a trained model that can be used in monitoring, which will be described later, may be used as appropriate.

[0041] In this embodiment, "learning" or "machine learning" refers to generating a "trained model" by performing learning using training data and an arbitrary learning algorithm. The trained model is updated as needed as learning progresses using multiple pieces of training data, and its output changes even when the input is the same. Therefore, the state of the trained model is not limited to a specific point in time. Herein, a model used in learning is referred to as a "learning model," and a learning model that has undergone a certain level of learning is referred to as a "trained model." The composition of "training data" may change depending on the learning algorithm used. Training data may include training data used for the training itself, verification data used to verify the trained model, and test data used to test the trained model. In the following description, the term "training data" is used to collectively refer to data related to training, and the term "training data" is used to refer to data used when performing the training itself. Note that this is not intended to clearly classify training data, verification data, and test data contained in the training data. For example, depending on the training, verification, and testing methods, all of the training data may also be training data.

[0042] The trained model receives image data output from the imaging device 700 as input and outputs feature quantities related to various welding information that appear in the image data. In this embodiment, the image data input to the trained model includes at least the molten pool, welding wire, and arc as objects (targets), and feature points obtained from each of these objects or multiple objects are extracted as coordinate data. Based on the coordinate data of the extracted feature points, the data processing device 800 then calculates the amount of tracking correction and correction amounts for the swing width, welding speed, and rod operation trajectory (angle), and transmits the obtained information on the correction amounts to the robot control device 600. Note that this image data will hereinafter also be referred to as a welding image.

[0043] The trained model is assumed to have already been generated and made available before the remote monitoring function according to this embodiment is executed. The trained model is updated as needed by the data processing device 800 or an external learning device, and is referred to appropriately when the data processing device 800 uses it.

[0044] In this embodiment, a trained model capable of extracting coordinate data of feature points will be described as an example, but the present invention is not limited to this. For example, a trained model that detects a predetermined area in a welding image, such as the range of a molten pool or a torch, may be used. In this embodiment, feature amounts in image data captured by the imaging device 700 are identified, and the image data and information based on the feature amounts are combined to generate streaming data.

[0045] [Function Configuration] Fig. 2 is a block diagram showing the system configuration shown in Fig. 1, focusing on the configuration related to the remote monitoring function according to this embodiment. Note that the configuration shown in Fig. 2 is also one example, and other components and devices may be included. Furthermore, for each component shown in Fig. 2, one component may be divided into multiple components, or multiple components may be integrated into one component.

[0046] The data processing device 800 includes a processing unit 810, a storage unit 820, a display unit 830, and an external interface 840. A touch panel display, an organic EL (Electro-Luminescence) display, or the like may be used as the display unit 830. The external interface 840 is a component that controls communication with external devices, and is communicably connected to, for example, the robot control device 600, the imaging device 700, the communication device 900, and the like.

[0047] The processing unit 810 of the data processing device 800 provides the functions of an information acquisition unit 811, a feature derivation unit 812, an abnormality detection unit 813, an information generation unit 814, a display control unit 815, a robot command unit 816, and a streaming distribution unit 817. The functions associated with each of these units may be realized by the processing unit 810 reading out and executing programs and data stored in the storage unit 820.

[0048] The information acquisition unit 811 acquires various pieces of information from devices such as the robot control device 600, the imaging device 700, and the communication device 900. The acquired information may include image data, command information, and detection data. The feature derivation unit 812 extracts features (in this example, coordinate data of feature points) from the image data by applying the trained model described above to the image data. The feature derivation unit 812 may update the available trained model as needed. The trained model may be configured to be acquired as needed from a learning device (not shown) that performs a learning process, or the data processing device 800 may update the trained model by executing the learning process itself.

[0049] The abnormality detection unit 813 detects abnormalities related to welding, for example, based on the feature amounts derived by the feature amount derivation unit 812 and detection data detected by sensors (not shown). When the abnormality detection unit 813 detects an abnormality, it executes a corresponding process based on the detected abnormality. Specifically, the abnormality detection unit 813 displays the detected abnormality on a UI (User Interface) screen (described later) and notifies various components of the abnormality.

[0050] The information generation unit 814 generates streaming data and information to be displayed on the terminal device 1000 or the display unit 830, using the image data captured by the imaging device 700, the feature amounts derived by the feature amount derivation unit 812, and the abnormality information detected by the abnormality detection unit 813. For example, the information generation unit 814 generates composite image data by combining the image data captured by the imaging device 700 with feature points derived from the image data. The information generation unit 814 may also generate composite image data by combining the image data captured by the imaging device 700 with information calculated by the control signal calculation unit 606 of the robot control device 600, using the coordinate data of the feature points derived from the image data.

[0051] Display control unit 815 displays, on display unit 830, the information generated by information generation unit 814, image data captured by imaging device 700, and the like. Robot command unit 816 issues an operation instruction to welding robot 100 or an instruction to welding power source 400 via robot control device 600 based on an instruction from terminal device 1000. Robot command unit 816 may also generate and provide a command to robot control device 600 based on the feature derived by feature derivation unit 812, abnormality information detected by abnormality detection unit 813, an instruction received from terminal device 1000, and the like. Robot command unit 816 may also transmit the control result of welding robot 100 by robot control device 600 to terminal device 1000 via communication device 900.

[0052] The streaming distribution unit 817 generates and distributes streaming data based on the image data generated by the information generation unit 814. Note that, in response to a request from the terminal device 1000 or the like, the image data captured by the imaging device 700 may be distributed as streaming data as is. The streaming distribution unit 817 also receives streaming distribution settings from the operator of the data processing device 800 or the terminal device 1000, and controls distribution based on these settings. Examples of the settings include data size and frame rate. By using streaming distribution, storage usage on the terminal device 1000 side can be reduced compared to data downloading. Furthermore, real-time video playback on the terminal device 1000 side can be ensured.

[0053] In robot control device 600, communication control unit 605 controls, for example, communication with data processing device 800 and communication with terminal device 1000 via communication device 900. Control signal calculation unit 606 acquires command information from data processing device 800 or terminal device 1000 via communication control unit 605 and generates a control signal for welding robot 100 or welding power source 400 based on the command information. Control signal calculation unit 606 then outputs the generated control signal to at least either welding robot 100 or welding power source 400. Control signal calculation unit 606 also acquires the control results of welding robot 100 and feedback signals from welding robot 100, and provides them to data processing device 800 and terminal device 1000.

[0054] The remote monitoring application 1001 of the terminal device 1000 receives information from the data processing device 800 and the robot control device 600 and displays a UI screen (described later). Examples of information received by the remote monitoring application 1001 include streaming data and welding-related information such as current values ​​and voltages. The remote monitoring application 1001 arranges and displays the acquired information as display items on the UI screen. Furthermore, the remote monitoring application 1001 receives instructions from a user via the UI screen and transmits control instructions based on the instructions to the data processing device 800 and the robot control device 600. The remote monitoring application 1001 may be implemented by a processing unit (not shown) included in the terminal device 1000 reading and executing programs and various data stored in a storage unit (not shown).

[0055] 2 is an example and is not limiting. For example, the terminal device 1000 may be configured to send and receive data directly to and from the robot control device 600 without going through the data processing device 800. The content of the data sent and received by the robot control device 600, the data processing device 800, and the terminal device 1000 may be specified in advance. For example, of the information to be displayed on the terminal device 1000 side, the robot control device 600 may directly send information detected by the robot control device 600 to the terminal device 1000. There may also be a type of data in which instructions inputted into the terminal device 1000 are directly sent to the robot control device 600.

[0056] [Processing flow] (Remote monitoring processing in data processing device) 3 is a flowchart of a remote monitoring process by data processing device 800 according to this embodiment. This process flow may be performed by having processing unit 810 of data processing device 800 read and execute a program stored in storage unit 820, thereby realizing the functions shown in FIG. 2. For ease of explanation, the processing entity of this process flow will be collectively described as data processing device 800. Data processing device 800 is also capable of transmitting and receiving data in cooperation with various devices such as welding system 50 and terminal device 1000.

[0057] In S301, the data processing device 800 causes the imaging device 700 to start capturing images of the workpiece Wo and the area around the torch 200. The imaging parameters at the start of imaging by the imaging device 700 may be specified in advance. Furthermore, if multiple imaging devices 700 are provided, each of the imaging devices 700 is caused to start capturing images. The captured image data may be still images captured at a predetermined time interval, or may be moving images captured at a predetermined frame rate.

[0058] In S302, the robot control device 600 starts welding. During welding, the imaging device 700 sequentially captures images to obtain image data, which are then provided to the data processing device 800.

[0059] In S303, the data processing device 800 extracts feature points from the captured image data. In this embodiment, the feature points are extracted using a trained model obtained by the above-described learning process. Specifically, a trained model is used that receives a welding image as input and outputs coordinate data of feature points, such as a phenomenon of interest contained in the welding image. Examples of feature points that are of interest include the position of the tip of the molten pool, the width of the molten pool, the position of the tip of the welding torch, and the position of the arc center, and these feature points are output as coordinate data.

[0060] In S304, the data processing device 800 generates composite image data that is the source of streaming data by combining the captured image data with the coordinate positions of feature points obtained from the image. Note that the information used to generate the composite image data is not limited to the feature points obtained as the output of the trained model, and may include other information derived from the feature points. For example, an abnormality determination may be performed based on the feature points, and the result of the abnormality determination may be combined with the image data.

[0061] In S305, the data processing device 800 sequentially streams the streaming data (hereinafter also referred to as "video data") generated in S304. In this example, the destination of the streaming data is the terminal device 1000. Note that various welding-related parameters and detection results from sensors may be transmitted to the terminal device 1000 from the data processing device 800, the robot control device 600, the welding power source 400, etc., simultaneously with the streaming transmission of the video data. Examples of the detection results include the welding current and the arc voltage. These detection results are transmitted to the terminal device 1000 and displayed on the terminal device 1000. In this embodiment, the robot control device 600 acquires detection values ​​of the welding current and the arc voltage from a detector (not shown) as needed, and transmits the detection values ​​acquired from the robot control device 600 to the terminal device 1000 and displays them on the terminal device 1000. The same display content as that of the terminal device 1000 may also be displayed on the display unit 830 of the data processing device 800.

[0062] In S306, data processing device 800 determines whether an instruction has been received via terminal device 1000. If an instruction has been received (YES in S306), the processing of data processing device 800 proceeds to S307. On the other hand, if an instruction has not been received (NO in S306), the processing of data processing device 800 proceeds to S312. In this embodiment, the instruction received via terminal device 1000 is described as either an instruction relating to the control of welding robot control device 600 or an instruction relating to changing the settings of data processing device 800 and connected devices in remote monitoring. Examples of instructions sent to welding robot control device 600 include an instruction relating to the control of welding robot 100 and an instruction relating to the control of the welding power source.

[0063] In S307, data processing device 800 determines whether the received instruction is an instruction related to the control of welding robot control device 600. If the instruction is related to the control of welding robot control device 600 (YES in S307), the processing of data processing device 800 proceeds to S308. On the other hand, if the instruction is not related to the control of welding robot control device 600 (NO in S307), the processing of data processing device 800 proceeds to S310.

[0064] In S308, data processing device 800 generates a control command for welding robot control device 600 based on the received instruction.

[0065] In S309, data processing device 800 transmits the control command generated in S308 to robot control device 600. Robot control device 600 controls welding robot 100 or welding power source 400 based on the control command from data processing device 800. Thereafter, the processing of data processing device 800 proceeds to S312.

[0066] In S310, data processing device 800 changes the settings of data processing device 800 and the connected devices based on the received instruction. Examples of the connected devices include robot control device 600, imaging device 700, and welding power source 400. The setting change here may be, for example, a change in the imaging conditions of imaging device 700 or parameters of streaming data, or the addition of information to be displayed on a UI screen (described later). If further feature point extraction is required following the setting change, processing similar to that in step S303 may be executed.

[0067] In S311, data processing device 800 transmits the data after the setting change in S310 to terminal device 1000. At this time, the number of data items to be transmitted may increase or decrease depending on the change content in S310. Then, the processing of data processing device 800 proceeds to S312.

[0068] In S312, data processing device 800 determines whether or not to end remote monitoring. The end of remote monitoring may be performed based on an instruction from terminal device 1000, or based on an instruction from an administrator of data processing device 800. Alternatively, remote monitoring may be ended when it is detected that the welding started in S302 has been completed. Note that even when remote monitoring has ended, welding itself may be allowed to continue. If remote monitoring is to be ended (YES in S312), this processing flow ends. If remote monitoring is not to be ended (NO in S312), the processing of data processing device 800 returns to S303, and the processing is repeated.

[0069] (Remote monitoring processing in terminal device) 4 is a flowchart of a remote monitoring process by the terminal device 1000 according to this embodiment. This process flow may be implemented by a processing unit (not shown) of the terminal device 1000 reading and executing a remote monitoring application 1001 stored in a storage unit (not shown). For ease of explanation, the processing entity of this process flow will be collectively described as the terminal device 1000. Furthermore, the terminal device 1000 can send and receive data to and from the welding system 50 via the network 1100.

[0070] In S401, terminal device 1000 launches remote monitoring application 1001 and starts remote monitoring of welding system 50. At this point, it is assumed that the remote monitoring process on the data processing device 800 side shown in FIG. 3 has started.

[0071] In S402, the terminal device 1000 starts acquiring streaming data being streamed by the data processing device 800. It may also start acquiring any data other than the streaming data from the connected device. In this embodiment, in addition to the streaming data, data such as current, voltage, and welding speed are acquired from the robot control device 600. These data may be acquired in temporal association with the streaming data.

[0072] In S403, the terminal device 1000 displays the acquired streaming data on a display unit (not shown). Display examples will be described later. Note that, as shown in FIG. 6 (described later), any acquired data may be displayed on a display unit (not shown). In this embodiment, data such as current, voltage value, and welding speed is displayed. In this embodiment, the remote monitoring application 1001 of the terminal device 1000 holds layout information constituting a UI screen, and appropriately arranges and displays data such as streaming data and current values ​​acquired from the data processing device 800, etc. Note that the layout information may be arbitrarily adjustable by the user of the terminal device 1000.

[0073] In S404, the terminal device 1000 determines whether or not a display-related instruction has been received from the user. Examples of display-related instructions include screen switching and adding or removing display items. Such instructions may be given by, for example, tapping, double-tapping, long pressing, pinching, flicking, or other operations on a touch panel display (not shown) provided in the terminal device 1000. If a display-related instruction has been received (YES in S404), the processing of the terminal device 1000 proceeds to S405. If a display-related instruction has not been received (NO in S404), the processing of the terminal device 1000 proceeds to S408.

[0074] In S405, the terminal device 1000 determines whether or not a setting change of the data processing device 800 and the connected devices is necessary based on the instruction related to display. For example, a setting change of the data processing device 800 and the connected devices may be necessary when additional information to be displayed needs to be acquired in response to the instruction related to display. If a setting change of the data processing device 800 and the connected devices is necessary (YES in S405), the processing of the terminal device 1000 proceeds to S407. On the other hand, if a setting change of the data processing device 800 and the connected devices is not necessary (NO in S405), the processing of the terminal device 1000 proceeds to S406.

[0075] In S406, the terminal device 1000 switches the display based on the display-related instruction. An example of the switching will be described later. Then, the processing of the terminal device 1000 proceeds to S408.

[0076] In S407, the terminal device 1000 requests the data processing device 800 to change the settings of the data processing device 800 and the connected devices based on the display-related instruction. In response to the request, the terminal device 1000 acquires data of the data processing device 800 and the connected devices after the setting change and switches the display. Thereafter, the processing of the terminal device 1000 proceeds to S408.

[0077] In S408, terminal device 1000 determines whether an instruction related to the control of welding robot control device 600 has been received from the user. Examples of instructions related to the control of welding robot control device 600 include changing the welding current, adjusting the scanning position, and changing the welding parameter set. Examples of instructions will be described later along with the UI screen. If an instruction related to the control of welding robot control device 600 has been received (YES in S408), the processing of terminal device 1000 proceeds to S409. If an instruction related to the control of welding robot control device 600 has not been received (NO in S408), the processing of terminal device 1000 proceeds to S410.

[0078] In S409, terminal device 1000 generates a control instruction based on the instruction related to the control of welding robot control device 600, and transmits the control instruction to data processing device 800. Then, the processing of terminal device 1000 proceeds to S410.

[0079] In S410, the terminal device 1000 determines whether or not to end the remote monitoring. The end of the remote monitoring may be based on an instruction from the data processing device 800, or on an instruction from the user of the terminal device 1000. If the remote monitoring is to be ended (YES in S410), this processing flow ends. If the remote monitoring is not to be ended (NO in S410), the processing of the terminal device 1000 returns to S403, and the processing is repeated.

[0080] [Example of remote monitoring screen] 5 and 6A to 6F, a description will be given of an example of the configuration of a remote monitoring screen as a UI screen displayed on the terminal device 1000. Each remote monitoring screen is provided by the remote monitoring application 1001, which collects various information and provides it via a display unit (not shown) of the terminal device 1000.

[0081] FIG. 5 is a diagram illustrating switching of remote monitoring screens on the terminal device 1000. On the remote monitoring screen 520, streaming data during welding, the set and measured values ​​of the welding current, and the set and measured values ​​of the welding voltage are displayed in real time. By performing a flick operation 521 on the remote monitoring screen 520, the screen is switched to a remote monitoring screen 530. The remote monitoring screen 530 shows the welding voltage and welding current in graphs. By performing a flick operation 531 on the remote monitoring screen 530, the screen can be switched to the remote monitoring screen 520. Note that, although flicking is used as an example of a switching operation, the present invention is not limited to this, and other operations may also be used. Furthermore, the displayed items may be switched depending on the attitude or orientation of the terminal device 1000.

[0082] The remote monitoring screen 640 shown in FIG. 6A includes streaming data 641 and a welding speed setting item 642. The setting item 642 includes a current setting value 643, a rotary dial 644 for specifying the setting value, an increase / decrease button 645, and a confirm button 646. On the remote monitoring screen 640, the welding speed can be specified within a range from a lower limit of 1.0 to an upper limit of 100.0 cpm by tapping the increase (+) and decrease (-) buttons 645. In addition to the increase / decrease button 645, control parameters can also be specified by scrolling the rotary dial 644. After specifying the control parameters, welding-related operations are performed by selecting the confirm button 646.

[0083] 6B includes streaming data 651 and a setting item 652 for setting the left and right scanning positions of torch 200. Setting item 652 includes a current setting value 653, a scroll bar 654 for specifying the setting value, and an OK button 655. On remote monitoring screen 650, the left and right scanning positions of torch 200 can be specified by operating scroll bar 654 within a range of -300 to 300 mm relative to the welding direction, i.e., the traveling direction of torch 200. Scroll bar 654 can be operated by sliding. After specifying the control parameters, welding-related operations are performed by selecting OK button 655.

[0084] 6C includes streaming data 661 and a setting item 662 for the left and right scanning positions of torch 200. Setting item 662 includes a current setting value 663 and a designation button 664 for designating the setting value. On remote monitoring screen 660, the left and right scanning positions of torch 200 can be designated by operating designation button 664 within a range of -300 to 300 mm relative to the welding direction, i.e., the traveling direction of torch 200. Designation button 664 can be operated by tapping.

[0085] Remote monitoring screen 670 shown in FIG. 6D includes streaming data 671 and setting items 672 for selecting a parameter set that is preset for the welding operation of welding robot 100. Setting items 672 include a current setting value 673, a drum roll 674 for specifying the setting value, and an OK button 675. On remote monitoring screen 670, a desired parameter set can be specified by selecting it from a list of multiple predefined parameter sets. The list is displayed by a drum roll 674 and can be rotated by swiping. After specifying the parameter set, welding-related operations are performed by selecting OK button 675.

[0086] The remote monitoring screen 680 shown in FIG. 6E includes streaming data 681 and control parameters 682, including set values ​​and actual measurement values ​​related to welding. In this example, the welding control parameters include the welding current and welding voltage. The streaming data 681 also displays information such as the width of the molten pool identified from the feature quantities, superimposed on the image data. The streaming data 681 in this example combines the positions of both ends of the molten pool tip (circles in the streaming data 681), the molten pool width (dotted line in the streaming data 681), and the wire tip position (cross mark in the streaming data 681) as feature points. The width W and length L of the molten pool, derived from the coordinate data of the feature points, are also displayed after being combined with the image data. In this embodiment, the width W and length L of the molten pool are calculated by the control signal calculation unit 606 based on the coordinate data of the feature points.

[0087] The remote monitoring screen 690 shown in Fig. 6F has the same configuration as Fig. 6E and shows the state when an abnormality is detected. In this example, when an abnormality (error number: ENo.X) is detected in streaming data 691, a message "ENo.X XX error occurring" is synthesized and displayed in response to this abnormality. Similar to the screen configuration in Fig. 6E, the control parameters 692 include set values ​​and actual measurement values ​​related to welding.

[0088] The user can grasp the welding status in real time and easily perform operations according to the status by using the remote monitoring screens shown in FIGS. 6A to 6F displayed on the terminal device 1000. Note that the configuration of the remote monitoring screen and the items displayed are merely examples and are not limited to these. For example, the remote monitoring screen may be configured so that the user can customize which setting items to display among the setting items shown in FIGS. 6A to 6F. Furthermore, the configuration is not limited to displaying only one setting item on one remote monitoring screen, and multiple setting items may be displayed on one remote monitoring screen.

[0089] Furthermore, the operation items for various operations displayed on the remote monitoring screen are not limited to those described above. For example, operation items of various types, such as a cross key or character input, may be included. These may be provided in consideration of operation using a touch panel display provided on the terminal device 1000, or may be configured assuming a physical user interface (for example, buttons, an operation bar, etc.) provided on the terminal device 1000.

[0090] As described above, this embodiment can suppress delays in acquiring and presenting monitoring information on a terminal device, and enables easy remote operation according to the status of the monitored object. In particular, in a welding system, it is possible to improve the real-time nature of monitoring and easily instruct welding-related control remotely. Therefore, it is possible to improve the convenience for users of the welding system.

[0091] <Other embodiments> In the present invention, a program or application for realizing the functions of one or more of the above-described embodiments can be supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device can read and execute the program.

[0092] Alternatively, the function may be realized by a circuit that realizes one or more functions, such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0093] As described above, the present specification discloses the following: (1) A robot (e.g., 100) and An imaging device (e.g., 700) that captures an image of the operating area of ​​the robot; A power supply (e.g., 400) that controls the output of a current or voltage; a data processing device (e.g., 800) for processing data related to the operation of the robot or the power source; a terminal device (e.g., 1000) configured to be able to communicate with the data processing device via a network (e.g., 1100); A remote monitoring system comprising: The data processing device includes: A generating unit (e.g., 814) that generates streaming data using image data captured by the imaging device; a distribution unit (e.g., 817) that distributes the streaming data; a robot command unit (e.g., 816) that issues an operation instruction to the robot or an operation instruction to the power source based on at least an instruction from the terminal device; and The terminal device a display control unit (e.g., 1001) that receives streaming data distributed from the data processing device and displays it on a monitoring screen (e.g., 640); an instruction receiving unit (e.g., 1001) that receives instructions for the robot or the power source on the monitoring screen and transmits the instructions to the data processing device; A remote monitoring system comprising: This configuration can reduce delays in the acquisition and presentation of monitoring information by the terminal device, and enables easy remote control according to the status of the monitored object.

[0094] (2) The data processing device further includes a derivation unit that derives a feature amount in the image data, The remote monitoring system according to (1), wherein the generating unit generates the streaming data using the image data and a feature amount in the image data. According to this configuration, it is possible to remotely monitor the state of the robot based on the feature amount obtained from the image data.

[0095] (3) The remote monitoring system described in (2) is characterized in that the derivation unit derives features of image data captured by the imaging device using a trained model generated by a learning process that takes image data as input and outputs features contained in the image data. With this configuration, features are derived from image data using a trained model, and based on this, it is possible to remotely monitor the status of the robot.

[0096] (4) The data processing device further includes a detection unit (e.g., 813) that detects an abnormality in the operation of the robot using the feature derived by the derivation unit, The remote monitoring system according to (2) or (3), wherein the generating unit generates the streaming data by further using information on an abnormality detected by the detecting unit. According to this configuration, abnormalities in the robot can be detected based on the feature values ​​obtained from the image data, and the detection results can be easily referenced remotely.

[0097] (5) The remote monitoring system described in (4) is characterized in that the robot command unit issues operation instructions to the robot or the power source based on information on an abnormality detected by the detection unit. According to this configuration, it becomes possible to easily control the robot and the power supply based on an abnormality in the robot identified based on the feature amount obtained from the image data.

[0098] (6) A remote monitoring system according to any one of (1) to (5), characterized in that the monitoring screen includes at least one of a display item (e.g., 641) for displaying the streaming data and an operation item (e.g., 642) for receiving operations on the robot or the power source. According to this configuration, it is possible to remotely instruct the operation of the robot or power supply while referring to the streaming data delivered in real time.

[0099] (7) The remote monitoring system according to any one of (1) to (7), wherein the monitoring screen is configured so that display items can be switched by user operation. This configuration enables users to easily monitor and instruct the robot and power supply remotely while switching between any display items on the monitoring screen.

[0100] (8) The remote monitoring system described in (6) is characterized in that the streaming data displayed in the display item includes any of image data captured by the imaging device, composite image data in which image data and features of the image data are combined, error information, and operation information of the robot. According to this configuration, the user can view streaming data made up of various image data on the monitoring screen.

[0101] (9) The remote monitoring system according to (6), wherein the operation items include any one of a cross key, a button, a drum roll, and a scroll bar. According to this configuration, the monitoring screen can be configured to include operation items for improving user operability.

[0102] (10) The robot is a welding robot (e.g., 100), The remote monitoring system according to any one of (1) to (9), wherein the power source is a welding power source. This configuration enables the welding system to easily remotely instruct welding control while improving the real-time monitoring performance, thereby improving the user convenience of the welding system.

[0103] (11) A robot (e.g., 100) and An imaging device (e.g., 700) that captures an image of the operating area of ​​the robot; A power supply (e.g., 400) that controls the output of a current or voltage; a data processing device (e.g., 800) for processing data related to the operation of the robot or the power source; a terminal device (e.g., 1000) configured to be able to communicate with the data processing device via a network (e.g., 1100); A method for remotely monitoring a system, comprising: In the data processing device, a generating step (e.g., S304) of generating streaming data using image data captured by the imaging device; a distribution step (e.g., S305) of distributing the streaming data; a robot command process (e.g., S308, S309) of issuing an operation command to the robot or an operation command to a power source based on at least an instruction from the terminal device; and In the terminal device, a display control step (e.g., S402, S403) of receiving streaming data distributed from the data processing device and displaying it on a monitoring screen; an instruction receiving step (e.g., S408, S409) of receiving an instruction to the robot or the power source on the monitoring screen and transmitting the instruction to the data processing device; A remote monitoring method comprising: This configuration can reduce delays in the acquisition and presentation of monitoring information by the terminal device, and enables easy remote control according to the status of the monitored object.

[0104] (12) A data processing device (e.g., 800) that processes data related to the operation of a robot (e.g., 100) or a power supply (e.g., 400), a generating unit (e.g., 814) that generates streaming data using image data of the robot's operating area captured by an imaging device (e.g., 700); a distribution unit (e.g., 817) that distributes the streaming data to a terminal device (e.g., 1000) via a network (e.g., 1100); a robot command unit (e.g., 816) that issues at least one of an operation instruction to the robot or an operation instruction to the power source based on an instruction received from the terminal device via at least the network; A data processing device comprising: This configuration reduces delays in the acquisition and presentation of monitoring information on the terminal device and enables easy remote control according to the status of the monitored object, thereby improving user convenience in remote monitoring.

[0105] (13) A terminal device (e.g., 1000) configured to be able to communicate with a data processing device (e.g., 800) or a power source (e.g., 400) that processes data related to the operation of a robot (e.g., 100) via a network (e.g., 1100), a display control unit (e.g., 1001) that receives streaming data distributed from the data processing device and displays it on a monitoring screen (e.g., 640); an instruction receiving unit (e.g., 1001) that receives instructions for the robot or the power source on the monitoring screen and transmits the instructions to the data processing device; A terminal device comprising: This configuration reduces delays in the acquisition and presentation of monitoring information on the terminal device and enables easy remote control according to the status of the monitored object, thereby improving user convenience in remote monitoring.

[0106] (14) A data processing method for processing data related to the operation of a robot (e.g., 100), comprising: a generating step (e.g., S304) of generating streaming data using image data of the robot's operating area captured by an imaging device; a distribution step (e.g., S305) of distributing the streaming data to a terminal device (e.g., 1000) via a network (e.g., 1100); a robot command process (e.g., S308, S309) of issuing at least one of an operation command to the robot or an operation command to a power source based on an instruction from the terminal device received at least via the network; A data processing method comprising: This configuration reduces delays in the acquisition and presentation of monitoring information on the terminal device and enables easy remote control according to the status of the monitored object, thereby improving user convenience in remote monitoring.

[0107] (15) A control method for a terminal device (e.g., 1000) configured to be able to communicate with a data processing device (e.g., 800) or a power source (e.g., 400) that processes data related to the operation of a robot (e.g., 100) via a network (e.g., 1100), comprising: a display control step (e.g., S402, S403) of receiving streaming data distributed from the data processing device and displaying it on a monitoring screen; an instruction receiving step (e.g., S408, S409) of receiving an instruction to the robot or the power source on the monitoring screen and transmitting the instruction to the data processing device; A method for controlling a terminal device, comprising: This configuration reduces delays in the acquisition and presentation of monitoring information on the terminal device and enables easy remote control according to the status of the monitored object, thereby improving user convenience in remote monitoring.

[0108] (16) To a computer (e.g., 800), a generating unit (e.g., 814) that generates streaming data using image data of the operating area of ​​the robot (e.g., 100) captured by an imaging device (e.g., 700); a distribution unit (e.g., 817) that distributes the streaming data to a terminal device (e.g., 1100) via a network (e.g., 1100); a robot command unit (e.g., 816) that issues at least one of an operation instruction to the robot or an operation instruction to a power source (e.g., 400) based on at least an instruction from the terminal device received via the network; A program to execute. This configuration reduces delays in the acquisition and presentation of monitoring information on the terminal device and enables easy remote control according to the status of the monitored object, thereby improving user convenience in remote monitoring.

[0109] (17) A computer (e.g., 1000) configured to be able to communicate with a data processing device (e.g., 800) or a power supply (e.g., 400) that processes data related to the operation of a robot (e.g., 100) via a network (e.g., 1100), a display control unit (e.g., 1001) that receives streaming data distributed from the data processing device and displays it on a monitoring screen (e.g., 640); an instruction receiving unit (e.g., 1001) that receives instructions for the robot or the power source on the monitoring screen and transmits the instructions to the data processing device; A program to execute. This configuration reduces delays in the acquisition and presentation of monitoring information on the terminal device and enables easy remote control according to the status of the monitored object, thereby improving user convenience in remote monitoring. [Explanation of symbols]

[0110] 100...Welding robot 600...Robot control device 605...Communication control unit 606...Control signal calculation unit 700...imaging device 800...Data processing device 810... Processing section 811…Information acquisition department 812...Feature extraction unit 813...Abnormality detection unit 814...Information generation section 815...Display control unit 816...Robot Command Center 817…Streaming Distribution Department 820...Storage section 830...Display section 840...External interface 900...Communication equipment 1000...Terminal device 1001...Remote monitoring application 1100…Network

Claims

1. Robots and an imaging device that captures an image of the operating area of ​​the robot; a power supply that controls the output of a current or a voltage; a data processing device that processes data related to the operation of the robot or the power source; a terminal device configured to be able to communicate with the data processing device via a network; A remote monitoring system comprising: The data processing device includes: a generating unit that generates streaming data using image data captured by the imaging device; a distribution unit that distributes the streaming data; a robot command unit that issues at least one of an operation instruction to the robot and an operation instruction to the power source based on at least an instruction from the terminal device; and The terminal device a display control unit that receives streaming data distributed from the data processing device and displays it on a monitoring screen; an instruction receiving unit that receives an instruction to the robot or the power source on the monitoring screen and transmits the instruction to the data processing device; A remote monitoring system comprising:

2. the data processing device further includes a derivation unit that derives a feature amount in the image data, 2. The remote monitoring system according to claim 1, wherein the generating unit generates the streaming data using the image data and a feature amount in the image data.

3. The remote monitoring system according to claim 2, characterized in that the derivation unit derives features of the image data captured by the imaging device using a trained model generated by a learning process that takes image data as input and outputs features contained in the image data.

4. the data processing device further includes a detection unit that detects an abnormality in an operation of the robot using the feature amount derived by the derivation unit, 3. The remote monitoring system according to claim 2, wherein the generating unit generates the streaming data by further using information on an abnormality detected by the detecting unit.

5. 5. The remote monitoring system according to claim 4, wherein the robot command unit issues an operation instruction to the robot or the power source based on information about an abnormality detected by the detection unit.

6. 2. The remote monitoring system according to claim 1, wherein the monitoring screen includes at least one of display items for displaying the streaming data and operation items for receiving operations on the robot or the power source.

7. 2. The remote monitoring system according to claim 1, wherein the monitoring screen is configured so that display items can be switched by a user operation.

8. 7. The remote monitoring system according to claim 6, wherein the streaming data displayed in the display items includes any one of image data captured by the imaging device, composite image data in which image data and features of the image data are combined, error information, and operation information of the robot.

9. 7. The remote monitoring system according to claim 6, wherein the operation items include any one of a cross key, a button, a drum roll, and a scroll bar.

10. the robot is a welding robot, 2. The remote monitoring system according to claim 1, wherein the power source is a welding power source.

11. Robots and an imaging device that captures an image of the operating area of ​​the robot; a power supply that controls the output of a current or a voltage; a data processing device that processes data related to the operation of the robot or the power source; a terminal device configured to be able to communicate with the data processing device via a network; A method for remotely monitoring a system, comprising: In the data processing device, a generating step of generating streaming data using image data captured by the imaging device; a distribution step of distributing the streaming data; a robot command step of issuing at least one of an operation instruction to the robot or an operation instruction to the power source based on at least an instruction from the terminal device; and In the terminal device, a display control step of receiving streaming data distributed from the data processing device and displaying the data on a monitoring screen; an instruction receiving step of receiving an instruction for the robot or the power source on the monitoring screen and transmitting the instruction to the data processing device; A remote monitoring method comprising:

12. A data processing device that processes data related to the operation of a robot or a power source, a generating unit that generates streaming data using image data of the robot's motion area captured by an imaging device; a distribution unit that distributes the streaming data to a terminal device via a network; a robot command unit that issues at least one of an operation instruction to the robot or an operation instruction to the power source based on an instruction from the terminal device received at least via the network; A data processing device comprising:

13. A terminal device configured to be able to communicate with a data processing device that processes data related to the operation of a robot or a power source via a network, a display control unit that receives streaming data distributed from the data processing device and displays it on a monitoring screen; an instruction receiving unit that receives an instruction to the robot or the power source on the monitoring screen and transmits the instruction to the data processing device; A terminal device comprising:

14. A data processing method for processing data related to the operation of a robot, comprising: a generation step of generating streaming data using image data of the robot's motion area captured by an imaging device; a distribution step of distributing the streaming data to a terminal device via a network; a robot command step of issuing at least one of an operation instruction to the robot and an operation instruction to a power source based on an instruction from the terminal device received at least via the network; A data processing method comprising:

15. A control method for a terminal device configured to be able to communicate with a data processing device that processes data related to the operation of a robot or a power source via a network, comprising: a display control step of receiving streaming data distributed from the data processing device and displaying the data on a monitoring screen; an instruction receiving step of receiving an instruction for the robot or the power source on the monitoring screen and transmitting the instruction to the data processing device; A method for controlling a terminal device, comprising:

16. On the computer, a generating unit that generates streaming data using image data of the robot's motion area captured by an imaging device; a distribution unit that distributes the streaming data to a terminal device via a network; a robot command unit that issues at least one of an operation instruction to the robot and an operation instruction to a power source based on an instruction from the terminal device received at least via the network; A program to execute.

17. a computer configured to be able to communicate with a data processing device or a power source via a network, which processes data related to the operation of the robot; a display control unit that receives streaming data distributed from the data processing device and displays it on a monitoring screen; an instruction receiving unit that receives an instruction to the robot or the power source on the monitoring screen and transmits the instruction to the data processing device; A program to execute.

Images