Information Processing Systems

Abstract

To shorten the time required for cause investigations and recovery work upon occurrence of a problem.SOLUTION: An information processing system comprises: a signal output section that outputs an external signal; an imaging section that acquires the external signal from the signal output section and synchronizes and outputs a photographic image generated by taking an image of an object with the external signal; and a creating section that acquires the photographic image and the external signal from the imaging section, adds information about the external signal to the photographic image synchronized with the external signal, and creates a moving image having a plurality of photographic images continuing in time series based on the photographic image added with the information about the external signal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system. [Background technology]

[0002] In manufacturing sites, equipment outages directly lead to losses, so it is important to quickly identify the cause and quickly restore equipment after an outage. However, because it is not known when an equipment outage will occur, cameras are sometimes used to constantly monitor the equipment, but constant monitoring is not very practical in terms of resources (time, storage capacity).

[0003] A technology is known that makes it easier to confirm events by combining event data detected by a camera with video feeds captured by other cameras (see Patent Document 1). Factory drive recorders have also been developed as tools to assist verification at manufacturing sites. Compared to general drive recorders, factory drive recorders are capable of recording at higher resolutions and frame rates, and are capable of spot recording triggered by changes or input from external signals. Furthermore, because factory drive recorders allow cameras and other equipment to be retrofitted to equipment, they enable monitoring of problems that are difficult to anticipate during the design stage of the production line, making them suitable for verification at manufacturing sites. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-98778 Summary of the Invention [Problem to be solved by the invention]

[0005] Factory drive recorders can record video of the manufacturing site using signals from external sensors connected to the camera as triggers, so by reviewing the video, it is possible to understand the work situation in the event of a problem. However, if the status of the equipment at the manufacturing site cannot be determined even after reviewing the video, it will take time to identify the cause of the problem and carry out recovery work.

[0006] The present invention has been made in view of the above-mentioned circumstances, and an object of the present invention is to provide a technique that can shorten the time required to investigate the cause of a problem and to carry out recovery work when a problem occurs. [Means for solving the problem]

[0007] An information processing system according to one aspect of the present invention is an information processing system comprising: a signal output unit that outputs an external signal; an imaging unit that acquires the external signal from the signal output unit and outputs an imaged image generated by imaging an object in synchronization with the external signal; and a creation unit that acquires the imaged image and the external signal from the imaging unit, adds information related to the external signal to the imaged image synchronized with the external signal, and creates a video having a plurality of the imaged images that are consecutive in chronological order based on the imaged image to which the information related to the external signal has been added.

[0008] The user can see the video and check the information about the external signal added to the captured image, thereby understanding the state of the external signal output from the signal output device. Because the captured image and the external signal are synchronized, the user can understand the timing at which the state of the external signal output from the signal output device changes. This makes it easier to investigate the cause of any problems that may occur. This will shorten the time required for verification and recovery work.

[0009] In the present invention, the signal output unit may be multiple, the imaging unit may acquire the external signals from the multiple signal output units and output the captured images in synchronization with the multiple external signals, and the creation unit may acquire the captured images and the multiple external signals from the imaging unit, add information related to the multiple external signals to the captured images synchronized with the multiple external signals, and create a video having a chronologically consecutive multiple captured images based on the captured images to which information related to the multiple external signals has been added. A user can understand the status of the external signals output from the multiple signal output devices by viewing the video and checking the information related to the multiple external signals added to the captured images.

[0010] In the present invention, the signal output unit may be multiple, the imaging unit may be multiple, the multiple imaging units may acquire the external signals from the signal output units connected to each of the multiple imaging units and output the captured images and the external signals in synchronization, the creation unit may acquire the captured images and the external signals from each of the multiple imaging units, add information about the external signals output from the multiple signal output units to the captured images synchronized with the external signal output from one of the multiple signal output units, and create a video having a plurality of chronologically consecutive captured images based on the captured images to which information about the multiple external signals has been added. A user can understand the status of the external signals output from the multiple signal output devices by viewing the video and checking the information about the multiple external signals added to the captured images.

[0011] In the present invention, the signal output unit may be multiple, the imaging unit may be multiple, the multiple imaging units may acquire the external signals from the multiple signal output units connected to each of the multiple imaging units, and output the captured images and the multiple external signals in synchronization, the creation unit may acquire the captured images and the multiple external signals from each of the multiple imaging units, add information about the external signals output from the multiple signal output units to the captured images synchronized with the external signal output from one of the multiple signal output units, and create a video having the multiple captured images that are consecutive in time series based on the captured images to which the information about the multiple external signals has been added. A user can understand the status of the external signals output from the multiple signal output devices by viewing the video and checking the information about the multiple external signals added to the captured images.

[0012] In the present invention, the signal output unit may be a sensor or a PLC. In the present invention, at least one of the plurality of signal output units may be a sensor, and at least one of the plurality of signal output units may be a PLC. In the present invention, the information related to the external signal may include at least one of letters, numbers, symbols, character strings, number strings, graphs, time charts, and icons. [Effects of the Invention]

[0013] According to the present invention, when a problem occurs, the time required to investigate the cause and carry out recovery work can be reduced. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating a schematic configuration of an information processing system. [Figure 2] FIG. 2 is a block diagram illustrating a schematic configuration of the information processing device. [Figure 3] FIG. 3 is a flowchart showing the flow of processing by the information processing device. [Figure 4] FIG. 4 is a diagram showing an example of a captured image. [Figure 5] FIG. 5 is a diagram illustrating an example of information regarding an external signal. [Figure 6] FIG. 6 is a diagram showing an example of a captured image. DETAILED DESCRIPTION OF THE INVENTION

[0015] [Application example] Hereinafter, application examples of the present invention will be described with reference to the drawings. As shown in FIG. 1, an information processing system 1 according to this application example includes a plurality of signal output devices 10 (10A to 10J), a plurality of imaging devices 11 (11A to 11C), and an information processing device 12. The signal output device 10 is an example of a signal output unit. The imaging device 11 is an example of an imaging unit. The information processing system 1 is employed, for example, in a manufacturing site. In the configuration example shown in FIG. 1, the signal output devices 10A to 10C are connected to the imaging device 11A, the signal output devices 10D to 10F are connected to the imaging device 11B, and the signal output devices 10G to 10J are connected to the imaging device 11C. The imaging devices 11A to 11C are connected to the information processing device 12.

[0016] In the configuration example shown in FIG. 1, signal output devices 10A to 10C are connected to the imaging device 11A, but the number of signal output devices 10 connected to the imaging device 11A is not limited to three and may be one, two, four, or more. In the configuration example shown in FIG. 1, signal output devices 10D to 10F are connected to the imaging device 11B, but the number of signal output devices 10 connected to the imaging device 11B is not limited to three and may be one, two, four, or more. In the configuration example shown in FIG. 1, signal output devices 10G to 10J are connected to the imaging device 11C, but the number of signal output devices 10 connected to the imaging device 11C is not limited to four and may be one, two, three, five, or more. In the configuration example shown in FIG. 1, imaging devices 11A to 11C are connected to the information processing device 12, but the number of imaging devices 11 connected to the information processing device 12 is not limited to three and may be one, two, four, or more.

[0017] The signal output device 10 is a particle sensor, an image sensor, a flow rate sensor, a pressure sensor, a photoelectric sensor, a proximity sensor, a safety sensor (for example, a safety light curtain), a PLC (Programmable Logic Controller), or the like. The signal output device 10 outputs an external signal. The signal output device 10 may be a sensor that performs sensing to obtain various information about the surroundings and outputs an external signal according to the sensing results. The signal output device 10 may also be a controller that acquires sensing results from various sensors, controls a control target, and outputs an external signal according to the sensing results. The signal output device 10 and the imaging device 11 are connected by wire or wirelessly. The imaging device 11 acquires the external signal output from the signal output device 10. When multiple signal output devices 10 are connected to the imaging device 11, the imaging device 11 acquires the external signals output from the multiple signal output devices 10.

[0018] The imaging device 11 is a camera that captures an object and generates a captured image. The imaging device 11 may be an industrial camera. The imaging device 11 generates a captured image at a predetermined frame rate. The imaging device 11 outputs the captured image in synchronization with an external signal acquired from the signal output device 10. When multiple signal output devices 10 are connected to the imaging device 11, the imaging device 11 outputs the captured image in synchronization with multiple external signals acquired from the multiple signal output devices 10. The imaging device 11 and the information processing device 12 are connected via a wired or wireless connection. The information processing device 12 acquires the captured image and the external signal output from the imaging device 11. The external signal output from the signal output device 10 is input to the information processing device 12 via the imaging device 11. This makes it easy for the information processing device 12 to acquire the external signal and the captured image synchronized with the external signal.

[0019] The information processing device 12 may be configured by a computer having, for example, a CPU (processor), RAM, a non-volatile storage device (for example, a ROM, a flash memory, a hard disk, etc.), an I / O, etc. The information processing device 12 may be, for example, a personal computer (PC), an industrial personal computer (IPC), an embedded computer, a smartphone, a tablet terminal, or other processing device. All or part of the functions may be configured using circuits such as ASICs or FPGAs. Alternatively, the information processing device 12 may perform predetermined processing in cooperation with other computers by utilizing distributed computing or cloud computing technology.

[0020] The information processing device 12 adds information about the external signal to a captured image synchronized with the external signal. The information processing device 12 may combine the captured image synchronized with the external signal with the information about the external signal. When the information processing device 12 acquires multiple external signals from the imaging device 11, it adds information about the multiple external signals to the captured image. The information about the external signal may be information indicating the state of the external signal. The information about the external signal includes at least one of letters, numbers, symbols, character strings, number strings, graphs, time charts, and icons. The information processing device 12 creates a video having multiple captured images that are consecutive in time series, based on the captured images to which the information about the external signal has been added.

[0021] The user can view the video (image) and check the information about the external signal added to the captured image, thereby understanding the state of the external signal output from the signal output device 10. Because the captured image and the external signal are synchronized, the user can understand the timing at which the state of the external signal output from the signal output device 10 changes. This makes it possible to shorten the time required to investigate the cause and carry out recovery work when a problem occurs.

[0022] <Embodiment> FIG. 2 is a block diagram schematically illustrating the configuration of the information processing device 12. The information processing device 12 includes an acquisition unit 121, a generation unit 122, an addition unit 123, a creation unit 124, a storage unit 125, a display unit 126, and an operation unit 127. The acquisition unit 121 acquires an external signal and a captured image synchronized with the external signal from the imaging device 11. The generation unit 122 generates information related to the external signal based on the external signal. The addition unit 123 adds information related to the external signal to the captured image synchronized with the external signal. The addition unit 123 may combine the captured image synchronized with the external signal with the information related to the external signal. The creation unit 124 creates a video having a plurality of captured images that are consecutive in time series, based on the captured image to which information related to the external signal has been added. The storage unit 125 stores the video created by the creation unit 124. The display unit 126 displays the video created by the creation unit 124 or the video stored in the storage unit 125. The operation unit 127 is a controller that issues instructions to the information processing device 12 and operates the information processing device 12 .

[0023] The CPU of the information processing device 12 executes a program stored in the storage unit 125, causing the acquisition unit 121, generation unit 122, addition unit 123, and creation unit 124 to perform predetermined processes. At least two of the acquisition unit 121, generation unit 122, addition unit 123, and creation unit 124 may be integrated. The storage unit 125 includes RAM and a non-volatile storage device (e.g., ROM, flash memory, hard disk, etc.). The display unit 126 is, for example, a CRT (Cathode Ray Tube) display, a liquid crystal display, a plasma display, an organic EL (Electro Luminescence) display, etc. The operation unit 127 may be composed of input devices such as a keyboard, a mouse, operation buttons, and a touch panel. Not all of the components of the information processing device 12 shown in FIG. 2 are essential, and components of the information processing device 12 may be added or deleted as appropriate.

[0024] FIG. 3 is a flowchart showing the flow of processing by the information processing device 12. The processing shown in FIG. 3 is started when the information processing device 12 receives a recording instruction. Recording is a process of creating a video including a plurality of captured images that are chronologically continuous. Recording instructions include an instruction for continuous recording and an instruction for triggered recording. Continuous recording is a process of recording at a set time. In continuous recording, for example, a video including a plurality of captured images from time A to time B is created. Trigger recording is a process of starting recording when a set condition is met. In trigger recording, recording is started at a predetermined time (for example, 5 minutes) before and after the timing when the set condition is met. A video including a plurality of captured images is created. In trigger recording, recording may be started when the amount of change between a plurality of chronologically consecutive captured images exceeds a threshold. In trigger recording, a captured image may be compared with a preset still image, and recording may be started when the amount of change exceeds a threshold.

[0025] The information processing device 12 acquires an external signal from the imaging device 11 (S1). The information processing device 12 acquires a captured image synchronized with the external signal from the imaging device 11 (S2). The information processing device 12 generates information related to the external signal (S3). The information processing device 12 adds information related to the external signal to the captured image synchronized with the external signal (S4). The information processing device 12 determines whether or not it is time to end recording (S5). If it is time to end recording (S5; YES), the information processing device 12 creates a video having a plurality of captured images that are consecutive in time series, based on the captured image to which information related to the external signal has been added (S6). If it is not time to end recording (S5; NO), the process returns to S1.

[0026] (Video creation example (1)) An example will be described in which the information processing device 12 creates a moving image by adding information about an external signal output from one signal output device 10 (for example, signal output device 10A) to a captured image. The imaging device 11A acquires the external signal from the signal output device 10A connected to the imaging device 11A, and outputs the captured image and the external signal (the external signal output from the signal output device 10A) in synchronization with each other.

[0027] The information processing device 12 acquires captured images and external signals from the imaging device 11A. The information processing device 12 adds information about the external signals to captured images synchronized with the external signals, and creates a video having a plurality of consecutive captured images in chronological order based on the captured images to which the information about the external signals has been added. In this case, the video is composed of a plurality of captured images output from the imaging device 11A, and information about the external signals output from the signal output device 10A is added to the captured images included in the video. A user can understand the state of the external signals output from the signal output device 10A by viewing the video and checking the information about the external signals added to the captured images. Because the captured images and the external signals are synchronized, the user can understand the timing at which the state of the external signals output from the signal output device 10A changes. This makes it possible to shorten the time required to investigate the cause of a problem and perform recovery work when a problem occurs.

[0028] (Video creation example (2)) An example will be described in which the information processing device 12 creates a moving image by adding information about external signals output from multiple signal output devices 10 (for example, signal output devices 10A to 10C) to a captured image. The imaging device 11A acquires external signals from the signal output devices 10A to 10C connected to the imaging device 11A, and outputs the captured image and the multiple external signals (external signals output from the signal output devices 10A to 10C) in synchronization. The information processing device 12 acquires the captured image and the multiple external signals from the imaging device 11A.

[0029] The information processing device 12 adds information about the external signals to captured images synchronized with the external signals, and creates a video having a plurality of chronologically consecutive captured images based on the captured images to which the information about the external signals has been added. In this case, the video is composed of a plurality of captured images output from the imaging device 11A, and information about the external signals output from the signal output devices 10A to 10C is added to the captured images included in the video. By viewing the video and checking the information about the external signals added to the captured images, a user can understand the status of the external signals output from the signal output devices 10A to 10C. This makes it possible to shorten the time required for cause investigation and recovery work when a problem occurs, and also makes complex analysis possible.

[0030] (Video creation example (3)) An example will be described in which the information processing device 12 creates a moving image by adding information about external signals output from a plurality of signal output devices 10 (for example, signal output devices 10A, 10D, and 10G) to a captured image. The imaging device 11A acquires an external signal from the signal output device 10A connected to the imaging device 11A, and outputs the captured image and the external signal (the external signal output from the signal output device 10A) in synchronization with each other. The imaging device 11B acquires an external signal from the signal output device 10D connected to the imaging device 11B, and outputs the captured image and the external signal (the external signal output from the signal output device 10D) in synchronization with each other. The imaging device 11C acquires an external signal from the signal output device 10G connected to the imaging device 11C, and outputs the captured image and the external signal (the external signal output from the signal output device 10G) in synchronization with each other.

[0031] The information processing device 12 acquires a captured image and an external signal (an external signal output from the signal output device 10A) from the imaging device 11A. The information processing device 12 acquires a captured image and an external signal (an external signal output from the signal output device 10D) from the imaging device 11B. The information processing device 12 acquires a captured image and an external signal (an external signal output from the signal output device 10G) from the imaging device 11C. In this way, the information processing device 12 acquires captured images and external signals from the imaging devices 11A to 11C.

[0032] The information processing device 12 adds information about the external signal output from one of the signal output devices 10A, 10D, and 10G to a captured image synchronized with the external signal output from one of the signal output devices 10A, 10D, and 10G. The information processing device 12 creates a moving image having a plurality of captured images that are consecutive in time series, based on the captured images to which the information about the external signal output from the signal output devices 10A, 10D, and 10G has been added. Therefore, the captured images included in the moving image have information about the external signal output from the signal output devices 10A, 10D, and 10G added.

[0033] When information related to the external signals output from the signal output devices 10A, 10D, and 10G is added to captured images synchronized with the external signals output from the signal output device 10A, the moving image is composed of a plurality of captured images output from the imaging device 11A. When information related to the external signals output from the signal output devices 10A, 10D, and 10G is added to captured images synchronized with the external signals output from the signal output device 10D, the moving image is composed of a plurality of captured images output from the imaging device 11B. When information related to the external signals output from the signal output devices 10A, 10D, and 10G is added to captured images synchronized with the external signals output from the signal output device 10G, the moving image is composed of a plurality of captured images output from the imaging device 11C.

[0034] By viewing the video and checking the information about the multiple external signals added to the captured image, the user can understand the status of the external signals output from the signal output devices 10A, 10D, and 10G. This makes it possible to shorten the time required to investigate the cause and perform recovery work when a problem occurs, and also makes complex analysis possible.

[0035] For example, by viewing a video made up of a plurality of captured images output from imaging device 11A, a user can grasp the states of the external signals output from signal output devices 10A, 10D, and 10G. In this way, by viewing one video, a user can grasp the states of the external signals output from signal output device 10A connected to imaging device 11A, the external signals output from signal output device 10D connected to imaging device 11B, and the external signals output from signal output device 10G connected to imaging device 11C.

[0036] (Video creation example (4)) An example will be described in which the information processing device 12 creates a moving image by adding information about external signals output from multiple signal output devices 10 (e.g., signal output devices 10A to 10J) to captured images. The imaging device 11A acquires external signals from signal output devices 10A to 10C connected to the imaging device 11A, and outputs the captured images and the multiple external signals (external signals output from the signal output devices 10A to 10C) in synchronization with each other. The imaging device 11B acquires external signals from signal output devices 10D to 10F connected to the imaging device 11B, and outputs the captured images and the multiple external signals (external signals output from the signal output devices 10D to 10F) in synchronization with each other. The imaging device 11C acquires external signals from signal output devices 10G to 10J connected to the imaging device 11C, and outputs the captured images and the multiple external signals (external signals output from the signal output devices 10G to 10J) in synchronization with each other.

[0037] The information processing device 12 acquires a captured image and a plurality of external signals (external signals output from the signal output devices 10A to 10C) from the imaging device 11A. The information processing device 12 acquires a captured image and a plurality of external signals (external signals output from the signal output devices 10D to 10F) from the imaging device 11B. The information processing device 12 acquires a captured image and a plurality of external signals (external signals output from the signal output devices 10G to 10J) from the imaging device 11C. In this way, the information processing device 12 acquires a captured image and a plurality of external signals from each of the imaging devices 11A to 11C.

[0038] The information processing device 12 adds information about the external signal output from one of the signal output devices 10A to 10J to a captured image synchronized with the external signal output from one of the signal output devices 10A to 10J. The information processing device 12 creates a moving image having a plurality of captured images that are consecutive in time series, based on the captured images to which the information about the external signal output from the signal output devices 10A to 10J has been added. Therefore, the captured images included in the moving image have information about the external signal output from the signal output devices 10A to 10J added to them.

[0039] When information related to the external signal output from one of the signal output devices 10A to 10J is added to a captured image synchronized with the external signal output from one of the signal output devices 10A to 10C, the moving image is made up of a plurality of captured images output from the imaging device 11A. When information related to the external signal output from one of the signal output devices 10A to 10J is added to a captured image synchronized with the external signal output from one of the signal output devices 10D to 10F, the moving image is made up of a plurality of captured images output from the imaging device 11B. When information related to the external signal output from one of the signal output devices 10A to 10J is added to a captured image synchronized with the external signal output from one of the signal output devices 10G to 10J, the moving image is made up of a plurality of captured images output from the imaging device 11C.

[0040] By viewing the video and checking the information about the multiple external signals added to the captured image, the user can understand the status of the external signals output from the signal output devices 10A to 10J. This makes it possible to shorten the time required to investigate the cause and perform recovery work when a problem occurs, and also makes complex analysis possible.

[0041] For example, by viewing a moving image made up of a plurality of captured images output from imaging device 11A, a user can grasp the states of the external signals output from signal output devices 10A to 10J. In this way, by viewing one moving image, a user can grasp the states of the external signals output from signal output devices 10A to 10C connected to imaging device 11A, the states of the external signals output from signal output devices 10D to 10F connected to imaging device 11B, and the states of the external signals output from signal output devices 10G to 10J connected to imaging device 11C.

[0042] The imaging devices 11A to 11C may set time information in the captured image. The information processing device 12 may use the time information set in the captured image to add information about the external signal output from one of the signal output devices 10A to 10J to the captured image synchronized with the external signal output from that signal output device. An example of a process for adding information about each external signal to a captured image synchronized with an external signal is shown below. The information processing device 12 acquires an external signal A1 and a captured image A2 synchronized with the external signal A1 from the imaging device 11A. The information processing device 12 acquires an external signal B1 and a captured image B2 synchronized with the external signal B1 from the imaging device 11B. The information processing device 12 acquires an external signal C1 and a captured image C2 synchronized with the external signal C1 from the imaging device 11C.

[0043] The information processing device 12 may use the time information set in the captured images A2, B2, and C2 to add information about the external signal A1, information about the external signal B1, and information about the external signal C1 to the captured image A2 synchronized with the external signal A1. The information processing device 12 may use the time information set in the captured images A2, B2, and C2 to add information about the external signal A1, information about the external signal B1, and information about the external signal C1 to the captured image B2 synchronized with the external signal B1. The information processing device 12 may use the time information set in the captured images A2, B2, and C2 to add information about the external signal A1, information about the external signal B1, and information about the external signal C1 to the captured image C2 synchronized with the external signal C1.

[0044] Fig. 4 is a diagram showing an example of a captured image. Fig. 4 shows a captured image obtained by capturing an image of the robot 200 and the conveyor 300. The robot 200 has a robot arm 201 that performs a predetermined task on a workpiece transported by the conveyor 300. The workpiece may be, for example, a final product, an intermediate product, a semi-finished product, a part, or a material. Here, the image is captured by the imaging device 11A, and sensors 211 to 213 are connected to the imaging device 11A.

[0045] The sensor 211 is a monitoring sensor that monitors the operation of the robot arm 201. If the operating range of the robot arm 201 exceeds a predetermined range, the sensor 211 detects that the operation of the robot arm 201 is abnormal. If the operating range of the robot arm 201 exceeds the predetermined range, the sensor 211 outputs an ON signal as an external signal. If the operating range of the robot arm 201 is within the predetermined range, the sensor 211 outputs an OFF signal as an external signal.

[0046] The sensor 211 may be a current clamp attached to the cable of an indicator light of a monitoring device that monitors the operation of the robot arm 201. When the motion range of the robot arm 201 exceeds a predetermined range, current flows through the indicator light, causing the indicator light to light up. When the motion range of the robot arm 201 is within the predetermined range, no current flows through the indicator light, causing the indicator light to not light up. When current flows through the indicator light, the sensor 211 outputs an ON signal as an external signal. When no current flows through the indicator light, the sensor 211 outputs an OFF signal as an external signal.

[0047] The sensor 212 is a sensor that detects an abnormality in the conveyor 300. If the conveyor 300 is abnormal, the sensor 212 outputs an ON signal as an external signal. If the conveyor 300 is normal, the sensor 212 outputs an OFF signal as an external signal. For example, if the sensor 212 detects an abnormal sound from the conveyor 300, it detects that the conveyor 300 is abnormal.

[0048] The sensor 212 may be a current clamp attached to the cable of an indicator light of a detection device that detects abnormalities in the conveyor 300. When the conveyor 300 is abnormal, current flows through the indicator light, causing the indicator light to light up. When the conveyor 300 is normal, no current flows through the indicator light, causing the indicator light to not light up. When current flows through the indicator light, the sensor 212 outputs an ON signal as an external signal. When no current flows through the indicator light, the sensor 212 outputs an O signal as an external signal. Outputs the FF signal.

[0049] Sensor 213 is a vibration sensor that measures vibrations of robot arm 201. Sensor 213 is installed near the base of robot arm 201. If the measured vibration value is not within a predetermined range, sensor 213 outputs an ON signal as an external signal. If the measured vibration value is within the predetermined range, sensor 213 outputs an OFF signal as an external signal.

[0050] The imaging device 11A acquires external signals output from the sensors 211 to 213. The imaging device 11A synchronously outputs the captured image and the external signals acquired from the sensors 211 to 213. The information processing device 12 acquires the captured image and the external signal output from the imaging device 11A. The information processing device 12 generates information related to the external signal and adds the information related to the external signal to the captured image synchronized with the external signal. In an example of a captured image shown in FIG. 4, the information related to the external signal is superimposed in a predetermined area of ​​the captured image. The captured image with the information related to the external signal added is displayed on the display unit 126 of the information processing device 12. Alternatively, the information processing device 12 may transmit the captured image with the information related to the external signal added to an external display device, and the external display device may display the captured image with the information related to the external signal added.

[0051] In the example of the captured image shown in Fig. 4, the information about the external signal is composed of "sensor 211: OFF", "sensor 212: OFF", and "sensor 213: ON". As shown in Fig. 5, the information about the external signal may be composed of letters, numbers, and a time chart.

[0052] For example, if a scratch of unknown cause occurs on a workpiece at a site where the robot 200 is used, by analyzing the captured image shown in FIG. 4, it can be inferred that the cause is in the vicinity of the robot arm 201 used during processing. By visually checking the captured image shown in FIG. 4, a user can confirm whether the robot 200 is operating or not. The captured image shown in FIG. 4 is annotated with "sensor 211: OFF," "sensor 212: OFF," and "sensor 213: ON." Because "sensor 213: ON" is annotated in the captured image shown in FIG. 4, the user can understand that the vibration value measured by the sensor 213 is not within a predetermined range. The user can infer that the scratch on the workpiece is caused by the vibration of the robot arm 201 being outside the expected range.

[0053] The user can view the video using the information processing device 12 or an external display device and check the information about the external signals added to the captured image, thereby understanding the state of the external signals output from the sensors 211 to 213. Because the captured image and the external signals are synchronized, the user can understand the timing at which the state of the external signals output from the sensors 211 to 213 changes. For example, the user can understand the timing at which the external signal of the sensor 213 changes from OFF to ON. This makes it possible to understand the status of the equipment at the manufacturing site, and shorten the time required to investigate the cause and perform recovery work when a problem occurs.

[0054] Fig. 6 is a diagram showing an example of a captured image. Fig. 6 shows a captured image when an image is captured inside a clean room. A precision device 400 and a particle sensor 401 are installed inside the clean room. The particle sensor 401 detects the amount of particles inside the clean room. Sensors 411 to 414 are provided on the precision device 400. Here, an image capture device 11C captures images, and the sensors 411 to 414 are connected to the image capture device 11C.

[0055] The sensor 411 is a sensor that detects whether the door of the precision device 400 is opened or closed. For example, the sensor 411 is a proximity sensor. When the door of the precision device 400 is open, the sensor 411 outputs an ON signal as an external signal. When the door of the precision device 400 is closed, the sensor 411 outputs an OFF signal as an external signal.

[0056] The sensor 412 is a temperature sensor that measures the temperature inside the precision device 400. If the temperature inside the precision device 400 is not within a predetermined range, the sensor 412 outputs an ON signal as an external signal. If the temperature inside the precision device 400 is within the predetermined range, the sensor 412 outputs an OFF signal as an external signal.

[0057] The sensor 413 is a humidity sensor that measures the humidity inside the precision device 400. If the humidity inside the precision device 400 is not within a predetermined range, the sensor 413 outputs an ON signal as an external signal. If the humidity inside the precision device 400 is within the predetermined range, the sensor 413 outputs an OFF signal as an external signal.

[0058] The sensor 414 is a vibration sensor that measures vibrations of the precision device 400. If the measured vibration value is not within a predetermined range, the sensor 414 outputs an ON signal as an external signal. If the measured vibration value is within the predetermined range, the sensor 414 outputs an OFF signal as an external signal.

[0059] The imaging device 11C acquires external signals output from the sensors 411 to 414. The imaging device 11C synchronously outputs the captured image and the external signals acquired from the sensors 411 to 414. The information processing device 12 acquires the captured image and the external signal output from the imaging device 11C. The information processing device 12 generates information related to the external signal and adds the information related to the external signal to the captured image synchronized with the external signal. In an example of a captured image shown in FIG. 6, the information related to the external signal is superimposed in a predetermined area of ​​the captured image. The captured image with the information related to the external signal added is displayed on the display unit 126 of the information processing device 12. Alternatively, the information processing device 12 may transmit the captured image with the information related to the external signal added to an external display device, and the external display device may display the captured image with the information related to the external signal added.

[0060] For example, if particle sensor 401 detects a particle amount equal to or greater than a threshold, by analyzing the captured image shown in FIG. 6, it can be inferred that the precision device 400 is the cause of the increase in particle amount in the clean room. "Sensor 411: ON," "Sensor 412: OFF," "Sensor 413: ON," and "Sensor 414: OFF" are added to the captured image shown in FIG. 6. Because "Sensor 411: ON" and "Sensor 413: ON" are added to the captured image shown in FIG. 6, the user can understand that the door of precision device 400 is open and the humidity inside precision device 400 is not within a predetermined range. The user can infer that the increase in particle amount in the clean room is caused by the door of precision device 400 not being closed and the humidity inside precision device 400 being outside the predetermined range.

[0061] The user can view the video using the information processing device 12 or an external display device and check the information about the external signals added to the captured image, thereby understanding the state of the external signals output from the sensors 411 to 414. Because the captured image and the external signals are synchronized, the user can understand the timing at which the state of the external signals output from the sensors 411 to 414 changes. For example, the user can understand the timing at which the external signal of the sensor 411 changes from OFF to ON and the timing at which the external signal of the sensor 413 changes from OFF to ON. This makes it possible to understand the status of the equipment at the manufacturing site, and shorten the time required to investigate the cause of a problem and perform recovery work when one occurs.

[0062] Furthermore, each of the processes described above may be regarded as an information processing method of the information processing system 1. Each of the processes described above may be regarded as a method executed by a computer. A program for causing a computer to execute each of the processes described above may be provided to the computer via a network or from a computer-readable recording medium that non-temporarily stores data. Note that the above means and processes may be combined with each other as much as possible to constitute the present invention.

[0063] <Additional Notes> a signal output unit (10) that outputs an external signal; an imaging unit (11) that acquires the external signal from the signal output unit (10) and outputs a captured image generated by capturing an image of an object in synchronization with the external signal; a creation unit (10, 124) that acquires the captured image and the external signal from the imaging unit (11), adds information about the external signal to the captured image synchronized with the external signal, and creates a moving image having a plurality of the captured images that are continuous in time series based on the captured image to which the information about the external signal has been added; Equipped with Information Processing Systems (1). [Explanation of symbols]

[0064] 1. Information Processing System 10(10~10J)...Signal output device 11 (11A to 11C) Imaging device 12. Information processing device 121...Acquisition part 122...Generation section 123...Additional part 124···Creation Department 125...Storage section 126...Display section 127...Operation unit

Claims

1. A signal output unit that outputs an external signal; an imaging unit that acquires the external signal from the signal output unit and outputs a captured image generated by capturing an image of an object in synchronization with the external signal; a creating unit that acquires the captured images and the external signal from the imaging unit, adds information related to the external signal to the captured images synchronized with the external signal, and creates a moving image having a plurality of the captured images that are continuous in time series based on the captured images to which the information related to the external signal has been added; Equipped with the signal output unit is a plurality of units, the imaging unit acquires the external signals from the plurality of signal output units, and outputs the captured image and the plurality of external signals in synchronization with each other; the creation unit acquires the captured images and the plurality of external signals from the imaging unit, adds information about the plurality of external signals to the captured images synchronized with the plurality of external signals, and creates a moving image having the plurality of captured images that are continuous in time series based on the captured images to which information about the plurality of external signals has been added. Information processing system.

2. A signal output unit that outputs an external signal; an imaging unit that acquires the external signal from the signal output unit and outputs a captured image generated by capturing an image of an object in synchronization with the external signal; a creating unit that acquires the captured images and the external signal from the imaging unit, adds information related to the external signal to the captured images synchronized with the external signal, and creates a moving image having a plurality of the captured images that are continuous in time series based on the captured images to which the information related to the external signal has been added; Equipped with the signal output unit is a plurality of units, the imaging unit is a plurality of units, the plurality of imaging units acquire the external signals from the signal output units connected to each of them, and output the captured images and the external signals in synchronization with each other; the creation unit acquires the captured image and the external signal from each of the plurality of imaging units; adding information about the external signals output from one of the plurality of signal output units to the captured images synchronized with the external signals output from one of the plurality of signal output units, and creating a moving image having a plurality of captured images that are continuous in time series based on the captured images to which information about the plurality of external signals has been added; Information processing system.

3. A signal output unit that outputs an external signal; an imaging unit that acquires the external signal from the signal output unit and outputs a captured image generated by capturing an image of an object in synchronization with the external signal; a creating unit that acquires the captured images and the external signal from the imaging unit, adds information related to the external signal to the captured images synchronized with the external signal, and creates a moving image having a plurality of the captured images that are continuous in time series based on the captured images to which the information related to the external signal has been added; Equipped with the signal output unit is a plurality of units, the imaging unit is a plurality of units, the plurality of imaging units acquire the external signals from the plurality of signal output units connected thereto, and output the captured images and the plurality of external signals in synchronization with each other; the creation unit acquires the captured image and the plurality of external signals from each of the plurality of imaging units, adds information about the external signal output from one of the plurality of signal output units to the captured image synchronized with the external signal output from one of the plurality of signal output units, and creates a moving image having the plurality of captured images that are continuous in time series based on the captured image to which information about the plurality of external signals has been added. Information processing system.

4. At least one of the plurality of signal output units is a sensor, At least one of the plurality of signal output units is a PLC. The information processing system according to any one of claims 1 to 3.

5. the information about the external signal includes at least one of a character, a number, a symbol, a character string, a number string, a graph, a time chart, and an icon; The information processing system according to any one of claims 1 to 4.

Images