Information processing method, program, and information processing system

Abstract

To accurately detect abnormality even if there is a plurality of instruments.SOLUTION: An information processing method includes: a segment step of segmenting a subject included in a moving image for each detection object; a specification step of specifying the kind of detection objects, for each of the detection objects segmented in the segment step; an extraction step of extracting numeric values for each of the detection objects specified in the specification step; and a detection step of detecting abnormality for each of the detection objects from the numeric values extracted in the extraction step.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an information processing method, a program, and an information processing system. [Background technology]

[0002] Patent Document 1 discloses an anomaly detection device and an anomaly detection method that utilizes AI to determine the timing and frequency at which an anomaly occurs in a detection target based on waveform signal data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7431372 Summary of the Invention [Problem to be solved by the invention]

[0004] However, while the above-mentioned prior art allows for anomaly detection using only waveform signals, it is difficult to detect anomalies using data other than waveform signals. For example, the technology of Patent Document 1 is difficult to apply to anomaly detection in a device that requires checking instruments other than waveform signals. Also, there may be multiple instruments to be measured, and in such cases, it may be necessary to measure these instruments simultaneously.

[0005] In consideration of the above, an object of the present invention is to provide an information processing method, a program, and an information processing system that can accurately detect abnormalities even when multiple instruments are present. [Means for solving the problem]

[0006] According to one embodiment, the information processing method is executed by an information processing device and includes a segmentation step of segmenting a subject included in a moving image into each detection target, an identification step of identifying the type of detection target for each detection target segmented in the segmentation step, an extraction step of extracting a numerical value for each detection target identified in the identification step, and a detection step of detecting an abnormality for each detection target from the numerical value extracted in the extraction step.

[0007] According to one embodiment of the program, an information processing device is caused to execute an information processing method including a segmentation step of segmenting subjects included in a moving image into detection targets, an identification step of identifying the type of detection target for each detection target segmented in the segmentation step, an extraction step of extracting a numerical value for each detection target identified in the identification step, and a detection step of detecting an abnormality for each detection target from the numerical value extracted in the extraction step.

[0008] According to one embodiment, the information processing system is executed by an information processing device and includes a segmentation step for segmenting subjects included in a moving image into detection targets, an identification step for identifying the type of detection target for each detection target segmented in the segmentation step, an extraction step for extracting a numerical value for each detection target identified in the identification step, and a detection step for detecting an abnormality for each detection target from the numerical value extracted in the extraction step. [Effects of the Invention]

[0009] According to one embodiment, even when a plurality of meters are present, abnormalities can be detected with high accuracy. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment. [Figure 2]FIG. 2 is a diagram illustrating an example of a hardware configuration of a server according to an embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a user terminal according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of a server according to an embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a functional configuration of a user terminal according to an embodiment. [Figure 6] 10 is a flowchart illustrating an example of pre-processing for abnormality detection executed by the information processing system according to the embodiment. [Figure 7] 10 is a flowchart illustrating an example of an abnormality detection process executed by the information processing system according to the embodiment. [Figure 8] FIG. 2 is a diagram showing an overview of equipment 4 according to an embodiment. [Figure 9] FIG. 10 is a diagram showing an example of segmenting each instrument according to the embodiment. [Figure 10] FIG. 10 is a diagram showing instrument values ​​in a time series according to the embodiment. [Figure 11] 10A and 10B are diagrams illustrating an example of a display when an abnormality is detected according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment of an information processing system according to the present invention will be described below with reference to Figures 1 to 5. In each drawing, the same or equivalent components and parts are denoted by the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0012] (System Overview) First, an overview of the information processing system 10 according to this embodiment will be described. The information processing system 10 according to this embodiment is a system that can monitor an instrument that is a detection target from video image data and detect an abnormality. In this embodiment, the information processing system 10 is a system that can detect which instrument has detected an abnormality while monitoring a plurality of different instruments.

[0013] (System Configuration) FIG. 1 is a diagram illustrating an example of the configuration of an information processing system 10 according to this embodiment. As shown in FIG. 1, the information processing system 10 according to this embodiment includes a server device 1, a user terminal 2, and an image capturing device 3 that captures images of equipment 4 equipped with meters, all of which are communicably connected via a network N. The network N may be, for example, a wired local area network (LAN), a wireless LAN, the Internet, a public line network, a mobile data communication network, or a combination thereof. In the example of FIG. 1, the information processing system 10 includes one server device 1, one user terminal 2, and one image capturing device 3, but may also include multiple of each. Furthermore, the user terminal 2 may be an information processing device equipped with an image capturing device, and moving images captured by the image capturing device 3 may be communicated directly to the server device 1 without going through the user terminal 2.

[0014] The server device 1 is an example of an information processing device that generates prompts based on the situation of an image input by the user terminal 2 and generates a video from the image. The server device 1 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, or a combination of these. The specific configuration and operation of the server device 1 will be described later.

[0015] The user terminal 2 is an example of an information processing device that performs operations for inputting and displaying various information. The user terminal 2 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, a wearable device, or a combination of these. In this embodiment, an information processing device equipped with a photographing function is used as an example.

[0016] The photographing device 3 is an information processing device that photographs the equipment 4 equipped with an instrument that is a target for abnormality detection. The photographing device 3 may be, but is not limited to, a digital camera, a surveillance camera, a smartphone, or an AI camera. The photographing device 3 transmits video images of the instrument that is a target for abnormality detection to the user terminal 2. In the example of FIG. 1, it is assumed that the instrument equipped in the equipment 4 is photographed by the photographing device 3 that is fixedly installed, but this is not limited thereto and the device may be installed in an optimal manner for abnormality detection. Furthermore, in this embodiment, the video images photographed by the photographing device 3 are transmitted to the user terminal 2, but this is not limited thereto and the video images may be transmitted directly to the server device 1. In this embodiment, the equipment 4 is equipment such as a production line installed in a factory or the like.

[0017] (Hardware configuration - Server) 2 is a block diagram showing the hardware configuration of the server device 1. The server device 1 includes a processor 101, a memory 102, a storage 103, and a communication I / F 104, which are communicably connected to each other via a bus B.

[0018] The processor 101 controls each component of the server device 1 and realizes the functions of the server device 1 by loading various programs stored in the storage 103 into the memory 102 and executing them. The programs executed by the processor 101 include, but are not limited to, an OS (Operating System) and various programs described below. Execution of these programs by the processor 101 realizes part of the state visualization method according to this embodiment. The processor 101 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), or a combination thereof.

[0019] The memory 102 is, for example, a read-only memory (ROM), a random access memory (RAM), or a combination thereof. The ROM is, for example, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a combination thereof. The RAM is, for example, a dynamic random access memory (DRAM), a static random access memory (SRAM), a magnetoresistive random access memory (MRAM), or a combination thereof.

[0020] The storage 103 stores an OS, various programs (described later), and various data. The storage 103 is, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a storage class memory (SCM), or a combination of these.

[0021] The communication I / F 104 is an interface for connecting the server device 1 to external devices including the user terminal 2 via the network N and controlling communication. The communication I / F 104 is, for example, an adapter compliant with Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), Ethernet (registered trademark), or optical communication (e.g., Fibre Channel), but is not limited to these.

[0022] (Hardware configuration - user terminal) 3 is a block diagram showing the hardware configuration of the user terminal 2. The user terminal 2 includes a processor 201, a memory 202, a storage 203, a communication I / F 204, an input / output I / F 205, an input device 206, and an output device 207, which are communicatively connected to each other via a bus B. The communication I / F 204 is an interface for connecting the user terminal 2 to external devices including the server device 1 via a network N and for controlling communication. The communication I / F 204 is, for example, an adapter compliant with Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), Ethernet (registered trademark), or optical communication (e.g., Fibre Channel), but is not limited to these.

[0023] The input / output I / F 205 is an interface for connecting an input device 206 and an output device 207 to the user terminal 2. The input device 206 is, for example, a mouse, a keyboard, a touch panel, a microphone, a scanner, a camera, various sensors, operation buttons, or a combination of these. The output device 207 as a user interface is, for example, a display, a projector, a printer, a speaker, a vibrator, or a combination of these. In this embodiment, as an example, the output device 207 and the input device 206 are an integrally configured touch panel display.

[0024] In this embodiment, the program may be written to the memory 202 or the storage 203 during the manufacturing stage of the server device 1, or may be provided to the server device 1 via the network N. Alternatively, the program may be provided to the server device 1 via a non-transitory computer-readable recording medium such as a disk medium (not shown).

[0025] (Functional configuration - Server) Next, the functional configuration of the server device 1 will be described. Fig. 4 is a diagram showing an example of the functional configuration of the server device 1. When executing various programs, the server device 1 realizes various functions using the above-mentioned hardware resources. The server device 1 has a communication unit 11, a storage unit 12, and a control unit 13 as functional configurations realized by the server device 1. Each functional configuration is realized by the processor 101 reading and executing a program stored in the memory 102 or storage 103.

[0026] The control unit 13 includes an acquisition unit 131 that acquires video images sent from the user terminal 2 and a segmentation unit 132 that segments the objects of the video images acquired by the acquisition unit 131 into individual instruments for which abnormalities are to be detected. The control unit 13 also includes an identification unit 133 that identifies the type of each instrument segmented by the segmentation unit 132, an extraction unit 134 that extracts the numerical values ​​of each instrument identified by the identification unit 133, and a detection unit 135 that detects abnormalities in each instrument. The segmentation unit 132 executes an object extraction algorithm 121 to segment the objects included in the video into individual instruments. The identification unit 133 executes a detection target extraction algorithm 122 to identify the type of instrument each segmented by the segmentation unit 132 is. The extraction unit 134 executes a numerical value extraction algorithm 123 to extract the numerical values ​​of each instrument. Note that when extracting the numerical values ​​of each instrument, other image processing techniques may be used instead of executing the numerical value extraction algorithm 123, depending on the type of instrument. For example, if the identified instrument displays a digital value, the value may be extracted using OCR technology. Details of these will be described later.

[0027] (Functional configuration - User terminal) Next, the functional configuration of the user terminal 2 will be described. Figure 5 is a diagram showing an example of the functional configuration of the user terminal 2. When executing various programs, the user terminal 2 uses the above-mentioned hardware resources to realize various functions. The user terminal 2 has, as functional components realized by the user terminal 2, a communication unit 21, a storage unit 22 in which a program 221 is stored, and a control unit 23. Each functional component is realized by the processor 201 reading and executing the program 221 stored in the memory 202 or the storage 203. The control unit 23 includes an information acquisition control unit 231 that acquires information sent from the server device 1, and a display unit 232 that displays the acquired information on the output device 207.

[0028] Next, the processing flow of the information processing system according to this embodiment will be described with reference to Figs. 6 to 11. Fig. 6 is a flowchart of pre-processing for anomaly detection executed by the information processing system 10 according to this embodiment. Fig. 7 is a flowchart of processing for anomaly detection executed by the information processing system 10 according to this embodiment. Fig. 8 is a diagram showing an overview of the equipment 4 according to this embodiment. Fig. 9 is a diagram showing an example of segmenting each meter according to this embodiment. Fig. 10 is a diagram showing chronological meter values ​​according to this embodiment. Fig. 11 is a diagram showing an example of a display when an anomaly is detected according to this embodiment.

[0029] First, the preprocessing for detecting an abnormality in an instrument will be described with reference to Fig. 6. In step S101, the acquisition unit 131 acquires a moving image captured by the image capturing device 3 and sent via the user terminal 2. Next, in step S102, the segmentation unit 132 segments the subject of the moving image acquired in step S101 into each instrument. Here, the segmentation in step S102 will be described with reference to Figs. 8 and 9.

[0030] FIG. 8 is a diagram illustrating an example of a moving image acquired in step S101. In this embodiment, the image capturing device 3 captures an image of the equipment 4 including instruments K1 to K4 and buttons B1 and B2. FIG. 9 is a diagram illustrating an example of segmenting and displaying each of the instruments K1 to K4, which are the subjects of the moving image shown in FIG. 8. As shown in FIG. 9, the segmentation unit 132 executes the object extraction algorithm 121 to frame each of the instruments K1 to K4 in FIG. 8 and segment each of them. Specifically, each of the instruments K1 to K4 is segmented as shown by frames C1 to C4. In the example of FIG. 8, buttons B1 and B2 are present, but in this embodiment, in order to perform anomaly detection, instruments other than the instruments K1 to K4, such as buttons B1 and B2, are not detected. Note that in this embodiment, instruments other than the instruments K1 to K4 are not detected. However, this is not limited to this. For example, a method may be used in which all of the subjects are segmented and only those subjects determined to be instruments are extracted. It is sufficient for the instruments K1 to K4 to be detected and segmented.

[0031] In this embodiment, the number of instruments in the video and the appearance of each instrument are detected so that they can be determined. Specifically, the segmentation unit 132 can determine that the equipment 4 includes four instruments, K1 to K4. That is, the segmentation unit 132 performs segmentation after determining that instrument K1 is an analog display instrument, instrument K2 is a flow meter, instrument K3 is an analog display instrument, and instrument K4 is a digital display instrument. In this embodiment, the subject extraction algorithm 121 is executed to determine the number of instruments and even the approximate appearance of the instruments. However, without being limited to this, different algorithms may be executed, such as an algorithm that detects only the number of instruments or an algorithm that determines the appearance of the instruments.

[0032] Returning to the flowchart of FIG. 6, in step S103, the identification unit 133 executes the detection target extraction algorithm 122 to identify the type of each meter segmented in step S102. In the present embodiment, as an example, the identification unit 133 identifies each meter as follows: meter K1 is a voltmeter, meter K2 is a flowmeter, meter K3 is a pressure gauge, and meter K4 is a thermometer. The type of each meter is identified based on the number of meters and the appearance of the meters (analog meter, digital meter, etc.) detected in step S102. At this time, if the identified number does not match the number segmented in step S102, the identification unit 133 continues to identify the type of meter until the numbers match.

[0033] Next, in step S104, the extraction unit 134 executes the numerical value extraction algorithm 123 to extract the numerical value of each meter. For example, since the meter K1 is identified as a voltmeter, a numerical value corresponding to the voltmeter can be extracted. At this time, the extraction unit 134 also extracts the scale of each meter (for example, the interval between each scale). Then, in step S105, the extraction unit 134 detects the maximum and minimum values ​​of the meter from the extracted numerical values. At this time, the extraction unit 134 also extracts the current numerical values ​​of each of the meters K1 to K4. Next, in step S106, the meter types and numerical values ​​(maximum values, minimum values, scale intervals, etc.) of the meters K1 to K4 are stored in the storage unit 12.

[0034] Next, the process of detecting an abnormality in an instrument will be described with reference to Fig. 7. First, in step S201, the photographing device 3 photographs moving images of the instruments K1 to K4 included in the equipment 4, and the user terminal 2 acquires the moving images. Next, in step S202, the user terminal 2 transmits the acquired moving images to the server device 1. Next, in step S203, the server device 1 receives the moving images. Next, in step S204, the extraction unit 134 executes the numerical value extraction algorithm 123 to extract the numerical values ​​of the instruments K1 to K4. Then, in step S205, the detection unit 135 compares the values ​​with abnormal values, and in step S206, it is determined whether or not an abnormal value has been detected.

[0035] Here, abnormal value detection will be described. FIG. 10 is a diagram showing instrument values ​​over time. In this embodiment, an example will be described using the variation in the numerical value of instrument K1. The value of instrument K1 increases over time and then decreases at a certain point. For example, by pre-storing the value as a 24-hour change, an abnormality will not be determined simply because the numerical value changed. That is, when comparing with the abnormal value in step S205, the detection unit 135 takes the passage of time into consideration and detects an abnormality if the value deviates from the flow shown in FIG. 10. That is, rather than detecting an abnormality by comparing with a specific numerical value, an abnormality can be detected by comparing with the numerical value over time, taking into account changes in the numerical value over time. Note that if there is no change in the numerical value over time, an abnormal value may be set in advance and an abnormality may be detected by comparing with the set abnormal value. This allows abnormalities to be detected according to the type of instrument and the change in the numerical value of each instrument. Furthermore, when comparing with the abnormal value in step S205, the detection unit 135 may use a trained model to determine whether or not an abnormality exists. At this time, for example, if an abnormality is detected despite the normal state, the normal state can be re-learned by notifying the server device 1 that the user terminal 2 is normal. In this way, re-learning can improve the accuracy of abnormality detection. When re-learning, a flow for a user check can be added after step S206, and if re-learning is necessary, a process can be added in which the detected abnormality is not sent to the user terminal 2.

[0036] If an abnormal value is detected (YES), in step S207, the detection unit 135 determines which of the meters K1 to K4 detected the abnormal value. If no abnormality is detected, the process returns to step S205 and periodically compares the abnormal value with the value until an abnormality is detected. Note that the comparison with the abnormal value may be performed, for example, every hour, every minute, or every 1 ms; the frequency is not limited. Then, in step S208, the server device 1 transmits the detection result. Next, in step S209, the user terminal 2 receives the detection result transmitted from the server device 1, and in step S210, the display unit 232 of the user terminal 2 displays the detection result. An example of the detection result is shown in FIG. 11. FIG. 11 is a diagram showing an example of a display when an abnormality is detected. In the example of FIG. 11, an abnormality is detected in the meter K1, and the meter K1 is displayed in gray as shown in a frame C1'. In this embodiment, an instrument in which an abnormality has been detected is displayed in gray, but this is not limited to this, and it may be displayed in any other color, flashing, sounding, or other way that makes it clear that an abnormality has occurred.

[0037] In this embodiment, in step S204, the extraction unit 134 executes the numerical value extraction algorithm 123 to extract the numerical value, but the present invention is not limited to this, and the numerical value may be extracted using another method, such as image processing or OCR (Optical Character Recognition). For example, if the type of instrument is a simple analog meter, the minimum and maximum values ​​are known, and the scale is uniform, the numerical value may be obtained using an image processing method. Also, if the type of instrument is a simple digital display and the numerical value is known, the numerical value may be obtained using an OCR-based method.

[0038] In this way, in this embodiment, even when there are multiple different instruments, it is possible to accurately detect abnormalities for each instrument. Furthermore, because it is possible to detect what types of instruments are present and their numerical values ​​from the video images, it is possible to easily detect abnormalities without having to make detailed settings in advance.

[0039] As described above, according to this embodiment, when detecting an abnormality in a case where a plurality of different instruments are present, it is possible to accurately detect an abnormality for each instrument. Note that in this embodiment, a method of detecting an abnormality in an instrument from a moving image has been described, but this is not limited to this, and abnormality detection may also be performed from a signal, for example. Furthermore, image blur may be detected from a moving image to calculate the vibration of the equipment 4, and the calculated vibration may be determined to be abnormal or not, thereby detecting an abnormal vibration. In these cases, as a method of indicating an abnormality, for example, the color of the entire screen of the user terminal 2 may be changed to make the signal or vibration visible, so that it is possible to know where the abnormality is occurring.

[0040] Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope of achieving the object of the present invention are included in the present invention.

[0041] Furthermore, for example, the above-described series of processes can be executed by hardware or software. In other words, the functional configuration is merely an example and is not particularly limited. That is, it is sufficient that the information processing system has the function of being able to execute the above-described series of processes as a whole, and there is no particular limit to the type of functional block used to realize this function. Furthermore, the location of the functional block is also not particularly limited and may be arbitrary. For example, a functional block of a server may be transferred to a user terminal, etc. Conversely, a functional block of a user terminal may be transferred to a server, etc. Furthermore, one functional block may be configured as a single piece of hardware, a single piece of software, or a combination thereof.

[0042] Furthermore, for example, when a series of processes is executed by software, the programs constituting the software are installed onto a computer or the like from a network or a recording medium. The computer may be a computer incorporated into dedicated hardware. Furthermore, the computer may be a computer capable of executing various functions by installing various programs thereon, such as a server, a general-purpose smartphone, or a personal computer.

[0043] Furthermore, for example, the recording medium containing such a program may be configured not only as a removable medium (not shown) that is distributed separately from the device main body in order to provide the program to the user, but also as a recording medium that is provided to the user in a state that is pre-installed in the device main body.

[0044] In this specification, the steps of describing a program to be recorded on a recording medium include not only processes that are performed chronologically in accordance with the order, but also processes that are not necessarily performed chronologically but are performed in parallel or individually. In addition, in this specification, the term "system" refers to an overall device that is made up of a plurality of devices, a plurality of means, etc. [Explanation of symbols]

[0045] 10 Information Processing Systems 1 server 2. User terminal 131 Acquisition Department 132 segment section 133 specific section 134 Extraction part 135 detection unit

Claims

1. An information processing method executed by an information processing device, a segmentation step of detecting the number of detection targets among the objects included in the video by executing an object extraction algorithm, determining whether at least the detection targets are instruments with analog or digital displays, and segmenting the video for each detection target; an identifying step of identifying the type of the meter that is the detection target using a detection target extraction algorithm for each detection target segmented in the segmenting step; an extraction step of executing a numerical value extraction algorithm for each detection target identified in the identification step to extract a numerical value corresponding to the type of the identified meter; a detection step of detecting an anomaly for each detection target from the numerical values ​​extracted in the extraction step by comparing the numerical values ​​with a deviation from the flow of change over time, a predetermined abnormal value when there is no change in the numerical values ​​over time, or by using a trained model that can be retrained based on a notification from a user terminal; An information processing method including:

2. In the detection step, it is determined at regular intervals whether the object to be detected is abnormal. The information processing method according to claim 1 .

3. The detection target is an instrument installed in a production line facility. The information processing method according to claim 1 .

4. In the information processing device, a segmentation step of detecting the number of detection targets among the objects included in the video by executing an object extraction algorithm, determining whether at least the detection targets are instruments with analog or digital displays, and segmenting the video for each detection target; an identifying step of identifying the type of the meter that is the detection target using a detection target extraction algorithm for each detection target segmented in the segmenting step; an extraction step of executing a numerical value extraction algorithm for each detection target identified in the identification step to extract a numerical value corresponding to the type of the identified meter; a detection step of detecting an anomaly for each detection target from the numerical values ​​extracted in the extraction step by comparing the numerical values ​​with a deviation from the flow of change over time, a predetermined abnormal value when there is no change in the numerical values ​​over time, or by using a trained model that can be retrained based on a notification from a user terminal; A program for executing an information processing method including the steps of:

5. An information processing system executed by an information processing device, a segmentation step of detecting the number of detection targets among the objects included in the video by executing an object extraction algorithm, determining whether at least the detection targets are instruments with analog or digital displays, and segmenting the video for each detection target; an identifying step of identifying the type of the meter that is the detection target using a detection target extraction algorithm for each detection target segmented in the segmenting step; an extraction step of executing a numerical value extraction algorithm for each detection target identified in the identification step to extract a numerical value corresponding to the type of the identified meter; a detection step of detecting an anomaly for each detection target from the numerical values ​​extracted in the extraction step by comparing the numerical values ​​with a deviation from the flow of change over time, a predetermined abnormal value when there is no change in the numerical values ​​over time, or by using a trained model that can be retrained based on a notification from a user terminal; An information processing system that performs the following:

Images