Information processing device, information processing system, customer device, and information processing program

Abstract

An information processing device (100) is provided with: an image data acquisition means for acquiring malfunction representative image data resulting from imaging of a portion of time-series data about sensor values obtained by a sensor device (SD) during malfunction event in a customer apparatus (М0); a countermeasure acquisition means for acquiring countermeasures during the malfunction event; an image data conversion means for converting the portion of the time-series data about the sensor values obtained by the sensor device (SD) of the customer apparatus (M0), which is the subject of malfunction determination, into target image data; an image data determination means for determining the malfunction of the customer apparatus (M0) through comparison between the malfunction representative image data and the target image data; and a countermeasure notification means for, upon determination of the malfunction, notifying the customer apparatus (M0) of countermeasures. The image data conversion means converts first window frames, which are obtained by dividing the time-series data with a time duration of a first time, into a plurality of first target image data items by converting the first window frames into images while going back to a prescribed time at a prescribed first going-back time interval along the time axis of the time-series data.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing system, a customer device, an information processing method, and a non-transitory computer-readable medium. [Background technology]

[0002] Patent Document 1 describes a vibration measuring device for measuring vibrations of an object such as a mechanical facility, etc. The vibration measuring device of Patent Document 1 processes time-series data of the vibrations of the object as a processed image.

[0003] Patent Document 2 describes an analysis device that analyzes the relationships between multiple pieces of time-series data from industrial machinery, etc. The analysis device of Patent Document 2 analyzes the change trends of section data separated by a sliding window width adjustment unit at the separation positions of the time-series data.

[0004] Patent Document 3 describes a failure solution prediction system that generates a solution when an error occurs. The failure solution prediction system of Patent Document 3 generates a solution based on a process performed to deal with a failure.

[0005] Patent Document 4 describes a fault / fault determination device that classifies data obtained from multiple sensors by organizing the data in a time series for each predetermined time unit, and determines faults. The fault / fault determination device of Patent Document 4 classifies data by performing deep learning classification on graph data graphed for each of multiple sensors.

[0006] Patent Document 5 describes an information processing system that performs failure prediction based on history information acquired from equipment on the customer environment side. The information processing system of Patent Document 5 performs failure prediction by analyzing the history information of the equipment acquired from the customer environment side and analyzing the degree of influence of each item of the history information on a failure that occurs in the equipment.

[0007] Patent Document 6 describes an anomaly diagnosis method for diagnosing an anomaly in a facility to be diagnosed. The anomaly diagnosis method of Patent Document 6 diagnoses an anomaly in the operation state of the facility using multivariate time series data for a plurality of measurement items, and when an anomaly in the operation state of the facility is determined, diagnoses the measurement item causing the anomaly from the difference between a feature amount extracted from the multivariate time series data of a first section before the occurrence of the anomaly and a feature amount extracted from the multivariate time series data of a second section after the occurrence of the anomaly. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] International Publication No. 2020 / 255728 [Patent Document 2] Patent Publication No. 2021-076998 [Patent Document 3] JP 2019-153306 A [Patent Document 4] JP 2018-120487 A [Patent Document 5] JP 2017-027124 A [Patent Document 6] Patent No. 6851558 Summary of the Invention [Problem to be solved by the invention]

[0009] There are related techniques for predicting failures that search for similar patterns in data and predict failures etc. when data similar to past failure data is found. However, such techniques may have a fixed cut-out period for pattern extraction. Therefore, unless the range of pattern extraction is expanded, it may not be possible to detect the characteristics of the failure data, making it difficult to predict and determine failures with high accuracy.

[0010] In view of the above-mentioned problems, an object of the present disclosure is to provide an information processing device, an information processing system, a customer device, an information processing method, and a non-transitory computer-readable medium that are capable of determining a fault with high accuracy. [Means for solving the problem]

[0011] An information processing device according to one aspect of the present disclosure includes an image data acquisition means for acquiring failure representative image data that is an image of a part of time series data of sensor values ​​acquired by a sensor device of a customer device when the customer device is broken, a countermeasure acquisition means for acquiring a countermeasure at the time of the failure, an image data conversion means for converting the part of the time series data of the sensor values ​​acquired by the sensor device of the customer device that is to be subjected to a failure judgment into target image data, an image data determination means for determining a failure of the customer device by comparing the failure representative image data with the target image data, and an image data determination means for determining the failure of the customer device when the failure is judged to have occurred, the countermeasure being obtained by converting the part of the time series data of the sensor values ​​acquired by the sensor device of the customer device that is to be subjected to a failure judgment into target image data. and a countermeasure notification means for notifying a customer device, wherein the image data conversion means converts the target image data to include a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data with a time width of a first time, into an image from each first window frame while moving back along the time axis of the time series data to a predetermined time at a predetermined first going back time interval, and a plurality of second target image data pieces obtained by converting a second window frame, which divides the time series data with the time width of a second time larger than the first time, into an image from each second window frame while moving back along the time axis of the time series data to the predetermined time at a predetermined second going back time interval.

[0012] An information processing system according to one aspect of the present disclosure includes a customer device including a sensor device that acquires sensor values, an information processing device connected to the customer device via a network, a failure representative image data storage device that stores failure representative image data that is an image of a portion of time series data of the sensor values ​​acquired by the sensor device of the customer device when the customer device fails, and a countermeasure storage device that stores countermeasures to be taken at the time of the failure, wherein the information processing device includes a failure representative image data acquisition means that acquires the failure representative image data, a countermeasure acquisition means that acquires the countermeasures to be taken at the time of the failure, an image data conversion means that converts a portion of the time series data of the sensor values ​​acquired by the sensor device of the customer device that is to be subject to failure judgment into target image data, and the failure representative image data. The system has an image data determination means for determining a malfunction of the customer device by comparing it with the target image data, and a countermeasure notification means for notifying the customer device of the countermeasure when a malfunction is determined, and the image data conversion means converts the target image data to include a plurality of first target image data obtained by converting a first window frame, which divides the time series data with a time width of a first time, into an image from each first window frame while moving back along the time axis of the time series data to a predetermined time at a predetermined first going back time interval, and a plurality of second target image data obtained by converting a second window frame, which divides the time series data with a time width of a second time larger than the first time, into an image from each second window frame while moving back along the time axis of the time series data to the predetermined time at a predetermined second going back time interval.

[0013] A customer device according to one aspect of the present disclosure includes a sensor device for acquiring sensor values, and a display means for displaying countermeasures to be taken when a failure is determined, and further includes image data acquisition means for acquiring representative failure image data that is an image of a portion of time-series data of the sensor values ​​acquired by the sensor device of the device when the device is broken, a countermeasure acquisition means for acquiring countermeasures to be taken when the device is broken, an image data conversion means for converting the portion of the time-series data of the sensor values ​​acquired by the sensor device of the device that is to be subjected to failure determination into target image data, an image data determination means for determining a failure of the device by comparing the representative failure image data with the target image data, and an image data determination means for displaying the countermeasures to be taken when the device is determined to be broken. and a countermeasure notification means for notifying the device of a countermeasure to be taken. The image data conversion means is connected via a network to an information processing device having the information processing device, and the image data conversion means is connected to the information processing device for converting the target image data into the target image data including: a first window frame that divides the time series data with a time width of a first time, while moving back along the time axis of the time series data to a predetermined time at a predetermined first going back time interval, into a plurality of first target image data pieces that are converted into images from each first window frame; and a second window frame that divides the time series data with the time width of a second time larger than the first time, while moving back along the time axis of the time series data to the predetermined time at a predetermined second going back time interval.

[0014] An information processing method according to one aspect of the present disclosure includes an image data acquisition step of acquiring failure representative image data in which a part of time series data of sensor values ​​acquired by a sensor device of a customer device when the customer device is broken is imaged, a countermeasure acquisition step of acquiring a countermeasure at the time of the failure, an image data conversion step of converting the part of the time series data of the sensor values ​​acquired by the sensor device of the customer device to be subjected to a failure determination into target image data, an image data determination step of determining a failure of the customer device by comparing the failure representative image data with the target image data, and, when the failure is determined, acquiring the countermeasure. and a countermeasure notification step of notifying the customer device of the countermeasure, wherein in the image data conversion step, the target image data is converted to include a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data with a time width of a first time, into an image from each first window frame while going back along the time axis of the time series data to a predetermined time at a predetermined first going back time interval to the predetermined time, and a plurality of second target image data pieces obtained by converting a second window frame, which divides the time series data with the time width of a second time larger than the first time, into an image from each second window frame while going back along the time axis of the time series data to the predetermined time at a predetermined second going back time interval to the predetermined time.

[0015] A non-transitory computer-readable medium according to one aspect of the present disclosure includes an image data acquisition step of acquiring failure representative image data that is an image of a part of time series data of sensor values ​​acquired by a sensor device of a customer device when the customer device is broken; a countermeasure acquisition step of acquiring a countermeasure at the time of the failure; an image data conversion step of converting the part of the time series data of the sensor values ​​acquired by the sensor device of the customer device that is a target for failure determination into target image data; an image data determination step of determining a failure of the customer device by comparing the failure representative image data with the target image data; and a countermeasure notification step of notifying the customer device of the countermeasure when the failure is determined. and a notifying step, the image data conversion step converts the target image data to include a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data with a time width of a first time, into images from each first window frame while moving back along the time axis of the time series data to a predetermined time at a predetermined first going back time interval, and a plurality of second target image data pieces obtained by converting a second window frame, which divides the time series data with the time width of a second time larger than the first time, into images from each second window frame while moving back along the time axis of the time series data to the predetermined time at a predetermined second going back time interval. Effect of the Invention

[0016] The present disclosure makes it possible to provide an information processing device, an information processing system, a customer device, an information processing method, and a non-transitory computer-readable medium that are capable of determining a fault with high accuracy. [Brief description of the drawings]

[0017] [Figure 1] 1 is a configuration diagram illustrating an information processing system according to a first embodiment. [Diagram 2] 1 is a graph showing an image of time-series data of sensor values ​​acquired by a sensor device of a customer apparatus according to the first embodiment, in which the horizontal axis indicates time and the vertical axis indicates intensity. [Diagram 3] 4 is a diagram illustrating an example of failure representative image data stored in a failure representative image data storage device according to the first embodiment. FIG. [Figure 4] 1 is a block diagram illustrating an information processing device according to a first embodiment. [Diagram 5] 4 is a diagram illustrating an example of target image data converted by an image data conversion unit in the information processing device according to the first embodiment. FIG. [Figure 6] 4 is a diagram illustrating an example of target image data converted by an image data conversion unit in the information processing device according to the first embodiment. FIG. [Figure 7] FIG. 2 is a flowchart illustrating an information processing method according to the first embodiment. [Figure 8] FIG. 11 is a configuration diagram illustrating an information processing system according to a second embodiment. [Figure 9] FIG. 11 is a block diagram illustrating an information processing device according to a second embodiment. [Figure 10] 11 is a diagram illustrating an example of the relationship between each component of an information processing system according to a second embodiment and each component of an information processing device. FIG. [Figure 11] 11 is a flowchart illustrating the processing methods of an image data determination unit, a countermeasure notification unit, and an image data vectorization unit in the information processing device according to the second embodiment. FIG. [Figure 12] 13 is a diagram illustrating an example of failure representative image data extracted by an image data extracting unit in the information processing device according to the second embodiment. FIG. [Figure 13] 13 is a diagram illustrating an example of failure representative image data extracted by an image data extracting unit with changing time widths in the information processing device according to the second embodiment. FIG. [Figure 14] FIG. 11 is a flowchart illustrating a method for extracting failure representative image data in the information processing device according to the second embodiment. [Figure 15] FIG. 11 is a configuration diagram illustrating an information processing system according to a third embodiment. [Figure 16] FIG. 11 is a block diagram illustrating an information processing device according to a third embodiment. [Figure 17]FIG. 11 is a flow chart illustrating a method for processing simulated fault image data in the information processing device according to the third embodiment. [Figure 18] FIG. 13 is a block diagram illustrating an example of a customer device to which data is uploaded in an information processing system according to a fourth embodiment. [Figure 19] FIG. 13 is a block diagram illustrating a customer device that is not a target for uploading in an information processing system according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and duplicated descriptions will be omitted as necessary for clarity of explanation.

[0019] (Embodiment 1) An information processing system and an information processing device according to the first embodiment will be described. FIG. 1 is a configuration diagram illustrating an information processing system according to the first embodiment. As shown in FIG. 1, the information processing system 1 includes a customer device M01, etc., an information processing device 100, a representative failure image data storage device 10, and a countermeasure storage device 20. The customer device M01, etc., and the information processing device 100 function as information processing means. The representative failure image data storage device 10 and the countermeasure storage device 20 function as representative failure image data storage means and countermeasure storage means, respectively.

[0020] The information processing system 1 may include customer devices M01, M02, and M03. The customer devices M01, M02, and M03 are collectively referred to as customer device M0. Although three customer devices M0 are shown in the figure, the number of customer devices M0 is not limited to three, and may be one or two, or may be four or more. The customer devices M0 are connected to the information processing device 100 via a network NW in a communicable state.

[0021] The customer device M0 is equipped with various sensor devices SD. The various sensor devices SD monitor the status of each part and each member of the equipped customer device M0. For example, the various sensor devices SD monitor temperature, humidity, voltage, vibration, etc. as the status of each part and each member of the customer device M0.

[0022] FIG. 2 is a graph showing an image of time series data of sensor values ​​acquired by the sensor device SD of the customer device M0 according to the first embodiment, where the horizontal axis indicates time and the vertical axis indicates intensity. As shown in FIG. 2, each of the sensor devices SD of the customer devices M01, M02, and M03 acquires time series data 31 of sensor values ​​of elements A to C. The elements A to C indicate, for example, any of temperature, humidity, voltage, vibration, and the like. In this manner, the customer device M0 includes a sensor device SD that acquires a sensor value. The customer device M0 may include a plurality of sensor devices SD that acquire a plurality of different types of sensor values. The sensor value may include a first sensor value acquired by the first sensor device SD and a second sensor value acquired by a second sensor device SD different from the first sensor device SD. The customer device M0 outputs the time series data 31 of the acquired sensor values ​​to the information processing device 100 via the network NW.

[0023] The network NW may include a wireless communication line or a wired network line. The network NW may combine wireless and wired communication lines. The network NW may constitute a cloud.

[0024] The failure representative image data storage device 10 is connected to the information processing device 100 in a communicable state. FIG. 3 is a diagram illustrating failure representative image data stored in the failure representative image data storage device 10 according to the first embodiment. As shown in FIG. 3, the failure representative image data storage device 10 stores failure representative image data 11 and 12. The failure representative image data 11 and 12 are images of a part of the time series data 31 of the sensor value acquired by the sensor device SD of the customer device M0 when the customer device M0 fails. The failure representative image data 11 and 12 include the time series data 31 at the time of the failure and immediately before the failure.

[0025] The time width of the failure representative image data 12 is, for example, larger than the time width of the failure representative image data 11. For example, the failure representative image data 11 includes a feature 31a as a feature of the time series data 31. On the other hand, the failure representative image data 12 includes a feature 31b in addition to the feature 31a as a feature of the time series data 31. Therefore, when the cause of the failure is in the feature 31b, the failure representative image data 12 can detect the cause of the failure. The failure representative image data storage device 10 outputs the failure representative image data 11 and 12 to the information processing device 100.

[0026] The countermeasure storage device 20 is connected to the information processing device 100 in a communicable state. The countermeasure storage device 20 stores countermeasures when the customer device M0 fails. The countermeasures may include the cause and reason of the failure of the customer device M0, or may include countermeasures used in the past. The countermeasures may include countermeasures for each pattern of the representative failure image data 11 and 12, or may include countermeasures for each of the elements A to C. The countermeasure storage device 20 may store various information when the customer device M0 fails as countermeasures. The countermeasure storage device 20 outputs countermeasures corresponding to the representative failure image data 11 and 12 to the information processing device 100.

[0027] The information processing device 100 is connected to the customer device M0 via a network NW. The information processing device 100 is also connected to a failure representative image data storage device 10. The information processing device 100 is further connected to a countermeasure storage device 20.

[0028] Fig. 4 is a block diagram illustrating an information processing device according to embodiment 1. As shown in Fig. 4, the information processing device 100 includes an image data acquisition unit 110, a countermeasure acquisition unit 120, an image data conversion unit 130, an image data determination unit 140, and a countermeasure notification unit 150. The image data acquisition unit 110, the countermeasure acquisition unit 120, the image data conversion unit 130, the image data determination unit 140, and the countermeasure notification unit 150 each have the functions of an image data acquisition means, a countermeasure acquisition means, an image data conversion means, an image data determination means, and a countermeasure notification means.

[0029] The image data acquisition unit 110 acquires failure representative image data such as failure representative image data 11 and 12 from the failure representative image data storage device 10 .

[0030] The countermeasure acquisition unit 120 acquires countermeasures at the time of failure from the countermeasure storage device 20. The countermeasure acquisition unit 120 may acquire a first countermeasure when a failure is determined in the time series data 31 of the first sensor value. The countermeasure acquisition unit 120 may also acquire a second countermeasure when a failure is determined in the time series data 31 of the second sensor value. Furthermore, the countermeasure acquisition unit 120 may acquire a third countermeasure when a failure is determined in the time series data 31 of the first sensor value and the second sensor value. That is, when a failure is determined in the time series data 31 of the first sensor value and the second sensor value, not only may the first countermeasure and the second countermeasure be acquired in a superimposed manner, but also a third countermeasure different from the first countermeasure and the second countermeasure may be acquired.

[0031] The image data conversion unit 130 converts a part of the time-series data 31 of the sensor value acquired by the sensor device SD of the customer device M0 that is the target of the failure judgment into target image data. Fig. 5 and Fig. 6 are diagrams illustrating examples of the target image data converted by the image data conversion unit 130 in the information processing device 100 according to the first embodiment.

[0032] 5, the image data conversion unit 130 sets a first window frame F1 that divides the time series data into a first time interval. The image data conversion unit 130 moves the set first window frame F1 back along the time axis of the time series data 31 to a predetermined time at a predetermined first retroactive time interval. Then, while moving the first window frame F1 back, the image data conversion unit 130 generates a plurality of first object image data G11-G13 converted into images from each first window frame F1.

[0033] As shown in FIG. 6, the image data conversion unit 130 sets a second window frame F2 that divides the time series data 31 into a second time that is longer than the first time. The image data conversion unit 130 moves the set second window frame F2 back along the time axis of the time series data 31 to a predetermined time at a predetermined second retroactive time interval. Then, the image data conversion unit 130 generates a plurality of second target image data G21-G22 that are converted into images from each second window frame F2 while moving the second window frame F2 back. It is preferable that the second retroactive time interval is longer than the first retroactive time interval. In this way, the image data conversion unit 130 converts a part of the time series data 31 into target image data including the first target image data G11-G13 and the second target image data G21-G22.

[0034] The image data conversion unit 130 may generate multiple target image data by setting multiple windows with different time widths in addition to the first target image data and the second target image data. That is, after performing an operation by taking a second window frame F2 with a larger time width than the first window frame F1, the image data conversion unit 130 may perform an operation by taking a third window frame with a larger time width than the second window frame F2. This operation may be continued. This operation may be terminated when the window width reaches a predetermined maximum window width or more.

[0035] The image data determination unit 140 determines a failure of the customer device M0 by comparing the failure representative image data stored in the failure representative image data storage device 10 with the target image data converted by the image data conversion unit 130. The image data determination unit 140 may compare the target image data obtained by imaging a part of the time series data 31 of the first sensor value with the failure representative image data obtained by imaging a part of the time series data 31 of the first sensor value. The image data determination unit 140 may also compare the target image data obtained by imaging a part of the time series data 31 of the second sensor value with the failure representative image data obtained by imaging a part of the time series data 31 of the second sensor value.

[0036] When the image data determination unit 140 determines that a failure has occurred, the countermeasure notification unit 150 notifies the customer device M0 of the countermeasure stored in the countermeasure storage device 20 via the network NW. For example, when a failure has been determined in the time series data 31 of the first sensor value, the countermeasure notification unit 150 may notify the customer device M0 of a first countermeasure. Furthermore, when a failure has been determined in the time series data 31 of the second sensor value, the countermeasure notification unit 150 may notify the customer device M0 of a second countermeasure. Furthermore, when a failure has been determined in the time series data 31 of the first sensor value and the second sensor value, the countermeasure notification unit 150 may notify the customer device M0 of a third countermeasure. Furthermore, the countermeasure notification unit 150 may notify the customer device M0 of past countermeasures for each similar pattern of the representative failure image data.

[0037] The information processing device 100 described above is, for example, an information processing device such as a server. The information processing device 100 is not limited to a server, and may be a personal computer, a tablet, a mobile terminal, or the like, as long as it performs information processing. The information processing device 100 may include a processor, a memory, a storage device, and a communication device, which are not shown in the figure. The storage device may store the processes performed by each component of the information processing device 100 as a program. The processor may load the program from the storage device into the memory and execute the program. In this way, the processor realizes the functions of the image data acquisition unit 110, the countermeasure acquisition unit 120, the image data conversion unit 130, the image data determination unit 140, and the countermeasure notification unit 150.

[0038] Each of the components of the information processing device 100 may be realized by dedicated hardware. Also, a part or all of each component may be realized by a general-purpose or dedicated circuit, processor, or a combination of these. These may be configured by a single chip, or may be configured by multiple chips connected via a bus. A part or all of each component of each device may be realized by a combination of the above-mentioned circuits, etc. and a program. Also, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (field-programmable gate array), a quantum processor (quantum computer control chip), etc. may be used as the processor.

[0039] Furthermore, when some or all of the components of the information processing device 100 are realized by a plurality of information processing devices, circuits, etc., the plurality of information processing devices, circuits, etc. may be centrally arranged or distributed. For example, the information processing devices, circuits, etc. may be realized as a client-server system, a cloud computing system, etc., in a form in which each is connected via a communication network. Furthermore, the functions of the information processing device 100 may be provided in a SaaS (Software as a Service) format.

[0040] The above-mentioned customer device M0 may also have the same functions as the above-mentioned information processing device 100.

[0041] Next, a description will be given of an information processing method using the information processing device according to embodiment 1. Fig. 7 is a flow chart illustrating the information processing method according to embodiment 1. As shown in Fig. 7, the information processing method includes an image data acquisition step (step S11), a countermeasure acquisition step (step S12), an image data conversion step (step S13), an image data determination step (step S14), and a countermeasure notification step (step S15).

[0042] First, as shown in step S11, in an image data acquisition step, the image data acquisition unit 110 is caused to acquire failure representative image data. Next, as shown in step S12, in a countermeasure acquisition step, the countermeasure acquisition unit 120 is caused to acquire countermeasures at the time of a failure. Note that the order of step S12 is not limited to immediately after step S11, and may be before step S11 or between step S11 and step S15, so long as it is before the countermeasure notification step of step S15.

[0043] Next, as shown in step S13, in an image data conversion step, the image data conversion unit 130 converts a part of the time-series data 31 of the sensor values ​​acquired by the sensor device SD of the customer device M0 that is the target of the failure determination into target image data, as described above, for example, into target image data including at least the first target image data and the second target image data.

[0044] Next, as shown in step S14, in an image data determination step, the image data determination unit 140 compares the representative failure image data with the target image data to determine whether the customer device M0 is faulty. If the image data determination unit 140 determines that there is a failure (YES) in step S14, then in a countermeasure notification step, the countermeasure notification unit 150 notifies the customer device M0 of a countermeasure (NO) in step S14. On the other hand, if the image data determination unit 140 does not determine that there is a failure (NO), the process ends. In that case, the information processing method may be performed on the next time-series data 31 of the sensor value.

[0045] Next, the effects of this embodiment will be described. The information processing device 100 of this embodiment judges a fault by comparing target image data in which the time width of the window frame is changed with fault representative image data. Therefore, it is possible to detect faults and their causes that could not be detected with a fixed window frame. For example, it is possible to detect features 31a and 31b in the time series data 31. This makes it possible to judge a fault in the customer device M0 with high accuracy.

[0046] In addition, the image data conversion unit 130 of the information processing device 100 converts the time-series data 31 of the customer device M0 connected via the network NW into target image data. Therefore, various information can be collected online and a failure can be judged in real time.

[0047] In addition, in the method using machine learning related to failure prediction, it is difficult to organize collected data by type such as temperature and voltage in real time. Therefore, this method notifies the customer of the judgment of whether or not a failure will occur, but it is difficult to automatically suggest how to deal with it specifically, or for the operation side to utilize it in future operations. Therefore, in order to respond by type, it is necessary to exchange emails, etc. each time. In addition, even if the content has been dealt with in the past, it may be necessary to check it again, so it lacks accuracy and immediacy. In this way, in the analysis using machine learning, since the data is integrated and characterized, it is difficult to associate what kind of failure there is and how it was dealt with. Therefore, it is not possible to judge the failure with high accuracy and deal with it appropriately. In contrast, the present embodiment collects various information online and can judge the failure in real time, so that it is possible to deal with the failure quickly.

[0048] When a failure occurs in the customer device M0, in order to respond to the type of sensor value, it was necessary to communicate by e-mail or the like each time, but in this embodiment, it is possible to omit communication by e-mail or the like with the customer device M0, thereby reducing the burden. Also, even in the past, the contents of responses were not properly managed, and it was sometimes necessary to check them again, but in this embodiment, they are stored in the response storage device 20, so it is possible to manage the response measures for the failure.

[0049] In addition, a fault can be determined for each sensor value acquired by the sensor device SD. Therefore, faults that have occurred in the past can be managed according to their causes, and based on that information, the fault can be determined and a countermeasure can be notified to the customer device M0.

[0050] (Embodiment 2) Next, an information processing system and an information processing device according to the second embodiment will be described. Fig. 8 is a configuration diagram illustrating an information processing system according to the second embodiment. As shown in Fig. 8, the information processing system 2 includes customer devices M01, etc., customer devices M11, etc., an information processing device 200, a failure representative image data storage device 10, a countermeasure storage device 20, a customer log data storage device 30, a failure time storage device 40, a reproduction log data storage device 50, a customer image data storage device 60, a failure representative candidate image data storage device 70, and a learning vector data storage device 80. The customer devices M01, etc., customer devices M11, etc., and the information processing device 200 function as information processing means. The failure representative image data storage device 10, the countermeasure storage device 20, the customer log data storage device 30, the failure time storage device 40, the reproduction log data storage device 50, the customer image data storage device 60, the failure representative candidate image data storage device 70, and the learning vector data storage device 80 function as a failure representative image data storage means, a countermeasure storage means, a customer log data storage means, a failure time storage means, a reproduction log data storage means, a customer image data storage means, a failure representative candidate image data storage means, and a learning vector data storage means, respectively. In order to avoid cluttering the diagram, the wording of the storage device in each of the storage devices 10 to 80 has been omitted.

[0051] The information processing system 2 may include customer devices M01, M02, M11, and M12. As in the above embodiment, the customer devices M01 and M02 are called customer devices M0. The customer device M0 is a customer device to be uploaded. The customer devices M11 and M12 are called customer devices M1. The customer device M1 is a customer device not to be uploaded. In the figure, two customer devices M0 and two customer devices M1 are shown, but the number of customer devices M0 and M1 may be one or three or more. The customer devices M0 and M1 are connected to the information processing device 200 side via the network NW in a communicable state.

[0052] Each of the customer devices M0 and M1 is equipped with various sensor devices SD (not shown). As described above, the various sensor devices SD monitor the temperature, humidity, voltage, vibration, etc. as the state of each part and each member of the equipped customer device M0 or M1.

[0053] The customer device M0 that is the upload target uploads various information such as temperature, humidity, voltage, and vibration acquired by the sensor device SD to the information processing device 200 side via the network NW. Therefore, the information processing device 200 acquires various information acquired by the sensor device SD of the customer device M0 via the network NW. On the other hand, the customer device M1 that is not the upload target does not upload at least a part or all of the various information such as temperature, humidity, voltage, and vibration acquired by the sensor device SD to the information processing device 200 side via the network NW. Therefore, the information processing device 200 cannot acquire at least a part or all of the various information acquired by the sensor device SD of the customer device M1 via the network NW. The customer device M1 that is not the upload target is, for example, a local server. The information processing device 200 can acquire the permitted information of the customer device M1 via the network NW. In the following, unless otherwise specified, the permitted information of the customer device M1 will be described as being equivalent to the various information of the customer device M0.

[0054] The customer log data storage device 30 is connected to the customer devices M0 and M1 via the network NW in a communicable state. The customer log data storage device 30 is also connected to the information processing device 200 in a communicable state. The customer log data storage device 30 stores various information of the customer device M0. For example, the customer log data storage device 30 stores time series data 31 of various information acquired from the customer device M0. The customer log data storage device 30 outputs the time series data 31 of various information acquired from the customer device M0 to the information processing device 200. Specifically, the customer log data storage device 30 outputs the time series data 31 of the sensor value acquired by the sensor device SD of the customer device M0 that is the target of the failure determination to the information processing device 200.

[0055] The failure time storage device 40 is connected to the customer log data storage device 30 in a communicable state. The failure time storage device 40 is also connected to the information processing device 200 in a communicable state. The failure time storage device 40 stores the time when the customer device M0 failed. For example, the failure time storage device 40 may obtain the time when the failure occurred from the customer device M0 or from the information processing device 200.

[0056] The reproduction log data storage device 50 is connected in a communicable state to the information processing device 200. The reproduction log data storage device 50 stores the time series data 31 generated by the various sensor devices SD in order to generate learning image data. The reproduction log data storage device 50 stores a sufficient amount of time series data 31 for learning.

[0057] The customer image data storage device 60 is connected to the information processing device 200 in a communicable state. The customer image data storage device 60 stores target image data. The target image data is a part of the time-series data of the sensor value acquired by the sensor device SD of the customer device M0 that is the target of the failure judgment, which is converted into image data. For example, the target image data visualized by the image data conversion unit 130 is stored.

[0058] The failure representative candidate image data storage device 70 is connected to the information processing device 200 in a communicable state. The failure representative candidate image data storage device 70 may also be connected to the failure representative image data storage device 10 in a communicable state. The failure representative candidate image data storage device 70 stores a plurality of failure representative candidate image data. The failure representative candidate image data storage device 70 may include, for example, first failure representative candidate image data and second failure representative candidate image data.

[0059] The first failure representative candidate image data is an image of a window portion divided by a time width of a first time including the first failure and immediately before the first failure in the past time series data 31. The first failure representative candidate image data is extracted as the first failure representative image data when a predetermined condition is met. The predetermined condition is, for example, a case where the first candidate image data is an image of a window portion divided by the same time width of the first time for the first failure representative candidate image data, and there are a predetermined first number or more of first candidate image data having a predetermined first similarity or higher.

[0060] The second failure representative candidate image data is an image of a window portion divided by a time width of a second time including the second failure and immediately before the second failure in the past time series data 31. The second failure representative candidate image data is extracted as the second failure representative image data when a predetermined condition is met. The predetermined condition is, for example, a case where there is a second number or more of second candidate image data, which is an image of a window portion divided by the same time width of the second time for the second failure representative candidate image data, and has a predetermined first similarity or higher.

[0061] The training vector data storage device 80 is connected in a communicable state to the information processing device 200. The training vector data storage device 80 stores training vector data. The training vector data is, for example, vector data used when the image data vectorization unit 160 learns to compress image data into a low-dimensional vector and then restores the compressed image data, as described later.

[0062] Next, the information processing device 200 of the present embodiment will be described. FIG. 9 is a block diagram illustrating the information processing device 200 according to the second embodiment. FIG. 10 is a diagram illustrating the relationship between each configuration of the information processing system 2 according to the second embodiment and each configuration of the information processing device 200. FIG. 10 shows the main relationship between each configuration of the information processing system 2 and each configuration of the information processing device 200, and some relationships between them are omitted so as not to make the figure complicated. As shown in FIGS. 9 and 10, the information processing device 200 of the present embodiment further includes an image data vectorization unit 160, an image data extraction unit 170, and a countermeasure input unit 180 compared to the information processing device 100. The image data vectorization unit 160, the image data extraction unit 170, and the countermeasure input unit 180 have the functions of image data vectorization means, image data extraction means, and countermeasure input means, respectively.

[0063] The image data conversion unit 130 acquires time series data 31 of the sensor values ​​acquired by the sensor device SD of the customer device M0 that is the target of the failure judgment from the customer log data storage device 30. Then, the image data conversion unit 130 converts a part of the acquired time series data 31 into target image data. The image data conversion unit 130 outputs the converted target image data to the customer image data storage device 60.

[0064] The image data conversion unit 130 may generate failure representative candidate image data from the customer log data or the reproduction log data based on the information at the time of failure acquired from the failure time storage device 40. The image data conversion unit 130 outputs the generated failure representative candidate image data to the failure representative candidate image data storage device 70.

[0065] Furthermore, the image data conversion unit 130 acquires time-series data 31 of the sensor values ​​acquired by the sensor device SD of the customer device M0 from the reproduction log data storage device 50 in order to generate learning image data. Then, the image data conversion unit 130 converts a part of the time-series data 31 into target image data. The image data conversion unit 130 outputs the converted target image data to the image data vectorization unit 160.

[0066] The image data determination unit 140 determines whether or not the customer device M0 is malfunctioning by comparing the malfunction representative image data with the target image data, similar to the image data determination unit 140 of the first embodiment. The image data determination unit 140 may determine whether or not the customer device M0 is malfunctioning in real time using the target image data converted by the image data conversion unit 130, or may determine whether or not the customer device M0 is malfunctioning using the target image data stored in the customer image data storage device 60.

[0067] When comparing the failure representative image data with the target image data, the image data determination unit 140 may determine that a failure has occurred in the customer device M0 if the similarity between the two is equal to or greater than a predetermined similarity. The image data determination unit 140 may also determine that no failure has occurred in the customer device M0 if the similarity between the two is less than a predetermined similarity. The similarity between the failure representative image data and the target image data may be calculated, for example, from the inter-vector distance between vector data obtained by vectorizing the failure representative image data and vector data obtained by vectorizing the target image data. For example, the inter-vector distance of the vector data vectorized by the image data vectorization unit 160 may be used.

[0068] Fig. 11 is a flow chart illustrating a processing method of the image data determination unit 140, the countermeasure notification unit 150, and the image data vectorization unit 160 in the information processing device 200 according to the second embodiment. As shown in step S21 of Fig. 11, image data is vectorized. Specifically, the image data vectorization unit 160 vectorizes the failure representative image data and the target image data.

[0069] Next, as shown in step S22, the image data determination unit 140 determines whether the vector distance between the vectorized failure representative image data and the vectorized target image data is smaller than a predetermined threshold. In step S22, if the vector distance is smaller than the predetermined threshold (YES), a failure has occurred in the customer device M0. Therefore, as shown in step S23, the countermeasure notification unit 150 notifies the customer device M0 that a failure has occurred and the countermeasure. On the other hand, in step S22, if the vector distance is equal to or larger than the predetermined threshold (NO), the process ends. In this way, the image data determination unit 140 may determine the failure of the customer device M0 by comparing the failure representative image data and the target image data using the vector distance between them. Note that the similarity between the failure representative image data and the target image data may be calculated by other calculation methods, not limited to calculation from the vector distance.

[0070] The image data vectorization unit 160 is, for example, an autoencoder. The image data vectorization unit 160 vectorizes image data and converts it into vector data. Specifically, for example, the image data vectorization unit 160 vectorizes the failure representative image data and the target image data into vector data. The vectorized data is called vector data. The image data vectorization unit 160 may learn to compress the image data into a low-dimensional vector and then restore it. This allows the image data vectorization unit 160 to quickly vectorize the image data into vector data. Therefore, it is possible to quickly calculate the similarity between the failure representative candidate image data and the candidate image data. The image data vectorization unit 160 outputs the vector data used for learning to the learning vector data storage device 80.

[0071] The image data extraction unit 170 extracts the failure representative image data. FIG. 12 is a diagram illustrating the failure representative image data extracted by the image data extraction unit 170 in the information processing device 200 according to the second embodiment. As shown in FIG. 12, the image data extraction unit 170 first selects one failure representative candidate image data H11 from the failure representative candidate image data storage device 70. The selected failure representative candidate image data is called the first failure representative candidate image data H11. For example, the image data extraction unit 170 selects the first failure representative candidate image data H11 in the time series data 31 of the element A of the customer device M01. The first failure representative candidate image data H11 is obtained by imaging a window frame portion of the time series data 31, which is divided by a time width of the time TT11 including the time T1 and the time immediately before the time T1, by the image data conversion unit 130.

[0072] The image data conversion unit 130 generates a plurality of first candidate image data by imaging a window frame separated by a time width of time TT11 in the time series data 31 of element A of the customer devices M01 to M03. For example, the image data conversion unit 130 generates a plurality of first candidate image data by tracing the window frame back to a predetermined time. The plurality of first candidate image data are referred to as first candidate image data H12, H13, H14, ....

[0073] The image data vectorization unit 160 vectorizes the generated first candidate image data (H12, H13, H14, . . . ) and the first failure representative candidate image data H11.

[0074] The image data determination unit 140 determines the similarity between the first candidate image data (H12, H13, H14, ...) and the first failure representative candidate image data H11 based on the vector distance. The image data determination unit 140 then determines that the similarity between the three first candidate image data H12, H13, and H14 is equal to or greater than a predetermined similarity. In this case, the image data extraction unit 170 counts the number of first candidate image data H12, H13, and H14 similar to the failure representative candidate image data H11 as three.

[0075] The image data extraction unit 170 extracts the first failure representative candidate image data H11 as the first failure representative image data when there are a predetermined number or more of first candidate image data similar to the first failure representative candidate image data at a predetermined similarity or higher. For example, as shown in FIG. 12, the image data extraction unit 170 extracts the first failure representative candidate image data H11 as the first failure representative image data when there are three or more first candidate image data. In this way, the image data extraction unit 170 extracts the first failure representative image data when there are a predetermined number or more of first candidate image data similar to the first failure representative candidate image data, the first candidate image data being an image of a window frame portion separated by a time width of a first time, and having a predetermined first similarity or higher.

[0076] The image data extraction unit 170 may extract the second failure representative image data by changing the time width from time TT11 to time TT21 longer than time TT11. FIG. 13 is a diagram illustrating failure representative image data extracted by the image data extraction unit 170 by changing the time width in the information processing device 200 according to the second embodiment. As shown in FIG. 13, the image data extraction unit 170 selects one failure representative candidate image data H21 in which the time width is changed from time TT11 to time TT21 from the failure representative candidate image data storage device 70. The selected failure representative candidate image data is called the second failure representative candidate image data H21. The second failure representative candidate image data H21 is obtained by imaging a window frame portion of the time series data 31, which is divided by the time width of time TT21 including the time T1 and the time immediately before the time T1, by the image data conversion unit 130.

[0077] Then, similarly to the above-mentioned first failure representative candidate image data, the image data converter 130 generates a plurality of second candidate image data by imaging a window frame divided by the time width of the time TT21 in the time series data 31. For example, the image data converter 130 generates a plurality of second candidate image data by tracing the window frame back to a predetermined time.

[0078] The image data vectorization unit 160 vectorizes the generated second candidate image data and the second failure representative candidate image data H21.

[0079] The image data determination unit 140 determines the similarity between the second candidate image data and the second failure representative candidate image data H21 based on the inter-vector distance.

[0080] The image data extraction unit 170 extracts the second failure representative candidate image data H21 as the second failure representative image data when there are a predetermined number or more of second candidate image data having a predetermined similarity or more in the second failure representative candidate image data H21. In this way, the image data extraction unit 170 extracts the second failure representative image data when there are a predetermined second number or more of second candidate image data having a predetermined second similarity or more, which is the second candidate image data obtained by imaging a window frame portion divided by a time width of the second time for the second failure representative candidate image data. In this way, the image data extraction unit 170 extracts failure representative image data including the first failure representative image data and the second failure representative image data. By extracting failure representative image data having different time widths, the feature 31c of the time series data 31 can be detected.

[0081] The above-mentioned method for extracting the representative failure image data will be described with reference to a flowchart. FIG. 14 is a flowchart illustrating a method for extracting the representative failure image data in the information processing device 200 according to the second embodiment. As shown in step S31 of FIG. 14, the image data conversion unit 130 acquires the time series data 31 from the customer log data storage device 30 of the customer device M0. Next, as shown in step S32, the image data conversion unit 130 sets the time width of the window frame. Next, as shown in step S33, it is determined whether the window frame is within a range up to a predetermined time. In step S33, if the answer is YES that the window frame is within the range up to the predetermined time, the window frame is set at the position going back as shown in step S34. This sets the candidate image data.

[0082] Next, as shown in step S35, the image data vectorization unit 160 vectorizes the window frame portion of the two image data to be compared, the failure representative candidate image data and the candidate image data. Next, as shown in step S36, the vector distance is compared and it is determined whether it is smaller than a predetermined threshold. In step S36, if the vector distance is smaller than the predetermined threshold (YES), the candidate image data is stored in the failure representative candidate image data storage device 70 as shown in step S37. Then, as shown in step S38, it is determined whether there is another element in the same period and the same window frame. On the other hand, in step S36, if the vector distance is equal to or larger than the predetermined threshold (NO), it is determined whether there is another element in the same period and the same window frame.

[0083] If it is determined in step S38 that there is another element in the same period and in the same window frame (YES), the next element (for example, voltage after temperature) is set as shown in step S39. Then, steps S36 to S38 are repeated. On the other hand, if it is determined in step S38 that there is no other element in the same period and in the same window frame (NO), the process proceeds to step S33, where it is determined whether the window frame is within a predetermined time range.

[0084] If the answer in step S33 is NO, meaning that the window frame is outside the range up to the predetermined time, then as shown in step S40, the time width of the window frame is expanded by a predetermined size at the first position in the time series data 31. Then, as shown in step S41, it is determined whether the expanded window frame is equal to or larger than a predetermined maximum window frame size. If the answer in step S41 is NO, meaning that the expanded window frame is smaller than the predetermined maximum window frame size, then the process returns to step S33. On the other hand, if the answer in step S41 is YES, meaning that the expanded window frame is equal to or larger than the predetermined maximum window frame size, then the process ends.

[0085] In step S36, if the number of candidate image data determined to be similar to the failure representative candidate image data is equal to or greater than a predetermined number, the image data extracting section 170 extracts the failure representative candidate image data as the failure representative image data.

[0086] The countermeasure input unit 180 inputs a countermeasure to the countermeasure storage device 20. The countermeasure input unit 180 inputs a countermeasure used at the time of a past failure to the countermeasure storage device 20. For example, the countermeasure input unit 180 inputs a first countermeasure to the countermeasure storage device 20 when a failure was determined in the time series data 31 of the first sensor value in the past. The countermeasure input unit 180 also inputs a second countermeasure to the countermeasure storage device 20 when a failure was determined in the time series data 31 of the second sensor value in the past. Furthermore, the countermeasure input unit 180 inputs a third countermeasure to the countermeasure storage device 20 when a failure was determined in the time series data 31 of the first sensor value and the second sensor value in the past. The third countermeasure may include a countermeasure different from the first countermeasure and the second countermeasure. The countermeasure input unit 180 may also input a past countermeasure for each similar pattern of the representative failure image data.

[0087] Next, the effect of this embodiment will be described. In the information processing device 200 of this embodiment, the image data extraction unit 170 counts candidate image data similar to the failure representative candidate image data. Then, if there is a certain number of similar candidate image data, the image data extraction unit 170 extracts the failure representative candidate image data as the failure representative image data. Therefore, it is possible to extract failure representative image data that is not a sudden change in the sensor value but has a cause. Therefore, it is possible to determine the occurrence of a failure with high accuracy.

[0088] In addition, the failure representative image data includes the first failure representative image data and the first failure representative image data having a time width larger than the time width of the first failure representative image data. Therefore, it is possible to detect the feature 31c of the time series data 31 that cannot be detected in a short time window. This makes it possible to determine the failure of the customer device M0 with high accuracy. Other configurations and effects are included in the description of the first embodiment.

[0089] (Embodiment 3) Next, an information processing system and an information processing device according to a third embodiment will be described. In this embodiment, a failure is judged using pseudo-fault image data. FIG. 15 is a configuration diagram illustrating an information processing system according to the third embodiment. As shown in FIG. 15, the information processing system 3 further includes a pseudo-fault image data storage device 90. The pseudo-fault image data storage device 90 has a function as a pseudo-fault image data storage means. The pseudo-fault image data storage device 90 is connected to the information processing device 300 in a communicable state. The pseudo-fault image data storage device 90 stores the pseudo-fault image data.

[0090] The pseudo-fault image data is an image of a part of the time-series data of the pseudo sensor values ​​formed to show a pseudo-fault state of the customer device M0. For example, by using the pseudo-fault image data manually created from data empirically considered to cause a fault in the customer device M0, it is possible to check whether there is a similar fault in the past. In addition, in order to investigate the cause of the fault in the customer device M0, the relationship between each component and the fault can be investigated by forming pseudo sensor values ​​in which the sensor values ​​of specific components, etc. are pseudo-changed as parameters.

[0091] Fig. 16 is a block diagram illustrating an information processing device according to a third embodiment. As shown in Fig. 16, the information processing device 300 further includes a simulated fault image data acquisition unit 190. The simulated fault image data acquisition unit 190 has a function as a simulated fault image data acquisition means. The simulated fault image data acquisition unit 190 acquires the simulated fault image data stored in the simulated fault image data storage device 90. In this embodiment, the image data determination unit 140 determines a fault in the simulated fault state by comparing the simulated fault image data with the fault representative image data.

[0092] Fig. 17 is a flow chart illustrating a method for processing simulated fault image data in the information processing device 300 according to the third embodiment. As shown in step S51 in Fig. 17, failure representative image data and simulated fault image data are acquired as image data. Specifically, the image data acquisition unit 110 acquires failure representative image data from the failure representative image data storage device 10. In addition, the simulated fault image data acquisition unit 190 acquires simulated fault image data from the simulated fault image data storage device 90.

[0093] Next, as shown in step S52, the image data vectorization unit 160 vectorizes the failure representative image data and the pseudo-failure image data. Next, as shown in step S53, the image data determination unit 140 judges whether the vector distance between the vectorized failure representative image data and the pseudo-failure image data is smaller than a predetermined threshold. In step S53, if the vector distance is smaller than the predetermined threshold (YES), the countermeasure notification unit 150 notifies the customer device M0 that the pseudo-failure state corresponds to a failure and of a countermeasure, as shown in step S54. On the other hand, in step S53, if the vector distance is equal to or larger than the predetermined threshold (NO), the countermeasure notification unit 150 notifies the customer device M0 that the pseudo-failure state does not correspond to a failure and of a countermeasure, as shown in step S54. Then, the process ends.

[0094] Next, the effects of this embodiment will be described. In the information processing device 300 of this embodiment, the image data determination unit 140 determines a failure in the pseudo-failure state by comparing the pseudo-failure image data formed to show the pseudo-failure state of the customer device M0 with the failure representative image data. This makes it possible to confirm where and why the failure occurred in the customer device M0. This can be utilized in future failure countermeasures. Other configurations and effects are included in the description of the first and second embodiments.

[0095] (Embodiment 4) Next, an information processing system and an information processing device according to embodiment 4 will be described. In this embodiment, the customer device M0 that is the upload target and the customer device M1 that is not the upload target will be described in detail. First, the customer device M0 that is the upload target will be described.

[0096] Fig. 18 is a block diagram illustrating an example of a customer device M0 to be uploaded in an information processing system according to the fourth embodiment. As shown in Fig. 18, in the information processing system 4, the customer device M0 is connected to an information processing device 400 via a network NW. The customer device M0 includes one or more sensor devices SD, a display unit DP, and a log data storage unit LD. The sensor device SD, the display unit DP, and the log data storage unit LD each have functions as a sensing means, a display means, and a log data storage means. The information processing device 400 is similar to the information processing devices 100 to 300.

[0097] The sensor device SD acquires a sensor value of the customer device M0. The sensor device SD outputs the acquired sensor value to the log data storage unit LD as various information. The sensor device SD may also output the acquired sensor value as various information to the information processing device 400 via the network NW. The display unit DP displays the countermeasure notified by the countermeasure notification unit 150 of the information processing device 400. The display unit DP displays the countermeasure when a failure is determined. The log data storage unit LD stores log data of the customer device M0. The log data includes various information such as the sensor value acquired by the sensor device SD. The log data stored in the log data storage unit LD may be output to the information processing device 400 via the network NW.

[0098] Next, the non-uploaded customer device M1 will be described. FIG. 19 is a block diagram illustrating a non-uploaded customer device M1 in an information processing system according to the fourth embodiment. As shown in FIG. 19, in the information processing system 4, the customer device M1 is connected to an information processing device 400 via a network NW. The customer device M1 includes one or more sensor devices SD, a display unit DP, and a log data storage unit LD, as well as an information processing unit 400a, a failure representative image data storage unit 10a, and a countermeasure storage unit 20a. The information processing unit 400a has a function as an information processing means. The failure representative image data storage unit 10a and the countermeasure storage unit 20a have a function as a failure representative image data storage unit and a countermeasure storage unit, respectively.

[0099] The failure representative image data storage unit 10a has the same function as the failure representative image data storage device 10. Specifically, the failure representative image data storage unit 10a stores failure representative image data obtained by imaging a part of the time series data 31 of the sensor value acquired by the sensor device of the customer device M0 when the customer device M0 fails. The failure representative image data storage unit 10a may be, for example, a copy of the failure representative image data storage device 10. The failure representative image data storage unit 10a may store failure representative image data of the customer device M1. The countermeasure storage unit 20a may store countermeasures when the customer device M0 fails, or may store countermeasures when the customer device M1 fails.

[0100] The information processing unit 400a has an image data acquisition unit 110, a countermeasure acquisition unit 120, an image data conversion unit 130, and an image data determination unit 140 similar to those installed in the information processing device 400. The image data acquisition unit 110 acquires failure representative image data from the failure representative image data storage unit 10a. The countermeasure acquisition unit 120 acquires countermeasures at the time of failure from the countermeasure storage unit 20a. The image data conversion unit 130 converts a part of the time series data 31 of the customer device M1 stored in the log data storage unit LD into target image data. The image data conversion unit 130 converts into target image data including first target image data and second target image data. The image data determination unit 140 judges a failure of the customer device M1 by comparing the failure representative image data with the target image data. The display unit DP displays the countermeasure acquired by the countermeasure acquisition unit 120 from the countermeasure storage unit 20a when the customer device M1 is judged to be broken.

[0101] Next, the effects of this embodiment will be described. The customer device M0 of this embodiment is connected to the information processing device 400, and outputs various information of the customer device M0 to the information processing device 400. When the information processing device 400 determines that the customer device M0 has failed, the information processing device 400 notifies the customer device M0 of measures to be taken in the event of the failure. This allows the failure to be dealt with promptly.

[0102] Also, in the case where it is not desired to upload various information to the information processing device 400 side, as in the case of the customer device M1, the customer device M1 is provided with a failure representative image data storage unit 10a having the same function as the failure representative image data storage device 10. Also, the customer device M1 is provided with an information processing unit 400a having the same function as the information processing device 400. This allows the customer device M1 to obtain a failure determination result from the built-in failure representative image data storage unit 10a and information processing unit 400a. Other configurations and effects are included in the description of the first to third embodiments.

[0103] (Embodiment 5) Next, a fifth embodiment will be described. This embodiment is an example of a search method that considers combinations of elements. The search for combinations is performed after the search for the representative failure candidate image data for each element is completed.

[0104] First, combinations are searched for in the failure representative candidate image data, starting from element 1. Combinations of up to n elements are considered. n is an arbitrary number and can be set in advance. For example, if the number of elements is five, they are arranged as element 1, element 2, element 3, element 4, element 5. Here, element 1, element 2, ... are assumed to be temperature, humidity, ..., etc. For example, if the number of elements to be considered for combination is two, two combinations out of the five elements are considered. Therefore, in searching in the order of element 1, element 2, element 3, element 4, and element 5, when the element to be considered for combination is set to "1" and the element not to be considered is set to "0", the following 10 combinations are possible.

[0105] 11000, 10100, 10010, ...00011

[0106] This is treated as a binary number, and converted to a decimal number by inverting the numbers so that the numbers are arranged in ascending order, for example, 00110 → 01100. For example, "10100" is converted to a decimal number as "00101", and stored as "5" in the "Designation" column of the management data table below. TIFF0007658466000001.tif80166

[0107] The number of hits (number of similar candidate image data) during a search is stored in the column for each element in the management data table. Combinations are made starting with the smallest element. Numbers are stored in ascending order starting from 0 in the "item" of the management data table, and candidate image data similar to the representative failure candidate image data is stored in the "reference image" column in accordance with the "designation" column. In addition, in order to take combinations into account, data other than image data, such as maximum temperature values, may also be handled.

[0108] The present disclosure is not limited to the above-described embodiments, and may be modified as appropriate without departing from the spirit and scope of the present disclosure. For example, the configurations of the first to fifth embodiments may be combined.

[0109] Also, an information processing program that causes a computer to read and execute the above-mentioned information processing method is within the scope of the technical idea of ​​the embodiment. The information processing program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The information processing program may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or the communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0110] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.

[0111] (Appendix 1) an image data acquisition means for acquiring failure representative image data obtained by imaging a part of time series data of sensor values ​​acquired by a sensor device of the customer device when the customer device fails; A countermeasure acquisition means for acquiring a countermeasure at the time of the failure; an image data conversion means for converting a part of the time series data of the sensor value acquired by the sensor device of the customer device that is a target for failure determination into target image data; an image data determination means for determining a failure of the customer device by comparing the representative failure image data with the target image data; a countermeasure notification means for notifying the customer device of the countermeasure when the customer device is determined to be faulty; Equipped with The image data conversion means a first window frame that divides the time series data into a first time span, and the first window frame is moved back along the time axis of the time series data to a predetermined time at a first going back time interval, and a plurality of first target image data are converted into images from each first window frame; a second window frame that divides the time series data into a second time period that is greater than the first time period, and the second window frame is moved back along the time axis of the time series data to the predetermined time period at a predetermined second retroactive time interval, and a plurality of second target image data are converted into images from each second window frame; converting the target image data into Information processing device. (Appendix 2) Further comprising an image data extraction means for extracting the failure representative image data including the first failure representative image data and the second failure representative image data, The image data extraction means includes: extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first failure representative candidate image data when there are a first number or more of first candidate image data having a first similarity or higher; extracting, from the time-series data, second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure, the second candidate image data obtained by imaging the window portion divided by the time width of the second time, when there are a predetermined second number or more of the second candidate image data having a predetermined second similarity or higher; 2. An information processing device according to claim 1. (Appendix 3) The failure representative image data and the target image data are vectorized into vector data by an image data vectorization unit. the image data determination means determines whether the customer device is faulty by comparing the representative fault image data with the target image data using a vector distance between the representative fault image data and the target image data; 3. The information processing device according to claim 1 or 2. (Appendix 4) The sensor value is a first sensor value acquired by a first sensor device; a second sensor value acquired by a second sensor device different from the first sensor device; Including, The image data determination means comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value; comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value; The countermeasure acquisition means includes: acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value; acquiring a second countermeasure in the case where a fault is determined in the time-series data of the second sensor value; acquiring a third countermeasure in the case where a fault is determined in the time-series data of the first sensor value and the second sensor value; 4. The information processing device according to claim 1, (Appendix 5) a pseudo-fault image data acquisition means for acquiring pseudo-fault image data obtained by imaging a part of the time-series data of the pseudo sensor value formed so as to indicate a pseudo-fault state of the customer device; The image data determination means determining a fault in the pseudo-fault state by comparing the pseudo-fault image data with the fault representative image data; 5. The information processing device according to claim 1. (Appendix 6) a customer device including a sensor device for acquiring a sensor value; an information processing device connected to the customer device via a network; a failure representative image data storage device that stores failure representative image data obtained by imaging a part of the time series data of the sensor value acquired by the sensor device of the customer device when the customer device fails; a countermeasure storage device that stores countermeasures to be taken when the failure occurs; Equipped with The information processing device includes: A failure representative image data acquisition means for acquiring the failure representative image data; A countermeasure acquisition means for acquiring a countermeasure at the time of the failure; an image data conversion means for converting a part of the time series data of the sensor value acquired by the sensor device of the customer device that is a target for failure determination into target image data; an image data determination means for determining a failure of the customer device by comparing the representative failure image data with the target image data; a countermeasure notification means for notifying the customer device of the countermeasure when the customer device is determined to be faulty; having The image data conversion means a first window frame that divides the time series data into a first time span, and the first window frame is moved back along the time axis of the time series data to a predetermined time at a first going back time interval, and a plurality of first target image data are converted into images from each first window frame; a second window frame that divides the time series data into a second time period that is greater than the first time period, and the second window frame is moved back along the time axis of the time series data to the predetermined time period at a predetermined second retroactive time interval, and a plurality of second target image data are converted into images from each second window frame; converting the target image data into Information processing system. (Appendix 7) The information processing device further includes an image data extraction unit for extracting the failure representative image data including first failure representative image data and second failure representative image data, The image data extraction means includes: extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first failure representative candidate image data when there are a first number or more of first candidate image data having a first similarity or higher; extracting, from the time-series data, second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure, the second candidate image data obtained by imaging the window portion divided by the time width of the second time, when there are a predetermined second number or more of the second candidate image data having a predetermined second similarity or higher; 7. An information processing system according to claim 6. (Appendix 8) The image data vectorization means further includes an image data vectorization means for vectorizing the failure representative image data and the target image data into vector data, the image data determination means determines whether the customer device is faulty by comparing the representative fault image data with the target image data using a vector distance between the representative fault image data and the target image data; 8. An information processing system according to claim 6 or 7. (Appendix 9) The sensor value is a first sensor value acquired by a first sensor device; a second sensor value acquired by a second sensor device different from the first sensor device; Including, The image data determination means comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value; comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value; The countermeasure acquisition means includes: acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value; acquiring a second countermeasure in the case where a fault is determined in the time-series data of the second sensor value; acquiring a third countermeasure in the case where a fault is determined in the time-series data of the first sensor value and the second sensor value; An information processing system according to any one of appendix 6 to 8. (Appendix 10) The information processing device further includes a pseudo-fault image data acquisition means for acquiring pseudo-fault image data obtained by imaging a part of the time-series data of the pseudo sensor value formed so as to indicate a pseudo-fault state of the customer device, The image data determination means determining a fault in the pseudo-fault state by comparing the pseudo-fault image data with the fault representative image data; An information processing system according to any one of claims 6 to 9. (Appendix 11) A first customer device including a sensor device for acquiring a sensor value; a second customer device including the sensor device for acquiring the sensor value; an information processing device connected to the first customer device via a network; Equipped with The second customer device is a failure representative image data storage means for storing failure representative image data that is an image of a part of the time series data of the sensor value acquired by the sensor device of the first customer device when the first customer device fails; A countermeasure storage means for storing countermeasures to be taken when the failure occurs; An image data acquisition means for acquiring the failure representative image data; A countermeasure acquisition means for acquiring a countermeasure at the time of the failure; an image data conversion means for converting a part of the time series data of the sensor value acquired by the sensor device of the second customer device that is a target for failure determination into target image data; an image data determination means for determining a failure of the second customer device by comparing the representative failure image data with the target image data; a display means for displaying the countermeasure when the malfunction is determined; having The image data conversion means a first window frame that divides the time series data into a first time span, and the first window frame is moved back along the time axis of the time series data to a predetermined time at a first going back time interval, and a plurality of first target image data are converted into images from each first window frame; a second window frame that divides the time series data into a second time period that is greater than the first time period, and the second window frame is moved back along the time axis of the time series data to the predetermined time period at a predetermined second retroactive time interval, and a plurality of second target image data are converted into images from each second window frame; converting the target image data into Information processing system. (Appendix 12) The second customer device further includes an image data extraction means for extracting the failure representative image data including the first failure representative image data and the second failure representative image data, The image data extraction means includes: extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first failure representative candidate image data when there are a first number or more of first candidate image data having a first similarity or higher; extracting, from the time-series data, second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure, the second candidate image data obtained by imaging the window portion divided by the time width of the second time, when there are a predetermined second number or more of the second candidate image data having a predetermined second similarity or higher; 12. The information processing system according to claim 11. (Appendix 13) The image data vectorization means further includes an image data vectorization means for vectorizing the failure representative image data and the target image data into vector data, the image data determination means determines a failure of the second customer device by comparing the failure representative image data with the target image data using a vector distance between the failure representative image data and the target image data. 13. An information processing system according to claim 11 or 12. (Appendix 14) The sensor value is a first sensor value acquired by a first sensor device; a second sensor value acquired by a second sensor device different from the first sensor device; Including, The image data determination means comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value; comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value; The countermeasure acquisition means includes: acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value; acquiring a second countermeasure in the case where a fault is determined in the time-series data of the second sensor value; acquiring a third countermeasure in the case where a fault is determined in the time-series data of the first sensor value and the second sensor value; An information processing system according to any one of appendixes 11 to 13. (Appendix 15) The second customer device further includes a pseudo-fault image data acquisition means for acquiring pseudo-fault image data obtained by imaging a part of the time-series data of the pseudo sensor value formed so as to indicate a pseudo-fault state of the second customer device, The image data determination means determining a fault in the pseudo-fault state by comparing the pseudo-fault image data with the fault representative image data; An information processing system according to any one of appendixes 11 to 14. (Appendix 16) A sensor device for acquiring a sensor value; a display means for displaying a countermeasure to be taken when a malfunction is determined; Equipped with an image data acquisition means for acquiring failure representative image data obtained by imaging a part of the time series data of the sensor value acquired by the sensor device of the apparatus when the apparatus fails; A countermeasure acquisition means for acquiring a countermeasure at the time of the failure; an image data conversion means for converting a part of the time series data of the sensor values ​​acquired by the sensor device of the apparatus that is the subject of a failure judgment into target image data; an image data determination means for determining a failure of the device by comparing the representative failure image data with the target image data; a countermeasure notification means for notifying the device of the countermeasure when the device is determined to be faulty; via a network to an information processing device having The image data conversion means a first window frame that divides the time series data into a first time span, and the first window frame is moved back along the time axis of the time series data to a predetermined time at a first going back time interval, and a plurality of first target image data are converted into images from each first window frame; a second window frame that divides the time series data into a second time period that is greater than the first time period, and the second window frame is moved back along the time axis of the time series data to the predetermined time period at a predetermined second retroactive time interval, and a plurality of second target image data are converted into images from each second window frame; The information processing device converts the target image data into the target image data including customer equipment. (Appendix 17) The information processing device further includes an image data extraction unit for extracting the failure representative image data including first failure representative image data and second failure representative image data, The image data extraction means includes: extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first failure representative candidate image data when there are a first number or more of first candidate image data having a first similarity or higher; a second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure among the time series data, and a second candidate image data obtained by imaging the window portion divided by the time width of the second time, when the second candidate image data having a predetermined second similarity or higher is a predetermined second number or more, the second failure representative candidate image data is extracted as the second failure representative image data. 17. The customer device of claim 16. (Appendix 18) The image data vectorization means further includes an image data vectorization means for vectorizing the failure representative image data and the target image data into vector data, the image data determination means is connected to the information processing device and determines a failure of the customer device by comparing the representative failure image data with the target image data using a vector distance; 18. A customer device according to claim 16 or 17. (Appendix 19) The sensor value is a first sensor value acquired by a first sensor device; a second sensor value acquired by a second sensor device different from the first sensor device; Including, The image data determination means comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value; comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value; The countermeasure acquisition means includes: acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value; acquiring a second countermeasure in the case where a fault is determined in the time-series data of the second sensor value; acquiring a third countermeasure in the case where a fault is determined in the time-series data of the first sensor value and the second sensor value; A customer device according to any one of appendixes 16 to 18. (Appendix 20) The information processing device further includes a pseudo-fault image data acquisition means for acquiring pseudo-fault image data obtained by imaging a part of the time-series data of the pseudo sensor value formed so as to indicate a pseudo-fault state of the customer device, The image data determination means a device connected to the information processing device that determines a failure in the pseudo-failure state by comparing the pseudo-failure image data with the failure representative image data; 20. A customer device according to any one of appendixes 16 to 19. (Appendix 21) A sensor device for acquiring a sensor value; An information processing means; a display means for displaying a countermeasure to be taken when a malfunction is determined; Equipped with The information processing means includes: an image data acquisition means for acquiring failure representative image data obtained by imaging a part of the time series data of the sensor value acquired by the sensor device of the apparatus when the apparatus fails; a countermeasure acquisition means for acquiring the countermeasure when the failure occurs; an image data conversion means for converting a part of the time series data of the sensor values ​​acquired by the sensor device of the apparatus that is the subject of a failure judgment into target image data; an image data determination means for determining a failure of the device by comparing the representative failure image data with the target image data; having The image data conversion means a first window frame that divides the time series data into a first time span, and the first window frame is moved back along the time axis of the time series data to a predetermined time at a first going back time interval, and a plurality of first target image data are converted into images from each first window frame; a second window frame that divides the time series data into a second time period that is greater than the first time period, and the second window frame is moved back along the time axis of the time series data to the predetermined time period at a predetermined second retroactive time interval, and a plurality of second target image data are converted into images from each second window frame; converting the target image data into customer equipment. (Appendix 22) The information processing means further includes an image data extraction means for extracting the failure representative image data including first failure representative image data and second failure representative image data, The image data extraction means includes: extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first failure representative candidate image data when there are a first number or more of first candidate image data having a first similarity or higher; extracting, from the time-series data, second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure, the second candidate image data obtained by imaging the window portion divided by the time width of the second time, when there are a predetermined second number or more of the second candidate image data having a predetermined second similarity or higher; 22. The customer device of claim 21. (Appendix 23) The image data vectorization means further includes an image data vectorization means for vectorizing the failure representative image data and the target image data into vector data, the image data determination means determines whether the customer device is faulty by comparing the representative fault image data with the target image data using a vector distance between the representative fault image data and the target image data; 23. A customer device according to claim 21 or 22. (Appendix 24) The sensor value is a first sensor value acquired by a first sensor device; a second sensor value acquired by a second sensor device different from the first sensor device; Including, The image data determination means comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value; comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value; The countermeasure acquisition means includes: acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value; acquiring a second countermeasure in the case where a fault is determined in the time-series data of the second sensor value; acquiring a third countermeasure in the case where a fault is determined in the time-series data of the first sensor value and the second sensor value; A customer device according to any one of Supplementary Notes 21 to 23. (Appendix 25) The information processing means further includes a pseudo-fault image data acquisition means for acquiring pseudo-fault image data obtained by imaging a part of the time-series data of the pseudo sensor value formed so as to indicate a pseudo-fault state of the device, The image data determination means determining a fault in the pseudo-fault state by comparing the pseudo-fault image data with the fault representative image data; A customer device according to any one of Supplementary Notes 21 to 24. (Appendix 26) an image data acquisition step of acquiring failure representative image data obtained by imaging a part of time series data of sensor values ​​acquired by a sensor device of the customer device when the customer device is broken; a countermeasure acquisition step of acquiring a countermeasure for the failure; an image data conversion step of converting a part of the time series data of the sensor values ​​acquired by the sensor device of the customer device that is a target for failure determination into target image data; an image data determination step of determining whether or not the customer device has a malfunction by comparing the representative malfunction image data with the target image data; a countermeasure notification step of notifying the customer device of the countermeasure when the fault is determined; Equipped with In the image data conversion step, a first window frame that divides the time series data into a first time span, and the first window frame is moved back along the time axis of the time series data to a predetermined time at a first going back time interval, and a plurality of first target image data are converted into images from each first window frame; a second window frame that divides the time series data into a second time period that is greater than the first time period, and the second window frame is moved back along the time axis of the time series data to the predetermined time period at a predetermined second retroactive time interval, and a plurality of second target image data are converted into images from each second window frame; converting the target image data into the target image data including Information processing methods. (Appendix 27) A failure representative image data extraction step of extracting the failure representative image data including the first failure representative image data and the second failure representative image data, In the failure representative image data extraction step, extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first failure representative candidate image data when there are a first number or more of first candidate image data having a first similarity or higher; extracting, from the time-series data, second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure, the second candidate image data obtained by imaging the window portion divided by the time width of the second time, when there are a second number or more of the second candidate image data having a second similarity or higher; 27. An information processing method according to claim 26. (Appendix 28) The failure representative image data and the target image data are vectorized into vector data, In the image data determination step, a comparison is made between the representative failure image data and the target image data using a vector distance, thereby determining whether or not the customer device has a failure; 28. An information processing method according to claim 26 or 27. (Appendix 29) The sensor value is a first sensor value acquired by a first sensor device; a second sensor value acquired by a second sensor device different from the first sensor device; Including, In the image data determination step, comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value; comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value; In the countermeasure acquisition step, acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value; acquiring a second countermeasure in the case where a fault is determined in the time-series data of the second sensor value; acquiring a third countermeasure in the case where a malfunction is determined in the time-series data of the first sensor value and the second sensor value; An information processing method according to any one of appendix 26 to 28. (Appendix 30) The method further includes a pseudo-fault image data acquisition step of acquiring pseudo-fault image data by imaging the time-series data of the pseudo sensor values ​​formed so as to indicate a pseudo-fault state of the customer device, In the image data determination step, determining a failure of the simulated failure state by comparing the simulated failure image data with the failure representative image data; 30. An information processing method according to any one of appendix 26 to 29. (Appendix 31) an image data acquisition step of acquiring failure representative image data obtained by imaging a part of time series data of sensor values ​​acquired by a sensor device of the customer device when the customer device is broken; a countermeasure acquisition step of acquiring a countermeasure for the failure; an image data conversion step of converting a part of the time series data of the sensor values ​​acquired by the sensor device of the customer device that is a target for failure determination into target image data; an image data determination step of determining whether or not the customer device has a malfunction by comparing the representative malfunction image data with the target image data; a countermeasure notification step of notifying the customer device of the countermeasure when the fault is determined; When you run the following on your computer: In the image data conversion step, a first window frame that divides the time series data into a first time span, and the first window frame is moved back along the time axis of the time series data to a predetermined time at a first going back time interval, and a plurality of first target image data are converted into images from each first window frame; a second window frame that divides the time series data into a second time period that is greater than the first time period, and the second window frame is moved back along the time axis of the time series data to the predetermined time period at a predetermined second retroactive time interval, and a plurality of second target image data are converted into images from each second window frame; converting the target image data into the target image data including A non-transitory computer-readable medium having stored thereon an information processing program that causes a computer to execute the following: (Appendix 32) A failure representative image data extraction step of extracting the failure representative image data including the first failure representative image data and the second failure representative image data is further performed by the computer; In the failure representative image data extraction step, extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first failure representative candidate image data when there are a first number or more of first candidate image data having a first similarity or higher; extracting, from the time-series data, second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure, the second candidate image data obtained by imaging the window portion divided by the time width of the second time, when there are a second number or more of the second candidate image data having a second similarity or higher; A non-transitory computer-readable medium having stored thereon the information processing program described in appendix 31 that causes a computer to execute the steps. (Appendix 33) The computer further executes an image data vectorization step of vectorizing the failure representative image data and the target image data into vector data; In the image data determination step, a comparison is made between the representative failure image data and the target image data using a vector distance, thereby determining whether or not the customer device has a failure; A non-transitory computer-readable medium having stored thereon an information processing program according to claim 31 or 32 that causes a computer to execute the above-mentioned steps. (Appendix 34) The sensor value is a first sensor value acquired by a first sensor device; a second sensor value acquired by a second sensor device different from the first sensor device; Including, In the image data determination step, comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value; comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value; In the countermeasure acquisition step, acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value; acquiring a second countermeasure in the case where a fault is determined in the time-series data of the second sensor value; acquiring a third countermeasure in the case where a malfunction is determined in the time-series data of the first sensor value and the second sensor value; A non-transitory computer-readable medium storing the information processing program according to any one of claims 31 to 33, which causes a computer to execute the above-mentioned steps. (Appendix 35) A pseudo-fault image data acquisition step is further performed by the computer to acquire pseudo-fault image data by imaging the time-series data of the pseudo sensor values ​​formed so as to indicate a pseudo-fault state of the customer device; In the image data determination step, determining a failure of the simulated failure state by comparing the simulated failure image data with the failure representative image data; A non-transitory computer-readable medium storing the information processing program according to any one of claims 31 to 34, which causes a computer to execute the above-mentioned. [Explanation of symbols]

[0112] 1, 2, 3, 4 Information Processing System 10. Fault representative image data storage device 10a Failure representative image data storage unit 11, 12 Representative failure image data 20 Countermeasures Storage Device 20a Countermeasures memory section 30 Customer log data storage device 31 Time Series Data 40 Failure time storage device 50 Reproduced log data storage device 60 Customer image data storage device 70 Fault representative candidate image data storage device 80 Learning Vector Data Storage Device 90 Pseudo-failure image data storage device 100, 200, 300, 400 Information processing device 110 Image data acquisition unit 120 Measures Acquisition Department 130 Image data conversion unit 140 Image data determination unit 150 Countermeasure Notification Department 160 Image Data Vectorization Unit 170 Image data extraction unit 180 Countermeasure input section 190 Pseudo-fault image data acquisition unit 400a Information processing section A, B, C elements DP display F1, F2 window frames G11, G12, G13, G21, G22 Target image data LD Log data storage unit М0, М01, М02, M03 customer equipment М1, М11, М12 customer equipment NW Network SD Sensor Device

Claims

1. an image data acquisition means for acquiring failure representative image data obtained by imaging a part of time series data of sensor values ​​acquired by a sensor device of the customer device when the customer device fails; A countermeasure acquisition means for acquiring a countermeasure at the time of the failure; an image data conversion means for converting a part of the time series data of the sensor value acquired by the sensor device of the customer device that is a target for failure determination into target image data; an image data determination means for determining a failure of the customer device by comparing the representative failure image data with the target image data; a countermeasure notification means for notifying the customer device of the countermeasure when the customer device is determined to be faulty; Equipped with The image data conversion means a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval; a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval; converting the target image data into Information processing device.

2. Further comprising an image data extraction unit for extracting the failure representative image data including the first failure representative image data and the second failure representative image data, The image data extraction means includes: extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first failure representative candidate image data being first candidate image data obtained by imaging the window portion divided by the time width of the first time, when there are a first number or more pieces of first candidate image data having a first similarity or higher; extracting, from the time-series data, second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure, the second candidate image data obtained by imaging the window portion divided by the time width of the second time, when there are a second number or more of the second candidate image data having a second similarity or higher; the second failure representative candidate image data is extracted as the second failure representative image data; The information processing device according to claim 1 .

3. The failure representative image data and the target image data are vectorized into vector data by an image data vectorization unit. the image data determination means determines whether the customer device is faulty by comparing the representative fault image data with the target image data using a vector distance between the representative fault image data and the target image data; 3. The information processing device according to claim 1 or 2.

4. The sensor value is a first sensor value acquired by a first sensor device; a second sensor value acquired by a second sensor device different from the first sensor device; Including, The image data determination means comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value; comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value; The countermeasure acquisition means includes: acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value; acquiring a second countermeasure in the case where a fault is determined in the time-series data of the second sensor value; acquiring a third countermeasure in the case where a fault is determined in the time-series data of the first sensor value and the second sensor value; 3. The information processing device according to claim 1 or 2.

5. a pseudo-fault image data acquisition means for acquiring pseudo-fault image data obtained by imaging a part of the time-series data of the pseudo sensor value formed so as to indicate a pseudo-fault state of the customer device; The image data determination means determining a fault in the pseudo-fault state by comparing the pseudo-fault image data with the fault representative image data; 3. The information processing device according to claim 1 or 2.

6. a customer device including a sensor device for acquiring a sensor value; an information processing device connected to the customer device via a network; a failure representative image data storage device that stores failure representative image data obtained by imaging a part of the time series data of the sensor value acquired by the sensor device of the customer device when the customer device fails; a countermeasure storage device that stores countermeasures to be taken when the failure occurs; Equipped with The information processing device includes: A failure representative image data acquisition means for acquiring the failure representative image data; A countermeasure acquisition means for acquiring a countermeasure at the time of the failure; an image data conversion means for converting a part of the time series data of the sensor value acquired by the sensor device of the customer device that is a target for failure determination into target image data; an image data determination means for determining a failure of the customer device by comparing the representative failure image data with the target image data; a countermeasure notification means for notifying the customer device of the countermeasure when the customer device is determined to be faulty; having The image data conversion means a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval; a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval; converting the target image data into Information processing system.

7. a first customer device including a sensor device for acquiring a sensor value; a second customer device including the sensor device for acquiring the sensor value; an information processing device connected to the first customer device via a network; Equipped with The second customer device is a failure representative image data storage means for storing failure representative image data that is an image of a part of the time series data of the sensor value acquired by the sensor device of the first customer device when the first customer device fails; A countermeasure storage means for storing countermeasures to be taken when the failure occurs; An image data acquisition means for acquiring the failure representative image data; A countermeasure acquisition means for acquiring a countermeasure at the time of the failure; an image data conversion means for converting a part of the time series data of the sensor value acquired by the sensor device of the second customer apparatus that is a target for failure determination into target image data; an image data determination means for determining a failure of the second customer device by comparing the representative failure image data with the target image data; a display means for displaying the countermeasure when the malfunction is determined; having The image data conversion means a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval; a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval; converting the target image data into Information processing system.

8. A sensor device that acquires a sensor value; a display means for displaying a countermeasure to be taken when a malfunction is determined; Equipped with an image data acquisition means for acquiring failure representative image data obtained by imaging a part of the time series data of the sensor value acquired by the sensor device of the apparatus when the apparatus fails; A countermeasure acquisition means for acquiring a countermeasure at the time of the failure; an image data conversion means for converting a part of the time series data of the sensor values ​​acquired by the sensor device of the apparatus that is the subject of a failure determination into target image data; an image data determination means for determining a failure of the device by comparing the representative failure image data with the target image data; a countermeasure notification means for notifying the device of the countermeasure when the device is determined to be faulty; via a network to an information processing device having The image data conversion means a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval; a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval; The information processing device converts the target image data into the target image data including customer equipment.

9. A sensor device that acquires a sensor value; An information processing means; a display means for displaying a countermeasure to be taken when a malfunction is determined; Equipped with The information processing means includes: an image data acquisition means for acquiring failure representative image data obtained by imaging a part of the time series data of the sensor value acquired by the sensor device of the apparatus when the apparatus fails; a countermeasure acquisition means for acquiring the countermeasure when the failure occurs; an image data conversion means for converting a part of the time series data of the sensor values ​​acquired by the sensor device of the apparatus that is the subject of a failure determination into target image data; an image data determination means for determining a failure of the device by comparing the representative failure image data with the target image data; having The image data conversion means a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval; a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval; converting the target image data into customer equipment.

10. an image data acquisition step of acquiring failure representative image data obtained by imaging a part of time series data of sensor values ​​acquired by a sensor device of the customer device when the customer device is broken; a countermeasure acquisition step of acquiring a countermeasure for the failure; an image data conversion step of converting a part of the time series data of the sensor values ​​acquired by the sensor device of the customer device that is a target for failure determination into target image data; an image data determination step of determining whether or not the customer device has a malfunction by comparing the representative malfunction image data with the target image data; a countermeasure notification step of notifying the customer device of the countermeasure when the fault is determined; When you run the following on your computer: In the image data conversion step, a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval; a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time greater than the first time, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval; converting the target image data into the target image data including An information processing program that causes a computer to execute

Images