Abnormality prediction system, method, and computer program

Through the combination of data reconstruction and recursive neural network, the mismatch problem between multiple data time series is solved, the prediction accuracy of the neural network is improved, and efficient abnormal event prediction in mismatch is achieved.

CN119096241APending Publication Date: 2024-12-06NIKKISO CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202380036645.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-28
Filing Date
2023-04-21
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

When using neural networks to predict abnormal events, measurement time mismatch or dimension mismatch between multiple data time series leads to a reduction in prediction accuracy, and it is difficult for the prior art to effectively deal with this situation.

Method used

Through the data reconstruction system, multiple measurement parameters and relative time values ​​are reconstructed into multi-dimensional array data, and the recursive neural network structure is used for calculation to generate input data that can compensate for mismatches to improve prediction accuracy.

Benefits of technology

Even if there is a mismatch between dimensions and time between data time series, multi-dimensional array data that can be processed normally can be generated, thereby improving the prediction accuracy of the neural network.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119096241A_ABST
    Figure CN119096241A_ABST
Patent Text Reader

Abstract

Provided is an abnormality prediction system capable of predicting the occurrence of an abnormal event with high accuracy from a plurality of data time sequences using a neural network, even when mismatch occurs between the plurality of data time sequences obtained by a plurality of measurement instruments. The abnormality prediction system includes: a data reconstruction system configured to reconstruct multi-dimensional array data including a plurality of measurement parameters and a plurality of relative time values respectively allocated to the plurality of measurement parameters as array elements from a plurality of data time sequences obtained by a plurality of measurement devices; and an inference processing section configured to calculate prediction information related to an abnormal event by performing an operation based on a neural network structure having an input layer for receiving the multidimensional array data, an intermediate layer structure including one or more recurrent neural networks, and an output layer.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a technology for predicting the occurrence of abnormal events using a machine learning model, and in particular to a technology for predicting the occurrence of abnormal events based on a plurality of data time series obtained from a plurality of measuring devices using a neural network model based on machine learning. Background Art

[0002] In recent years, a technology for predicting, classifying or optimizing one or more parameters by analyzing input data consisting of one or more parameters using a machine learning method has been provided. For example, in the method of machine learning, decision trees and support vector machines (SVM) have been used in the past, but in recent years, neural network models (Neural Network Model) that simulate the network structure of nerve cells in the brain have also been widely used. The neural network model can also be called an artificial neural network (ANN), or simply a neural network. Most of the neural networks used for machine learning are hierarchical neural networks composed of a combination of multiple processing layers. In general, a hierarchical neural network includes: an input layer for receiving input data, one or more intermediate layers for processing the output data of the input layer, and an output layer for outputting the final processing result based on the output data of the intermediate layer.

[0003] For example, Patent Document 1 (Japanese Patent Publication No. 2021-527906) discloses a technology that can generate a signature matrix representing the system status of multiple time series, and based on the generated signature matrix, use a combination of multiple convolutional neural networks and multiple long short-term memory networks (LSTM networks) to predict the occurrence of abnormal events (see Patent Document 1). Figures 1 to 4 and paragraphs

[0015] to

[0047] ).

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2021-527906 (International Publication No. 2020 / 046806) Summary of the invention

[0007] Technical problem to be solved by the invention

[0008] When using a neural network to predict the occurrence of an abnormal event, it seems that higher prediction accuracy can be expected by predicting the occurrence of an abnormal event based on multiple data time series obtained from multiple different measuring devices related to the system, compared to predicting the occurrence of an abnormal event based on a data time series obtained from a measuring device related to the system.

[0009] However, in the case where a measurement time mismatch or a dimension mismatch occurs between multiple data time series, even if the multiple data time series are integrated through preprocessing to generate input data applied to the NN model, the prediction accuracy may not be improved if there is a mismatch in the generated input data.

[0010] As an example of measurement time mismatch, there is a case where the measurement interval of the first measuring device is several seconds (i.e., the first measuring device provides measurement quantities every several seconds), while the measurement interval of the second measuring device is several minutes (i.e., the second measuring device provides measurement quantities every several minutes). As another example of measurement time mismatch, there is a case where the first measuring device provides measurement quantities at equal time intervals, while the second measuring device provides measurement quantities at unequal time intervals. On the other hand, as an example of dimensional mismatch, there is a case where the first measuring device provides a data time series of two-dimensional measurement quantities [x(t), y(t)], while the second measuring device provides a data time series of one-dimensional measurement quantity z(t). Here, each of x(t), y(t), z(t) is a measurement value (i.e., a scalar), and t is a variable representing a discrete measurement time.

[0011] In view of the above, an object of the present disclosure is to provide an abnormality prediction system, method and computer program that can use a neural network to predict the occurrence of abnormal events with high accuracy even when a mismatch occurs between multiple data time series obtained from multiple measuring devices.

[0012] Technical solutions to the problem

[0013] According to the first aspect of the present disclosure, an abnormality prediction system is an abnormality prediction system for predicting the occurrence of at least one abnormal event based on multiple data time series obtained by multiple measuring devices, and the abnormality prediction system includes: a data reconstruction system, the data reconstruction system is configured to reconstruct multidimensional array data including multiple measurement parameters and multiple relative time values ​​respectively assigned to the multiple measurement parameters as array elements based on the multiple data time series; and an inference operation unit, the inference operation unit is configured to calculate prediction information related to the abnormal event by performing operations based on a neural network structure having an input layer for accepting the multidimensional array data, an intermediate layer structure including one or more recursive neural networks, and an output layer.

[0014] The method according to the second aspect of the present disclosure is a method for predicting the occurrence of at least one abnormal event based on multiple data time series obtained by multiple measuring devices, and the method includes: a step of reconstructing multidimensional array data including multiple measurement parameters and multiple relative time values ​​respectively assigned to the multiple measurement parameters as array elements based on the multiple data time series; and a step of calculating prediction information related to the abnormal event by performing operations based on a neural network structure having an input layer for accepting the multidimensional array data, an intermediate layer structure including one or more recursive neural networks, and an output layer.

[0015] The computer program according to the third aspect of the present disclosure is a computer program read out from a non-volatile memory and executed by one or more processors, and is configured to cause the one or more processors to implement the method according to the second aspect.

[0016] Effects of the Invention

[0017] According to the first to third aspects of the present disclosure, based on multiple data time series obtained by multiple measuring devices, multidimensional array data including multiple measurement parameters and multiple relative time values ​​respectively assigned to the multiple measurement parameters as array elements are reconstructed, and the multidimensional array data is used as input to perform operations based on a neural network structure. As a result, even if a dimension mismatch or a measurement time mismatch occurs between multiple data time series, multidimensional array data having a structure that can compensate for the mismatch through a neural network structure can be generated, or multidimensional array data without mismatch can be generated. Here, the so-called structure that can compensate for the mismatch through a neural network structure refers to a structure that is reconstructed so that even if the neural network structure receives multidimensional array data with mismatch, the multidimensional array data can be processed normally. In such multidimensional array data, not only multiple measurement parameters are included, but also multiple relative time values ​​respectively assigned to (in other words, imprinted on) these measurement parameters are included as time information, so the measurement parameter group that correctly corresponds to the time information can be used as input to perform operations based on the neural network structure. As a result, prediction information can be calculated with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a block diagram schematically showing the configuration of an abnormality prediction system according to one embodiment of the present disclosure.

[0019] Figure 2 This is a diagram showing an example of a case where a mismatch in measurement time occurs between a plurality of data time series.

[0020] Figure 3 This is a diagram showing an example of a case where dimensional mismatch occurs between a plurality of data time series.

[0021] Figure 4 is a block diagram schematically illustrating a neural network structure.

[0022] Figure 5 It is a schematic configuration diagram of an information processing device for realizing the abnormality prediction device involved in the embodiment.

[0023] Figure 6 is a flowchart schematically showing an example of a processing procedure for predicting the occurrence of an abnormal event.

[0024] Figure 7 is a diagram illustrating multidimensional array data having a structure that can compensate for mismatches using a neural network structure.

[0025] Figure 8 is a diagram illustrating multidimensional array data having a structure that can compensate for mismatches using a neural network structure.

[0026] Fig. 9 is a diagram illustrating a neural network structure.

[0027] Fig.10 is a diagram illustrating a neural network structure.

[0028] Fig.11 is a diagram illustrating a neural network structure.

[0029] Fig.12 The diagram shows in tabular form the relationship between the seven measurement parameters actually used in the embodiment and the two-dimensional input vector sequence.

[0030] Fig.13 is a graph showing the results of the example.

[0031] Fig.14 It is a figure which shows the result of Example in a tabular form.

[0032] Fig.15 is a diagram illustrating a neural network structure.

[0033] Fig.16 This is a diagram showing an example of a forward propagation type neural network in a neural network structure.

[0034] Fig.17 A and Fig.17 B shows multi-dimensional array data without both measurement time mismatch and dimension mismatch.

[0035] Fig.18 is a diagram illustrating a neural network structure.

[0036] Fig.19 It is a diagram used to illustrate the method of convolution and nonlinear transformation.

[0037] Fig. 20 is a diagram illustrating a neural network structure.

[0038] Fig.21 is a diagram illustrating a neural network structure.

[0039] Fig. 22 is a diagram illustrating a neural network structure.

[0040] Fig.23 is a diagram illustrating a neural network structure.

[0041] Fig.24 is a diagram illustrating a neural network structure.

[0042] Fig.25 It is a block diagram schematically showing the configuration of an abnormality prediction system according to one embodiment of the present disclosure.

[0043] Fig.26 : is a flowchart schematically showing an example of a processing procedure of loop control.

[0044] Fig. 27 It is a block diagram schematically showing the configuration of an abnormality prediction system according to one embodiment of the present disclosure.

[0045] Fig.28 is a flowchart schematically illustrating an example of a process for predicting the occurrence of an abnormal event.

[0046] Fig.29 It is a block diagram schematically showing the configuration of an abnormality prediction system according to one embodiment of the present disclosure.

[0047] Fig.30 This is a flowchart schematically showing an example of the processing procedure of the loop control. DETAILED DESCRIPTION

[0048] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the components given the same reference numerals throughout the drawings have the same configuration and the same function.

[0049] Figure 1 Schematically shows a block diagram of the structure of an abnormality prediction system 1 according to an embodiment of the present disclosure. Figure 1 As shown, the abnormal prediction system 1 includes: a plurality of devices E 0 ,…,E K-1 , each device is used to observe or process the target object Tgt; and an abnormality prediction device 10, which is configured to be based on the data from these devices E 0 ,…,E K-1Provide multiple data time series D 0 ,…,D K-1 To predict the occurrence of at least one abnormal event. Here, K represents the device E 0 To E K-1 The number of devices E is a positive integer of 3 or more. 0 ,…,E K-1 The number of devices is not limited to three or more, and the structure of the abnormality prediction system 1 may be appropriately changed to include two devices.

[0050] Equipment E 0 ,…,E K-1 Each has a measuring device S 0 ,…,S K-1 Each measuring device S k is configured to physically (eg, optically, electrically or acoustically) measure the k Equipment E k The state of the object Tgt or the state of the target object Tgt, and output the measurement quantity D obtained from the measurement result. k (t) is a time series composed of data time series D k Here, the subscript k is an integer in the range of 0 to K-1, and the English letter t is a variable representing a discrete measurement time. k (t) can be provided as a scalar or a vector. Data time series D k For example, as shown below.

[0051] D k =[D k (t 0 ), D k (t 1 ), D k (t 2 ), …]

[0052] Here, t 0 ,t 1 ,t 2 , ... is a combination of date and time or a measurement time indicating only the time. Regarding the information of the measurement time, the measurement device S n The measured time t and the measured amount D k (t) are transmitted to the abnormality prediction device 10 together, or a timer (not shown) in the abnormality prediction device 10 may be used to count each of the transmitted measured values ​​D k (t) Assign measurement time t.

[0053] Examples of the target object Tgt include a human organism, an animal or plant organism, or an unfinished product or finished product on a production line. 0 To EK-1 For example, it can be a medical device, a manufacturing device, or a monitoring device. 0 ,…,S K-1 As long as there is monitoring equipment E 0 To E K-1 When the device Ek is configured as a medical device such as an artificial dialysis machine, a radiotherapy machine, an artificial heart-lung machine (e.g., extracorporeal membrane oxygenation ECMO: Extracorporeal Membrane Oxygenation), or an electronic sphygmomanometer, the measuring device S in the device Ek is sufficient. k Generates a measurement value D indicating the operating state of the device Ek, such as the amount of change in the current or voltage inside the device Ek, or the blood circuit pressure or the dialysate circuit pressure. k (t), or generate a measurement quantity Dk(t) representing the biological state (vital sign, etc.) of the target object Tgt, such as pulse rate, blood oxygen saturation, or blood pressure. As the measurement quantity Dk(t) representing the biological state, a multidimensional vector composed of the measurement values ​​of the amino acid spectrum of the blood of a person as the target object Tgt (for example, the concentrations of 41 amino acids and metabolic related substances) can also be used. For example, a 41-dimensional vector can be constructed as the measurement quantity Dk(t) based on the measurement values ​​related to the 41 substances in the amino acid spectrum of the blood. k (t).

[0054] It should be noted that if Figure 1 As shown, the measuring device S 0 , S 1 ,…,S K-1 Assembled in equipment E 0 、E 1 ,…,E K-1 In. Figure 1 In the example, a device Ek has a measuring device S k The structure of the control system 1 may be appropriately modified so that one device Ek has a plurality of measuring devices.

[0055] The abnormality prediction device 10 communicates with the device E via a wired transmission path such as a transmission cable or a wireless transmission path. 0 ,…,E K-1 Connect. Device E 0 ,…,E K-1 Having a measuring device S 0 ,…,S K-1 The data time series D 0 ,…,D K-1 The abnormality prediction device 10 has the function of transmitting the abnormality prediction device 10 to the abnormality prediction device 10 via a wired transmission path or a wireless transmission path. 0,…,E K-1 Receive data time series D 0 ,…,D K-1 function.

[0056] like Figure 1 As shown, the abnormality prediction device 10 includes: a data reconstruction system 20, which is configured to 0 ,…,E K-1 Provided data time series D 0 ,…,D K-1 The data reconstruction system 20 comprises: a data input unit 23, which receives and temporarily stores the data time series D. 0 ,…,D K-1 and a data reconstruction unit 25, the data reconstruction unit is configured to input the data time series D via the data input unit 23 0 ,…,D K-1 The multi-dimensional array data is generated. The estimation operation unit 27 can output the calculated prediction information to an external device (not shown).

[0057] The data reconstruction unit 25, even if the data time series D 0 To D K-1 If there is a mismatch in dimension or measurement time between the two, it is possible to generate multidimensional array data having a structure that can compensate for the mismatch by the NN structure 30 of the inference operation unit 27, or to generate multidimensional array data without mismatch. Here, the structure that can compensate for the mismatch by the NN structure 30 refers to the structure that can compensate for the mismatch according to the data time series D 0 To D K-1 The reconstructed structure enables the NN structure 30 to normally process the multidimensional array data even if it receives multidimensional array data with mismatch.

[0058] Figure 2 It means that in the data time series D 0 , D 1 , D 2 , D 3 This is a diagram showing an example of a situation where a mismatch in measurement time occurs between the two. Figure 2 In the example, the data time series D 0 The measured quantities D are arranged at equal time intervals. 0 (t 0 )、D 0 (t 1 )、D 0(t 2 )、D 0 (t 3 )、D 0 (t 4 )、D 0 (t 5 )、D 0 (t 6 ),…. Data time series D 1 The measured quantities D are arranged at equal time intervals. 1 (t 0 )、D 1 (t 2 )、D 1 (t 4 )、D 1 (t 6 ),..., but the data time series D 1 The measurement interval ratio of the data time series D 0 The measurement interval is long. Data time series D 2 The measured quantities D are arranged at equal time intervals. 2 (t 0 )、D 2 (t 1 )、D 2 (t 2 )、D 2 (t 3 )、D 2 (t 4 )、D 2 (t 5 )、D 2 (t 6 ),..., but the data time series D 2 The measurement interval ratio of the data time series D 1 The measurement interval is still long. Data time series D 3 The measured quantities D are arranged at unequal time intervals. 3 (t 0 )、D 3 (t 4 )、D 3 (t 6 ),…. Figure 2 As shown, for example, it can be seen that in the data time series D 0 , D 2 The amount D is missing 0 (t 0 ) corresponds to the measured amount D 2 (t 0 ); in the data time series D 2 , D 3 The amount D is missing2 (t 1 ) corresponds to the measured amount D 3 (t 1 ). Assume that if we directly integrate such a data time series D 0 To D 3 If input data applied to the NN structure 30 is generated, the desired prediction accuracy may not be obtained because the input data contains a measurement time mismatch.

[0059] Figure 3 It means that in the data time series D 0 , D 1 , D 2 An example of a situation where a dimension mismatch occurs between . Figure 3 In the example, the data time series D 0 is determined by the parameter x 0 (t), y 0 (t) is a two-dimensional vector D 0 (t) time series; data time series D 1 is determined by the parameter x 1 (t) is a one-dimensional vector (i.e. scalar) D 1 (t) time series; data time series D 2 is determined by the parameter x 2 (t) is a one-dimensional vector (i.e. scalar) D 2 (t) time series. It can be seen that in the data time series D 0 With D 1 There is a dimension mismatch between them; in the data time series D 0 With D 2 There is a dimensional mismatch between them. Suppose, if we directly integrate such a data time series D 0 To D 2 To generate input data applied to the NN structure 30, the input data may contain a dimension mismatch that is not suitable for the NN structure 30. In this case, even if the operation is performed based on the NN structure 30, the desired prediction accuracy may not be obtained.

[0060] The data reconstruction unit 25 can be based on the data time series D 0 ,…,D K-1 Reconstruct multidimensional array data including a plurality of measurement parameters and relative time values ​​τ respectively assigned to the measurement parameters as array elements. Here, the relative time value τ represented by Greek is a value set based on a common time reference and is a variable representing discrete time. The measurement parameter is a scalar measurement value.

[0061] Each measurement parameter of the array element as a multidimensional array data is from the data time series D0 ,…,D K-1 The measured value of the scalar selected in (e.g., Figure 3 The determination parameter x 0 (t), y 0 (t), x 1 (t) or x 2 (t)), or a scalar value added by interpolation or zero padding. Zero padding means adding zero values. A relative time value τ is assigned to each of these measurement parameters. The multidimensional array data must be accepted by the NN structure 30 of the inference operation unit 27, so it is configured to not include missing array elements. The data reconstruction unit 25 can be constructed by extracting the data time series D 0 ,…,D K-1 Alternatively, the multidimensional array data without missing array elements can be reconstructed by selecting the distribution of measured values ​​and relative time values ​​τ from the data time series D 0 ,…,D K-1 A combination of measured values, interpolation or zero filling, and allocation of relative time value τ is selected and reconstructed to reconstruct multidimensional array data that does not include missing array elements. A specific example of multidimensional array data will be described later.

[0062] Reference Figure 1 The inference operation unit 27 is able to calculate prediction information related to abnormal events by performing operations based on the NN structure 30 using the multidimensional array data input from the data reconstruction unit 25 . Figure 4 is a block diagram schematically showing the NN structure 30. Figure 4 As shown, the NN structure 30 has: an input layer 40, which is used to receive multidimensional array data; an intermediate layer structure 50, which includes one or more recurrent neural networks (RNNs) for processing the multidimensional array data input via the input layer 40; and an output layer 60, which is configured to output prediction information based on the output of the intermediate layer structure 50. The NN structure 30 is a structure that can be trained by supervised learning or unsupervised learning according to a known machine learning algorithm such as stochastic gradient descent (SGD). The inference operation unit 27 can use the learned NN structure 30 to calculate the prediction information. The prediction information is information indicating a group of risk indicating the degree of at least one abnormal event and the predicted time when the abnormal event may occur. The details of the NN structure 30 will be described later.

[0063] All or part of the constituent elements of the above-mentioned abnormal prediction device 10 can be implemented by a computer including one or more processors, or can also be implemented by multiple computers interconnected via a communication path. All or part of the constituent elements of the abnormal prediction device 10 can be implemented by one or more processors, which contain one or more computing devices (Processing Units) that perform processing by the code (instruction group) of software or firmware read out from a non-volatile memory (computer-readable storage medium). For example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit) or an NPU (Neural Network Processing Unit) can be used as a computing device. GPU and NPU are computing devices designed to have a structure suitable for operations (such as tensor operations) based on artificial neural networks (ANN). As an NPU, a TPU (Tensor Processing Unit) developed by Google, a VPU (Vision Processing Unit) developed by Intel, or an IPU (Intelligent Processor Unit) developed by Graphcore can be used. Alternatively, all or part of the constituent elements of the abnormality prediction device 10 may be implemented by one or more processors having semiconductor integrated circuits such as FPGA (Field-Programmable Gate Array). Alternatively, all or part of the constituent elements of the abnormality prediction device 10 may be implemented by one or more processors that are a combination of semiconductor integrated circuits such as FPGA and computing devices such as CPU and GPU.

[0064] Figure 5This is a schematic configuration diagram of an information processing device (computer) 300 as an example of a hardware configuration for realizing the abnormality prediction device 10. The information processing device 300 is configured to include: a processor 301, a random access memory (RAM: Random Access Memory) 302, a non-volatile memory 303, a large-capacity memory 304, an input-output interface 305, and a signal path 306. The signal path 306 is a bus for interconnecting the processor 301, the RAM 302, the non-volatile memory 303, the large-capacity memory 304, and the input-output interface 305. The RAM 302 is a data storage area used when the processor 301 performs digital signal processing. In the case where the processor 301 has a built-in computing device such as a CPU or a GPU, the non-volatile memory 303 has a data storage area for storing the code of the software executed by the processor 301. For example, Figure 1 The data input unit 23 can be realized through the input-output interface 305, Figure 1 The data reconstruction unit 25 and the estimation operation unit 27 can be realized by the processor 301, the RAM 302 and the nonvolatile memory 303. The large-capacity memory 304 can store the prediction information calculated by the processor 301.

[0065] Next, refer to Figure 6 , the processing process of the above-mentioned abnormality prediction device 10 is explained below. Figure 6 This is a flowchart schematically showing an example of a processing procedure for predicting the occurrence of an abnormal event.

[0066] Reference Figure 6 The data reconstruction unit 25 reads the data time series D from the data group temporarily stored in the data input unit 23. 0 ,…,D K-1 (Step S11), according to the data time series D 0 ,…,D K-1 Multidimensional array data including a plurality of measurement parameters and a plurality of relative time values ​​τ respectively assigned to the plurality of measurement parameters as array elements is reconstructed (step S12).

[0067] Next, the inference operation unit 27 calculates prediction information related to the abnormal event by performing operations based on the neural network structure 30 by taking the multidimensional array data as input (step S13).

[0068] After that, when the inference operation unit 27 determines to continue the process (“Yes” in step S20), the process returns to step S11. At this time, the data reconstruction unit 25 reads a new data time series D from a set of data temporarily stored in the data input unit 23. 0 ,…,D K-1(Step S11), according to the new data time series D 0 ,…,D K-1 Reconstruct the multidimensional array data (step S12). Next, the inference operation unit 27 uses the multidimensional array data as input and performs operations based on the neural network structure 30 to calculate the prediction information related to the abnormal event (step S13). In this way, the abnormality prediction device 10 can predict the occurrence of abnormal events one by one and in real time. Finally, when the inference operation unit 27 determines that the processing is not to be continued ("No" in step S20), the processing ends.

[0069] Various examples of multidimensional array data generated by the data reconstruction unit 25 will be described below, and various examples of the NN structure 30 configured to receive and process the multidimensional array data will be described below.

[0070] As described above, the data reconstruction unit 25 can 0 ,…,D K-1 Reconstruct multidimensional array data including a plurality of measurement parameters and relative time values ​​τ respectively assigned to the measurement parameters as array elements. 0 To D K-1 Even if there is a mismatch in dimension or measurement time, the data reconstruction unit 25 can generate multidimensional array data having a structure that can compensate for the mismatch through the NN structure 30 of the inference operation unit 27. Figure 7 and Figure 8 The multi-dimensional array data MD1 and MD2 having a structure in which the mismatch can be compensated by the NN structure 30 are illustrated. Figures 9 to 11 The figures illustrate examples of NN structures configured to process the multidimensional array data MD1 and MD2 respectively while compensating for the mismatch even when the multidimensional array data MD1 and MD2 have the mismatch respectively.

[0071] Figure 7 The multidimensional array data MD1 shown is composed of multiple vector sequences G 0 , G 1 , …, G K-1 The kth vector sequence G k For multiple two-dimensional vectors v k The sequence of (τ) (τ = τ k,0 , τ k,1 , τ k,2 ,…,τ k,N-1 ), each two-dimensional vector v k (τ) has the measurement parameter p k(τ) and the relative time value τ assigned to it as an array element. Here, the subscript k is an arbitrary integer in the range of 0 to K-1, and N is a positive integer representing a variable sequence length (i.e., the number of two-dimensional vectors). Each vector sequence G k With the parameter p for specifying the measurement k (τ k,0 ), p k (τ k,1 ), p k (τ k,2 ),…,p k (τ k,N-1 ) and is used to specify relative time values ​​τ k,0 , τ k,1 , τ k,2 ,…,τ k,N-1 For example, in the data time series D provided from the data input unit 23 0 ,…,D K-1 In the case where each of the data is composed of a scalar time series, the data reconstruction unit 25 can 0 ,…,D K-1 Reconstruct the two-dimensional vector sequence G respectively 0 , G 1 , …, G K-1 .

[0072] on the other hand, Figure 8 The multidimensional array data MD2 shown is composed of multiple vector sequences G 0 , G 1 , …, G K-1 The 0th vector sequence G 0 For multiple three-dimensional vectors v 0 The sequence of (τ) (τ = τ 0,0 , τ 0,1 , τ 0,2 ,…,τ 0,N-1 ), each three-dimensional vector v 0 (τ) has the measurement parameter p 0 (τ),q 0 (τ) and the relative time values ​​τ assigned to them as array elements. The vector sequence G 0 With the parameter p for specifying the measurement 0 (τ 0,0 ), p 0 (τ 0,1 ), p 0 (τ 0,2 ),…,p 0 (τ 0,N-1 ) dimension, used to specify the measurement parameter q 0 (τ 0,0 ),q 0(τ 0,1 ),q 0 (τ 0,2 ),…,q 0 (τ 0,N-1 ) and is used to specify relative time values ​​τ 0,0 , τ 0,1 , τ 0,2 ,…,τ 0,N-1 The dimension of the k-th vector sequence G other than the 0th k For multiple two-dimensional vectors v k The sequence of (τ) (τ = τ k,0 , τ k,1 , τ k,2 ,…,τ k,N-1 ), each two-dimensional vector v k (τ) has the measurement parameter p k (τ) and the relative time value τ assigned to it as an array element. The vector sequence G k With the parameter p for specifying the measurement k (τ k,0 ), p k (τ k,1 ), p k (τ k,2 ),…,p k (τ k,N-1 ) and is used to specify relative time values ​​τ k,0 , τ k,1 , τ k,2 ,…,τ k,N-1 Therefore, the multidimensional array data MD2 is in the 0th vector sequence G 0 and the kth vector sequence G other than the 0th k There is a dimension mismatch between . For example, Figure 3 As shown, in the 0th data time series D 0 The kth data time series D except the 0th one is composed of a two-dimensional vector time series k When the data time series D is composed of scalars, the data reconstruction unit 25 can 0 Reconstruct the three-dimensional vector sequence G 0 , and can be based on the data time series D 1 ,…,D K-1 Reconstruct the two-dimensional vector sequence G respectively 1 , …, G K-1 .

[0073] In addition, Figure 7 and Figure 8 In the example, K vector sequences G 0 , G 1 , …, G K-1The sequence lengths of are all the same value (=N), but are not limited to this. When i and j are set to any mutually different integers in the range of 1 to K (i≠j), the data reconstruction unit 25 can use the i-th vector sequence G i The length of the sequence and the i-th vector sequence G j The multi-dimensional array data MD1 and MD2 are respectively constructed in a manner where the sequence lengths are different from each other.

[0074] Fig. 9 31 is a diagram showing an example of a NN structure 31. Fig. 9 , the NN structure 31 has: an input layer 41, which accepts a vector sequence G 0 , …, G K-1 The multidimensional array data MDx is composed of: an intermediate layer structure 51, which includes a vector sequence G inputted via the input layer 41 and a vector sequence G inputted via the input layer 41. 0 , G 1 , …, G K-1 Recurrent neural network (RNN) 700 for parallel processing, recurrent neural network (RNN) 70 1 , ..., Recurrent Neural Network (RNN) 70 K-1 and an output layer 61, which is configured to output prediction information based on the output of the intermediate layer structure 51. The NN structure 31 can Figure 7 and Figure 8 The multidimensional array data MD1 and MD2 shown are respectively received and processed as multidimensional array data MDx. However, the multidimensional array data that can be processed by the NN structure 31 is not limited to Figure 7 and Figure 8 Data shown.

[0075] The prediction information is information indicating a group (Dr, Tp) of a risk Dr indicating the degree of an abnormal event and a predicted time Tp at which the abnormal event may occur. The risk Dr has a value in the range of approximately 0 to 1. Machine learning of the NN structure 31 can be performed so that the closer the risk Dr is to 0, the lower the degree of the abnormal event, and the closer the risk Dr is to 1, the higher the degree of the abnormal event.

[0076] like Fig. 9 As shown, the intermediate layer structure 51 except RNN70 0 , …, RNN70 K-1 In addition, it further includes: being configured to connect to RNN70 0 , …, RNN70 K-1 The output of the connection layer 75 and the forward propagation neural network 81 configured to process the output of the connection layer 75. Hereinafter, for convenience of explanation, the "forward propagation neural network" will be referred to as "forward propagation NN".

[0077] RNN70 0 To RNN70 K-1 are configured to be suitable for the input vector sequence G 0 To G K-1 The size (vector dimension) of the kth RNN is 70, and has basically the same structure. k Includes: Recursive Layer (RL) 71 k and the final output layer (LO) 72 k , the recursive layer is configured to process the kth vector sequence G input via the input layer 41 k Processing. Recursive layer 71 k It is a neural network layer with a directed closed loop inside, which can transform the input vector sequence G k Extract time-related features. As recursive layer 71 k For example, a Long Short-Term Memory (LSTM) layer, an improved version of the LSTM layer, a Gated Recurrent Unit (GRU) layer, or a Quasi-RNN (QRNN) layer can be used. The configuration of these LSTM layers, improved versions of the LSTM layers, GRU layers, and QRNN layers is disclosed in, for example, the following non-patent document 1.

[0078] ·Non-patent literature 1: Nesma M. Rezk, Madhura Purnaprajna, Tomas Nordstrom, Zain Ul-Abdin: Recurrent Neural Networks: An Embedded Computing Perspective. IEEE Access 8: 57967-57996 (2020).

[0079] Now, if the kth vector sequence G k By N vectors v k (τ 0 ),v k (τ 1 ),v k (τ 2 ),…,v k (τ N-1 ) (N is a positive integer representing the length of a variable sequence), then the recursive layer 71 k is configured to sequentially process N vectors v k (τ 0 ),…,v k (τ N-1 ) is processed to generate N output vectors hk (τ 0 ),…,h k (τ N-1 ) sequence. The final output layer 72 k is configured to select the final output vector h in the sequence of output vectors k (τ N-1 ) and output.

[0080] It should be noted that RNN70 0 To RNN70 K-1 Each of them can be constructed separately to process a sequence of input vectors of any size and generate an output vector of any size. Here, the size of a vector refers to the number of elements in the vector (the vector dimension). For example, the 0th RNN70 0 can be configured to take a sequence of three-dimensional vectors (vectors of size 3) as input to generate a 40-dimensional output vector (vector of size 40), while the first RNN70 1 can be configured to take a sequence of two-dimensional vectors (vectors of size 2) as input to generate a 50-dimensional output vector (vector of size 50).

[0081] Connection layer 75 can connect RNN70 0 , …, RNN70 K-1 The output vector h 0 (τ N-1 ),…,h K-1 (τ N-1 ). For example, in each output vector h k (τ N-1 ) is 40 (i.e., a 40-dimensional vector), the connection layer 75 can connect the output vector h 0 (τ N-1 ),…,h K-1 (τ N-1 ) generates a connection vector of size 40×K (i.e., a 40×K dimensional vector) and outputs it.

[0082] The forward propagation type NN 81 has two sub-neural networks 90 and 100 arranged in parallel with each other. Hereinafter, for the sake of convenience, the "sub-neural network" will be referred to as "sub-NN".

[0083] Sub-NN90 and sub-NN100 are respectively configured to process the connection vector input from the connection layer 75. Sub-NN90 includes: a nonlinear transformation layer (NLL) 91, a linear transformation layer (LL) 92, a nonlinear transformation layer (NLL) 93, a linear transformation layer (LL) 94, a normalization layer (NL) 95, a linear transformation layer (LL) 97, a nonlinear transformation layer (NLL) 98, and a linear transformation layer (LL) 99.

[0084] The nonlinear transformation layer 91 is configured to perform a nonlinear transformation using an activation function on each element of the connection vector input from the connection layer 75. As the activation function, for example, a ramp function or a sigmoid function such as a logistic sigmoid function or a hyperbolic tangent function can be used. The ramp function is also called ReLU (Rectified Linear Unit: linear rectification function).

[0085] The linear transformation layer 92 is configured to perform a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 91. If the weight matrix is ​​represented by W, the bias vector is represented by b, and v is represented by in Denote the input vector, with v out represents the output vector, then in general, the linear transformation can be expressed as follows.

[0086] v out =W·v in +b

[0087] The nonlinear transformation layer 93 is configured to perform a nonlinear transformation using an activation function such as a ramp function on each element of the output vector of the linear transformation layer 92. The linear transformation layer 94 is configured to perform a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 93. The normalization layer 95 can generate a normalized vector by normalizing each element of the vector input from the linear transformation layer 94. As normalization, for example, a known method such as subtractive normalization or divisive normalization can be applied. The linear transformation layer 97 is configured to perform a linear transformation using a weight matrix and a bias vector on the normalized vector. The nonlinear transformation layer 98 is configured to perform a nonlinear transformation using an activation function such as a ramp function on each element of the vector input from the linear transformation layer 97. In addition, the linear transformation layer 99 generates a scalar risk Dr by performing a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 98. The risk degree Dr is output to the outside via the nodes of the output layer 61 .

[0088] The basic structure of sub-NN100 is the same as that of sub-NN90, except for the values ​​of the parameter group (e.g., the elements of the weight matrix and the elements of the bias vector) updated by the machine learning algorithm. Fig. 9As shown, sub-NN100 has: nonlinear transformation layer (NLL) 101, linear transformation layer (LL) 102, nonlinear transformation layer (NLL) 103, linear transformation layer (LL) 104, normalization layer (NL) 105, linear transformation layer (LL) 107, nonlinear transformation layer (NLL) 108, and linear transformation layer (LL) 109.

[0089] The nonlinear transformation layer 101 is configured to perform a nonlinear transformation using an activation function such as a ramp function on each element of the connection vector input from the connection layer 75. The linear transformation layer 102 is configured to perform a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 101. The nonlinear transformation layer 103 is configured to perform a nonlinear transformation using an activation function such as a ramp function on each element of the output vector of the linear transformation layer 102. The linear transformation layer 104 is configured to perform a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 103. The normalization layer 105 can perform normalization on each element of the vector input from the linear transformation layer 104 to generate a normalized vector. The linear transformation layer 107 is configured to perform a linear transformation using a weight matrix and a bias vector on the normalized vector. The nonlinear transformation layer 108 is configured to perform a nonlinear transformation using an activation function such as a ramp function on the vector input from the linear transformation layer 107. The linear transformation layer 109 generates a scalar prediction time Tp by performing a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 108. The prediction time Tp is output to the outside via a node of the output layer 61.

[0090] As explained above, Fig. 9 The NN structure 31 shown has: RNN 70 0 、RNN70 1 , …, RNN70 K-1 , and a connection layer 75, the RNN70 0 、RNN70 1 , …, RNN70 K-1 It is possible to receive the vector sequence G inputted via the input layer 41 in parallel respectively. 0 , G 1 , …, G K-1 And process, this connection layer can connect RNN70 0 、RNN70 1 , …, RNN70 K-1 Therefore, even in the vector sequence G 0 , G 1 , …, G K-1 When there is a mismatch between Figure 8), the NN structure 31 can also compensate for the mismatch, so the inference operation unit 27 can perform a highly reliable operation based on the NN structure 31. k The NN structure 31 is composed of a sequence of multiple vectors, each of which has at least one measurement parameter and a relative time value assigned to the at least one measurement parameter as an array element, so that the NN structure 31 can process the measurement parameter group that correctly corresponds to the time information. As a result, the prediction information (Dr, Tp) can be calculated with high accuracy.

[0091] However, the NN structure 31 is configured to be able to predict the occurrence of one abnormal event, but is not limited to this. The configuration of the NN structure 31 can be appropriately modified to enable the prediction of the occurrence of two or more abnormal events. Fig.10 NN structure 32 is a diagram showing a modified example of NN structure 31. NN structure 32 is configured to be able to predict the occurrence of two types of abnormal events.

[0092] Reference Fig.10 , the NN structure 32 has: an input layer 41, which accepts a vector sequence G 0 , …, G K-1 The middle layer structure 52 includes a vector sequence G inputted via the input layer 41 and a vector sequence G inputted via the input layer 41. 0 , G 1 , …, G K-1 RNN70 for parallel processing 0 、RNN70 1 , …, RNN70 K-1 ; and an output layer 62, the output layer is configured to output two prediction information based on the output of the intermediate layer structure 52. The intermediate layer structure 52 further includes: a layer configured to connect the RNN70 0 , …, RNN70 K-1 The output of the connection layer 75 is connected, and the forward propagation type NN82 is configured to process the output of the connection layer 75.

[0093] The forward propagation type NN82 has two sub-NN90A and sub-NN100A configured in parallel with each other, and both sub-NN90A and sub-NN100A are configured to process the connection vector input from the connection layer 75.

[0094] In addition to the two linear transformation layers (LL) 99 1 , Linear transformation layer (LL) 99 2 Replaced Fig. 9 The structure of sub-NN90A is the same as that of sub-NN90 except for the linear transformation layer 99 of sub-NN90.1 The risk Dr1, which is a scalar quantity indicating the degree of the first abnormal event, is generated by performing a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 98. 2 The risk Dr2, which is a scalar indicating the degree of the second abnormal event, is generated by performing a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 98. The risks Dr1 and Dr2 are output to the outside via the nodes of the output layer 62.

[0095] In addition to the two linear transformation layers (LL) 109 1 , Linear transformation layer (LL) 109 2 Replaced Fig. 9 The structure of sub-NN100A is the same as that of sub-NN100 except for the linear transformation layer 109 of sub-NN100. 1 The predicted time Tp1 at which the first abnormal event may occur is generated by performing a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 108. 2 The predicted time Tp2 at which the second abnormal event may occur is generated by performing linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 108. The predicted times Tp1 and Tp2 are output to the outside via the nodes of the output layer 62.

[0096] It should be noted that in Fig. 9 In the NN structure 31, the structure of sub-NN90 can be changed so that three or more linear transformation layers replace the linear transformation layer 99 of sub-NN90, and the structure of sub-NN100 can be changed so that three or more linear transformation layers replace the linear transformation layer 109 of sub-NN100. In this way, it is possible to construct a NN structure that can predict the occurrence of three or more abnormal events.

[0097] Fig.11 NN structure 33 is a diagram showing another modified example of NN structure 31. NN structure 33 is configured to be able to predict the occurrence of two types of abnormal events.

[0098] Reference Fig.11 , the NN structure 33 has: an input layer 41, which accepts a vector sequence G 0 , …, G K-1 The middle layer structure 53 includes a vector sequence G inputted via the input layer 41 and a vector sequence G inputted via the input layer 41. 0 , G 1 , …, G K-1 RNN70 for parallel processing0 、RNN70 1 , …, RNN70 K-1 ; and an output layer 63, which is configured to output two prediction information based on the output of the intermediate layer structure 53. The intermediate layer structure 53 further includes: a layer configured to connect the RNN70 0 , …, RNN70 K-1 The output of the connection layer 75 is connected, and the forward propagation type NN83 is configured to process the output of the connection layer 75.

[0099] The forward propagation type NN83 has four sub-NN90, sub-NN100, sub-NN110, and sub-NN120 configured in parallel with each other, and each of the sub-NN90, sub-NN100, sub-NN110, and sub-NN120 is configured to process the connection vector input from the connection layer 75.

[0100] Fig.11 The structures of sub-NN90 and sub-NN100 shown are respectively Fig. 9 The structures of the sub-NN90 and sub-NN100 shown are the same. Sub-NN90 can generate a risk Dr1 indicating the degree of the first abnormal event, and sub-NN100 can generate a predicted time Tp1 at which the first abnormal event may occur. The risk Dr1 and the predicted time Tp1 are output to the outside via the nodes of the output layer 63.

[0101] Fig.11 The basic structures of sub-NN110 and sub-NN120 shown are respectively the same as those of sub-NN90 and sub-NN100, except for the values ​​of parameter groups (e.g., elements of the weight matrix and elements of the bias vector) updated by the machine learning algorithm.

[0102] like Fig.11As shown, the sub-NN 110 includes a nonlinear transformation layer (NLL) 111, a linear transformation layer (LL) 112, a nonlinear transformation layer (NLL) 113, a linear transformation layer (LL) 114, a normalization layer (NL) 115, a linear transformation layer (LL) 117, a nonlinear transformation layer (NLL) 118, and a linear transformation layer (LL) 119. The nonlinear transformation layer 111 is configured to perform a nonlinear transformation using an activation function such as a ramp function on each element of the connection vector input from the connection layer 75. The linear transformation layer 112 is configured to perform a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 111. The nonlinear transformation layer 113 is configured to perform a nonlinear transformation using an activation function such as a ramp function on each element of the output vector of the linear transformation layer 112. The linear transformation layer 114 is configured to perform a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 113. The normalization layer 115 can generate a normalized vector by normalizing each element of the vector input from the linear transformation layer 114. The linear transformation layer 117 is configured to perform a linear transformation using a weight matrix and a bias vector on the normalized vector. The nonlinear transformation layer 118 is configured to perform a nonlinear transformation using an activation function such as a ramp function on each element of the vector input from the linear transformation layer 117. In addition, the linear transformation layer 119 generates a risk Dr2 indicating the degree of the second abnormal event by performing a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 118. The risk Dr2 is output to the outside via the node of the output layer 63.

[0103] like Fig.11As shown, the sub-NN 120 includes a nonlinear transformation layer (NLL) 121, a linear transformation layer (LL) 122, a nonlinear transformation layer (NLL) 123, a linear transformation layer (LL) 124, a normalization layer (NL) 125, a linear transformation layer (LL) 127, a nonlinear transformation layer (NLL) 128, and a linear transformation layer (LL) 129. The nonlinear transformation layer 121 is configured to perform a nonlinear transformation using an activation function such as a ramp function on each element of the connection vector input from the connection layer 75. The linear transformation layer 122 is configured to perform a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 121. The nonlinear transformation layer 123 is configured to perform a nonlinear transformation using an activation function such as a ramp function on each element of the output vector of the linear transformation layer 122. The linear transformation layer 124 is configured to perform a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 123. The normalization layer 125 can generate a normalized vector by normalizing each element of the vector input from the linear transformation layer 124. The linear transformation layer 127 is configured to perform a linear transformation using a weight matrix and a bias vector on the normalized vector. The nonlinear transformation layer 128 is configured to perform a nonlinear transformation using an activation function such as a ramp function on each element of the vector input from the linear transformation layer 127. In addition, the linear transformation layer 129 generates a predicted time Tp2 at which a second abnormal event may occur by performing a linear transformation using a weight matrix and a bias vector on the vector input from the nonlinear transformation layer 128. The predicted time Tp2 is output to the outside via the node of the output layer 63.

[0104] It should be noted that in Fig. 9 In the NN structure 31, two or more sub-NN groups having the same basic structure as the group of sub-NN 90 and sub-NN 100 can be added. In this way, it is possible to construct an NN structure that can predict the occurrence of three or more abnormal events.

[0105] Next, refer to Figure 12 to Figure 14 , in the following description based on Fig. 9 An embodiment of the operation of the NN structure 31 is shown.

[0106] This embodiment is performed using a measurement data set obtained by monitoring (monitoring) the biological state of a person undergoing dialysis as the object Tgt through an artificial dialysis machine. The abnormal event that becomes the prediction object is a drop in blood pressure, and the prediction information is a set of the risk Dr of the drop in blood pressure and the predicted time Tp (unit: second) at which the drop in blood pressure may occur. The risk Dr has a value in the range of approximately 0 to 1. The closer the risk Dr is to 0, the lower the degree of drop in blood pressure, and the closer the risk Dr is to 1, the higher the degree of drop in blood pressure.

[0107] Fig.12The seven measurement parameters p actually used in the embodiment are shown in table form. k (τ) and the two-dimensional input vector sequence G k The relationship between the actual blood pressure circuit pressure p 0 (τ), dialysate circuit pressure p 1 (τ), water removal completion amount p 2 (τ), hematocrit value p 3 (τ), pulse rate p 4 (τ), oxygen saturation p 5 (τ) and blood pressure value p 6 (τ) as the monitoring parameter. These measured parameters p 0 (τ) to p 6 (τ) are all scalar quantities. Here, the water removal completion amount p 2 (τ) is the cumulative amount of water removed from the human body, and the hematocrit value p 3 (τ) is the ratio of the volume of blood cell components in the blood and is a value that can be used as an index of blood concentration. 6 (τ) is a value measured using a manchette sphygmomanometer. The manchette is a thin, long, ring-shaped band that is connected to the sphygmomanometer and wrapped around the upper arm of a person during blood pressure measurement.

[0108] Water removal completion amount p 2 (τ) is measured every 60 seconds, and the blood pressure value p 6 (τ) is measured every 30 minutes, and other measurement parameters p 0 (τ), p 1 (τ), p 3 (τ) to p 5 (τ) is measured every 20 seconds. Since the measurement intervals are inconsistent, the vector sequence G 0 To G 6 The multidimensional array data composed of the data has a measurement time mismatch. 0 To RNN70 6 Can accept vector sequences G in parallel separately 0 To G 6 and process it so that the mismatch can be compensated.

[0109] For the learning data and test data, 50 measurement data sets indicating abnormal events (blood pressure drop) were prepared. 1 To A 50 and 125 measurement data sets indicating no abnormal events occurred Φ 1 To Φ 125 The learning data is from 50 measurement data sets Λ 1 To50 40 data sets randomly selected from 1 To 40 and from 125 measured data sets Φ 1 To Φ 125 100 data sets Φ randomly selected from 1 To Φ 100 Using this learning data, the NN structure 31 learns the relationship between the measurement data and the presence or absence of abnormal events and the occurrence time of abnormal events. As test data for verifying the prediction accuracy, 50 measurement data sets Λ 1 To Λ5 0 The remaining 10 data sets and 125 measured data sets Φ 1 To Φ 125 The remaining 25 data sets.

[0110] Specifically, the start time of monitoring is taken as the starting point, and the division time is set in units of 20 minutes (=1200 seconds). The measurement data group of the time interval from the starting point to the division time excluding the time point before the abnormal event occurs is taken as the object of learning. In addition, the measurement data group of the shorter time intervals of only the last 20 minutes and 40 minutes from the time interval is also taken as the object of learning. Therefore, the learning data does not include data after the time point when the abnormal event occurs. The reason is that the purpose is to investigate the occurrence of abnormal events (blood pressure drop) caused by the measurement data before the abnormal event occurs and the predictability of the time of its occurrence. For data where no abnormal events are observed, 0 is assigned as the risk value, and a time point farther than the end point of the monitoring process (=20000 seconds) is uniformly assigned as the occurrence time.

[0111] For the test data, the time interval from the start of monitoring to the time point of 2 hours (=7200 seconds) is set as the first time interval. The measurement data group in multiple time intervals obtained by extending the length in units of 20 minutes (=1200 seconds) from the first time interval is used as the object of test data. The reason for doing this is to achieve the conditions for simulating the increase of measurement data in real time in actual treatment, and at the same time explore the earliest stage that can predict abnormal events. Using this test data, calculations are performed based on the learned NN structure 31. Data groups with a risk Dr exceeding 0.5 are judged to have abnormal events. In addition, for the prediction time Tp, the moment with a value of less than 5 hours of treatment time (=1800 seconds) was used for the first time.

[0112] Fig.13 is a graph showing the results of the example under the above conditions. Fig.13 In the curve graph, the horizontal axis corresponds to the actual time t (unit: second) when the abnormal event occurs, and the vertical axis corresponds to the predicted time τ when the abnormal event may occur. Fig.14It is a diagram showing the results of the embodiment in a tabular form. That is, the number of true positives (Naa) is 9, the number of false positives (Nab) is 1; the number of true negatives (Nbb) is 24, and the number of false negatives (Nba) is 1. Therefore, it is confirmed that the sensitivity (recall rate) is 90% (=Naa / (Naa+Nba)), the specificity is 96% (=Nbb / (Nab+Nbb)), the positive prediction accuracy is 90% (=Naa / (Naa+Nab)), and the negative prediction accuracy is 96% (=Nbb / (Nbb+Nba)). Such good detection performance. Although in Fig.13 Although not shown in the graph of , the predicted time Tp predicts the occurrence of abnormal events more than 20 minutes in advance. Therefore, since it is possible to ensure a time margin for taking measures to prevent a drop in blood pressure, it is considered to have practical advantages.

[0113] As mentioned above, Figures 9 to 11 Each of the NN structures 31, 32, and 33 shown is configured to accept only a plurality of vector sequences G 0 , G 1 , …, G K-1 Alternatively, the configuration of these NN structures may be modified to receive the multidimensional array data MDx and the attribute information of the target object Tgt.

[0114] As the attribute information of the target object Tgt, for example, a scalar or vector consisting of inherent information of a living body such as a human being can be used. More specifically, for example, as the attribute information, numerical values ​​representing age, gender, body mass index (BMI), and the presence or absence of a specific disease such as diabetes can be used. In this case, Figure 1 The data reconstruction unit 25 or the inference calculation unit 27 shown has an internal memory (not shown) that stores attribute information in advance, and provides the attribute information read from the internal memory to the NN structure. Alternatively, the data reconstruction unit 25 or the inference calculation unit 27 may be configured to obtain attribute information from an external information source (not shown) and provide the obtained attribute information to the NN structure.

[0115] Fig.15 It means as Fig. 9 FIG. 31B is a diagram of a modified example of the NN structure 31. The NN structure 31B has an input layer 42 for receiving a vector sequence G 0 , …, G K-1 The multidimensional array data MDx and the attribute information AD of the target object Tgt; the intermediate layer structure 51B, which is configured to receive the vector sequence G inputted via the input layer 42 0 , …, G K-1and attribute information AD; and an output layer 62, which is configured to output two kinds of prediction information based on the output of the intermediate layer structure 51B. The intermediate layer structure 51B includes: a layer configured to process the vector sequence G respectively 0 , G 1 , …, G K-1 RNN70 for parallel processing 0 、RNN70 1 , …, RNN70 K-1 ; Connection layer 75B; and a forward propagation type NN81B composed of sub-NN90B and sub-NN100B configured in parallel with each other.

[0116] The connection layer 75B is configured to connect the RNN 70 0 、RNN70 1 , …, RNN70 K-1 The generated K vectors are connected with the scalar value or vector of the attribute information AD to generate a connection vector. For example, the connection layer 75B can be configured to connect the K vectors respectively obtained from the RNN 70 0 、RNN70 1 , …, RNN70 K-1 The input K 20-dimensional vectors are connected to the 10-dimensional vector of attribute information AD to generate a (K×20+10)-dimensional connection vector.

[0117] Sub-NN90B and sub-NN100B are respectively configured to process the connection vector input from the connection layer 75B. Sub-NN90B has Fig. 9 Sub-NN90B has the same basic structure as sub-NN90 shown. Sub-NN90B has: nonlinear transformation layer (NLL) 91B, linear transformation layer (LL) 92B, nonlinear transformation layer (NLL) 93B, linear transformation layer (LL) 94B, normalization layer (NL) 95B, linear transformation layer (LL) 97B, nonlinear transformation layer (NLL) 98B, and linear transformation layer (LL) 99B. These nonlinear transformation layer 91B, linear transformation layer 92B, nonlinear transformation layer 93B, linear transformation layer 94B, normalization layer 95B, linear transformation layer 97B, nonlinear transformation layer 98B, and linear transformation layer 99B have the same basic structure as sub-NN90 shown. Fig. 9 The nonlinear transformation layer 91, the linear transformation layer 92, the nonlinear transformation layer 93, the linear transformation layer 94, the normalization layer 95, the linear transformation layer 97, the nonlinear transformation layer 98, and the linear transformation layer 99 in the sub-NN 90 shown have the same basic structure. The sub-NN 90B generates a scalar risk Dr, and the risk Dr is output to the outside via the nodes of the output layer 61.

[0118] On the other hand, the sub-NN100B also has Fig. 9The sub-NN100B has the same basic structure as the sub-NN100 shown. The sub-NN100B has: a nonlinear transformation layer (NLL) 101B, a linear transformation layer (LL) 102B, a nonlinear transformation layer (NLL) 103B, a linear transformation layer (LL) 104B, a normalization layer (NL) 105B, a linear transformation layer (LL) 107B, a nonlinear transformation layer (NLL) 108B, and a linear transformation layer (LL) 109B. These nonlinear transformation layers 101B, linear transformation layers 102B, nonlinear transformation layers 103B, linear transformation layers 104B, normalization layers 105B, linear transformation layers 107B, nonlinear transformation layers 108B, and linear transformation layers 109B have the same basic structure as the sub-NN100 shown. Fig. 9 The sub-NN100 shown has the same basic structure as the nonlinear transformation layer 101, the linear transformation layer 102, the nonlinear transformation layer 103, the linear transformation layer 104, the normalization layer 105, the linear transformation layer 107, the nonlinear transformation layer 108, and the linear transformation layer 109. The sub-NN100B generates a scalar prediction time Tp, and the prediction time Tp is output to the outside via the node of the output layer 61.

[0119] As explained above, Fig.15 In addition to being able to accept multidimensional array data MDx having measurement parameters and relative time values ​​assigned to these measurement parameters as array elements, the NN structure 31B shown can also accept and process attribute information AD of the object object Tgt. Therefore, the inference operation unit 27 can calculate the prediction information (Dr, Tp) related to the object object Tgt with higher accuracy based on the NN structure 31B.

[0120] Figures 9 to 11 and Fig.15 The structures of the forward propagation type NN81, forward propagation type NN82, forward propagation type NN83, and forward propagation type NN81B shown are examples and are not limited to these. Fig.16 The forward propagation type NN81C shown is used instead Fig. 9 The forward propagation type NN81 is shown. Fig.16 The forward propagation type NN81C shown has two sub-NN90C and sub-NN100C. Sub-NN90C is deleted by Fig. 9 The nonlinear transformation layer 91 in the sub-NN 90 is replaced by a nonlinear transformation layer (NLL) 96 between the normalization layer 95 and the linear transformation layer 97. On the other hand, the sub-NN 100C is constructed by deleting Fig. 9 The nonlinear transformation layer 101 in the sub-NN100 is configured with a nonlinear transformation layer (NLL) 106 arranged between the normalization layer 105 and the nonlinear transformation layer 107.

[0121] in addition, Figures 9 to 11 and Fig.15 Each of the sub-NN90, sub-NN100, sub-NN90A, sub-NN100A, sub-NN110, sub-NN120, sub-NN90B, and sub-NN100B shown also has a combination of a linear transformation layer, a normalization layer, and a linear transformation layer connected in series with each other (e.g., Fig. 9 94, the normalization layer 95, and the linear transformation layer 97 shown in FIG. 1 ). A combination of a linear transformation layer and a normalization layer obtained by deleting a linear transformation layer at a subsequent stage of the normalization layer from such a combination may be used (for example, a combination obtained by removing a linear transformation layer at a subsequent stage of the normalization layer from the combination may also be used). Fig. 9 A combination of the linear transformation layer 94 and the normalization layer 95 is obtained by deleting the linear transformation layer 97 from the combination of the linear transformation layer 94, the normalization layer 95 and the linear transformation layer 97 shown in FIG.

[0122] Next, an example of multidimensional array data without mismatch is described, and various examples of NN structures configured to be able to accept and process multidimensional array data without mismatch are described.

[0123] Fig.17 A and Fig.17 B is a diagram illustrating multi-dimensional array data MD3 without any mismatch. Fig.18 and Figure 20 to Figure 24 The figures are diagrams illustrating NN structures configured to receive and process multidimensional array data MD3.

[0124] Fig.17 The multidimensional array data MD3 shown in A is composed of a third-order tensor T1. The multidimensional array data MD3 has a measurement parameter p j (τ i ) and the relative time value τ assigned to it i As the array element of tensor T1. Here, subscript j is an integer in the range of 0 to J-1, J is a positive integer of 2 or more, subscript i is an integer in the range of 0 to N-1, and N is a variable positive integer. Fig.17 As shown in B, the multidimensional array data MD3 has a size of N×J×2. For example, Figure 1 The data reconstruction unit 25 shown in FIG. 1 can reconstruct the measurement parameter p having the subscript j from the scalar (measurement value) sequence constituting the j-th data time series Dj. j (τ i ), but is not limited to this. In addition, in the jth data time series D j In the case of a sequence of multidimensional vectors (e.g. Figure 3 The data time series D shown 0 In the case of , the data reconstruction unit 25 decomposes the multidimensional vector sequence into a plurality of scalar sequences, and can obtain the measurement parameters of the tensor T1 based on each scalar sequence.i When a measurement parameter is missing, the data reconstruction unit 25 may supplement the missing measurement parameter by interpolation or zero filling.

[0125] It should be noted that the size of the multidimensional array data composed of tensors is not limited to N×J×2, but can be N×J×(H c +1). Here, H c is an integer of 1 or more. For example, when receiving a time series supply of the measurement amount obtained by each of the plurality of color sensors, the data reconstruction unit 25 can reconstruct a data structure having N×J×(H c +1) size (H c is the number of color sensors).

[0126] Fig.18 : is a diagram showing an NN structure 34 as an example. The NN structure 34 has: an input layer 44, which is used to receive multidimensional array data MDy composed of third-order tensors; an intermediate layer structure 54, which is configured to process the multidimensional array data MDy input via the input layer 44; and an output layer 64, which is configured to output prediction information (Dr, Tp) based on the output of the intermediate layer structure 54. The intermediate layer structure 54 includes: a convolutional neural network (CNN) 130, a convolutional neural network 131 and a recurrent neural network (RNN) 132. The NN structure 34 is configured to be able to process the multidimensional array data MDy composed of third-order tensors. Fig.17 The multi-dimensional array data MD3 composed of the third-order tensor T1 shown is received and processed as multi-dimensional array data MDy.

[0127] The intermediate layer structure 54 further includes, in addition to the CNN 130, the CNN 131, and the RNN 132, a forward propagation type NN 133 configured to process the output of the RNN 132. The CNN 130, the CNN 131, the RNN 132, and the forward propagation type NN 133 are connected in series.

[0128] CNN130 has a convolution layer (CL) 141, a nonlinear transformation layer (NLL) 142, and a pooling layer (PL) 143. The convolution layer 141 is configured to perform convolution using M filters on the input third-order tensor data to generate second-order or third-order tensor data. Here, M is an integer of 1 or more representing the number of filters, that is, the number of channels. Convolution consists of a convolution operation using a filter and an addition operation using a bias. When one filter is used (when M=1), the convolution layer 141 can output second-order tensor (i.e., matrix) data, and when multiple filters are used (when M is 2 or more), the convolution layer 141 can output third-order tensor data. The nonlinear transformation layer 142 is configured to perform a nonlinear transformation using an activation function on each element of the tensor input from the convolution layer 141. As an activation function, for example, a ramp function, or a sigmoid function such as a logical sigmoid function or a hyperbolic tangent function can be used.

[0129] Fig.19 It is a diagram used to illustrate the method of convolution and nonlinear transformation. Fig.19 shows the input tensor Ty, the filter F 0 、F 1 ,…,F M-1 , Convolution operator 210 0 , 210 1 , ..., 210M-1, addition operator 211 0 , 211 1 , …, 211 M-1 , activation function f 0 (), f 1 (),…,f M-1 (), and output tensor Tz. The input tensor Ty is a tensor of size N×J×H. Here, N and J are positive integers of 2 or more, and H is an integer of 1 or more. When H=1, the input tensor Ty is a second-order tensor (matrix), and when the value of H is 2 or more, the input tensor Ty is a third-order tensor. Fig.18 The convolution layer 141 of the CNN 130 shown is configured to convolve a third-order tensor as input. The elements of the input tensor Ty can be represented as dijh. Here, the subscripts i and j are integers for specifying a square area specified in the input tensor Ty, and the subscript h is an integer in the range of 0 to H-1. M is a filter F 0 To F M-1 The number of channels is an integer of 3 or more. Fig.19 In the example, the number of channels M is three or more, but is not limited to this.

[0130] The filter F of the mth channel mIt is composed of a tensor with a size of P×Q×H. Here, P is a positive integer of 2 or more that is less than N, and Q is an integer of 2 or more that is less than J. When H=1, the filter F m It is composed of a second-order tensor (matrix). When the value of H is 2 or above, the filter F m It is composed of a third-order tensor. Fig.18 The convolution layer 141 of the CNN 130 is shown to be configured to perform convolution using at least one filter composed of a third-order tensor. m The element (weighting factor) can be expressed as w pqhm Here, subscript p is an integer in the range of 0 to P-1, subscript q is an integer in the range of 0 to Q-1, and h is an integer in the range of 0 to H-1.

[0131] Reference Fig.19 , convolution operator 210 0 , 210 1 , …, 210 M-1 The elements d of the input tensor Ty can be processed in parallel ijh Perform convolution operation. Addition operator 211 0 , 211 1 , …, 211 M-1 can be performed in parallel to bias b ij0 、b ij1 , …, b i、j、M-1 Respectively with the convolution operator 210 0 , 210 1 , …, 210 M-1 The output of the mth channel is added. ijm Provided as a second-order tensor (matrix).

[0132] Specifically, for example, convolution of each channel may be performed according to the following formula.

[0133] [Formula 1]

[0134]

[0135] After convolution is performed, as shown in the following formula, the activation function f is used on each channel m () is a nonlinear transformation.

[0136] [Formula 2]

[0137] z ijm =f m (u ijm )

[0138] By integrating the elements z computed through the nonlinear transformation of all channels ijm To form the output tensor Tz. Fig.19 As shown, the output tensor Tz has a size of R×L×M. Here, R is a positive integer of N or less, and L is a positive integer of J or less. In the case of a single channel (a case where one filter is used), the output tensor Tz is composed of a second-order tensor (matrix), and in the case of multiple channels (a case where multiple filters are used), the output tensor Tz is composed of a third-order tensor.

[0139] Reference Fig.18 The pooling layer 143 of CNN 130 is configured to perform a pooling operation on the data of the tensor input from the nonlinear transformation layer 142 according to a preset stride value to generate an output tensor. As the pooling process, a well-known method such as average pooling, maximum pooling, or Lp pooling can be applied.

[0140] CNN131 has a convolution layer (CL) 145, a nonlinear transformation layer (NLL) 146, and a pooling layer (PL) 147. The convolution layer 145 is configured to perform convolution using one or more filters (i.e., convolution operation using filters and addition operation using bias) on the tensor data input from the previous stage CNN130. The nonlinear transformation layer 146 is configured to perform a nonlinear transformation using an activation function such as a ramp function on each element of the tensor input from the convolution layer 145. In addition, the pooling layer 147 is configured to perform a pooling operation such as average pooling on the output of the nonlinear transformation layer 146 according to a preset stride value.

[0141] RNN132 has: a flattening layer (FL) 151, a recursive layer (RL) 152, and a final output layer (LO) 153. The flattening layer 151 can generate matrix data consisting of a vector sequence (i.e., a sequence of multiple vectors) based on the output of the CNN131 of the previous level. The recursive layer 152 is a neural network layer with a directed closed loop inside, which can extract time-related feature quantities from the input vector sequence. As the recursive layer 152, for example, an LSTM layer, an improved version of the LSTM layer, a GRU layer, or a QRNN layer can be used. The recursive layer 152 is configured to generate an output vector sequence by processing the vector sequence input from the flattening layer 151. The final output layer 153 is configured to select the final output vector in the output vector sequence and output it.

[0142] The forward propagation type NN133 has: a linear transformation layer (LL) 161, a normalization layer (NL) 162, a nonlinear transformation layer (NLL) 163, and a linear transformation layer (LL) 164A and a linear transformation layer (LL) 164B. The linear transformation layer 161 is configured to perform a linear transformation using a weight matrix and a bias vector on the vector input from the RNN132. The normalization layer 162 can perform normalization such as subtractive normalization on each element of the vector input from the linear transformation layer 161 to generate a normalized vector. The nonlinear transformation layer 163 is configured to perform a nonlinear transformation using an activation function such as a ramp function on each element of the normalized vector. In addition, the linear transformation layer 164A generates a scalar risk Dr by performing a linear transformation using a weight matrix and a bias vector on the output of the nonlinear transformation layer 163. The risk Dr is output to the outside via the node of the output layer 64. On the other hand, the linear transformation layer 164B generates a scalar prediction time Tp by performing a linear transformation using a weight matrix and a bias vector on the output of the nonlinear transformation layer 163. The prediction time Tp is output to the outside via a node of the output layer 64.

[0143] As explained above, Fig.18 The NN structure 34 shown is capable of accepting and processing multidimensional array data MDy having measurement parameters and relative time values ​​assigned to these measurement parameters as array elements, so that the inference operation unit 27 can calculate the prediction information (Dr, Tp) with high accuracy based on the NN structure 34.

[0144] Fig. 20 : is a diagram showing an NN structure 35 as an example. The NN structure 35 has: an input layer 44, which is used to receive multidimensional array data MDy composed of third-order tensors; an intermediate layer structure 55, which includes CNN130, CNN131, and RNN132 configured to process the multidimensional array data MDy input via the input layer 44; and an output layer 64, which is configured to output prediction information (Dr, Tp) based on the output of the intermediate layer structure 55. The intermediate layer structure 55 further has: a forward propagation type NN134 composed of two sub-NN134A and sub-NN134B configured in parallel with each other. Sub-NN134A and sub-NN134B are respectively configured to process the output vector of the final output layer 153 in RNN132.

[0145] like Fig. 20As shown, the sub-NN134A has: a linear transformation layer (LL) 161A, a normalization layer (NL) 162A, a nonlinear transformation layer (NLL) 163A, a linear transformation layer (LL) 164A, a nonlinear transformation layer (NLL) 165A, and a linear transformation layer (LL) 166A. The linear transformation layer 161A is configured to perform a linear transformation using a weight matrix and a bias vector on the output vector of the final output layer 153 of the RNN132. The normalization layer 162A can perform normalization such as subtractive normalization on each element of the vector input from the linear transformation layer 161A to generate a normalized vector. The nonlinear transformation layer 163A is configured to perform a nonlinear transformation using an activation function such as a ramp function on each element of the normalized vector. The linear transformation layer 164A is configured to perform a linear transformation using a weight matrix and a bias vector on the output of the nonlinear transformation layer 163A. The nonlinear transformation layer 165A is configured to perform nonlinear transformation using an activation function such as a ramp function on each element of the vector input from the linear transformation layer 164A. In addition, the linear transformation layer 166A generates a scalar risk Dr by performing a linear transformation using a weight matrix and a bias vector on the output of the nonlinear transformation layer 165A. The risk Dr is output to the outside via the node of the output layer 64.

[0146] On the other hand, Fig. 20 As shown, the sub-NN134B has: a linear transformation layer (LL) 161B, a normalization layer (NL) 162B, a nonlinear transformation layer (NLL) 163B, a linear transformation layer (LL) 164B, a nonlinear transformation layer (NLL) 165B, and a linear transformation layer (LL) 166B. The linear transformation layer 161B is configured to perform a linear transformation using a weight matrix and a bias vector on the output vector of the final output layer 153 of the RNN132. The normalization layer 162B can perform normalization such as subtractive normalization on each element of the vector input from the linear transformation layer 161B to generate a normalized vector. The nonlinear transformation layer 163B is configured to perform a nonlinear transformation using an activation function such as a ramp function on each element of the normalized vector. The linear transformation layer 164B is configured to perform a linear transformation using a weight matrix and a bias vector on the output of the nonlinear transformation layer 163B. The nonlinear transformation layer 165B is configured to perform nonlinear transformation using an activation function such as a ramp function on each element of the vector input from the linear transformation layer 164B. In addition, the linear transformation layer 166B generates a scalar prediction time Tp by performing a linear transformation using a weight matrix and a bias vector on the output of the nonlinear transformation layer 165B. The prediction time Tp is output to the outside via the node of the output layer 64.

[0147] As explained above, Fig. 20The NN structure 35 shown can also accept and process multidimensional array data MDy having measurement parameters and relative time values ​​assigned to these measurement parameters as array elements, so the inference operation unit 27 can calculate the prediction information (Dr, Tp) with high precision based on the NN structure 35.

[0148] However, the above-mentioned NN structure 34 and NN structure 35 are respectively configured to be able to predict the occurrence of one abnormal event, but are not limited to this. The structure of the NN structure 34 or the NN structure 35 can be appropriately deformed to enable the prediction of the occurrence of two or more abnormal events. Fig.21 NN structure 36 is a diagram showing a modified example of NN structure 35. NN structure 36 is configured to be able to predict the occurrence of three types of abnormal events.

[0149] Reference Fig.21 , the NN structure 36 has: an input layer 44, which is used to receive multidimensional array data MDy composed of third-order tensors; an intermediate layer structure 56, which includes CNN130, CNN131, and RNN132 configured to process the multidimensional array data MDy input via the input layer 44; and an output layer 66, which is configured to output three types of prediction information (Dr1, Tp1), (Dr2, Tp2), and (Dr3, Tp3) based on the output of the intermediate layer structure 56. The intermediate layer structure 56 further has: a forward propagation type NN135 composed of two sub-NN135A and sub-NN135B configured in parallel with each other. Sub-NN135A and sub-NN135B are respectively configured to process the output vector of the final output layer 153 in RNN132.

[0150] like Fig.21 As shown, the sub-NN135A has: a linear transformation layer (LL) 171A, a normalization layer (NL) 172A, a nonlinear transformation layer (NLL) 173A, a linear transformation layer (LL) 174A, a nonlinear transformation layer (NLL) 175A, and a linear transformation layer (LL) 176A configured in parallel with each other. 1 , Linear transformation layer (LL) 176A 2 , Linear transformation layer (LL) 176A 3 In addition to the values ​​of the parameter groups (e.g., elements of the weight matrix and elements of the bias vector) updated by the machine learning algorithm, these linear transformation layer 171A, normalization layer 172A, nonlinear transformation layer 173A, linear transformation layer 174A, and nonlinear transformation layer 175A have the same Fig. 20 The linear transformation layer 161A, the normalization layer 162A, the nonlinear transformation layer 163A, the linear transformation layer 164A and the nonlinear transformation layer 165A shown have the same basic structure.1 To linear transformation layer 176A 3 The i-th linear transformation layer 176A in i A scalar risk Dri indicating the degree of the i-th abnormal event can be generated by performing a linear transformation using a weight matrix and a bias vector on the output vector of the nonlinear transformation layer 175A (i is an arbitrary integer in the range of 1 to 3). The risks Dr1, Dr2, and Dr3 are output to the outside via the nodes of the output layer 66.

[0151] On the other hand, Fig.21 As shown, the sub-NN135B has: a linear transformation layer (LL) 171B, a normalization layer (NL) 172B, a nonlinear transformation layer (NLL) 173B, a linear transformation layer (LL) 174B, a nonlinear transformation layer (NLL) 175B, and a linear transformation layer (LL) 176B configured in parallel with each other. 1 , Linear transformation layer (LL) 176B 2 , Linear transformation layer (LL) 176B 3 In addition to the values ​​of the parameter groups (e.g., elements of the weight matrix and elements of the bias vector) updated by the machine learning algorithm, these linear transformation layer 171B, normalization layer 172B, nonlinear transformation layer 173B, linear transformation layer 174B, and nonlinear transformation layer 175B have the same Fig. 20 The linear transformation layer 161B, the normalization layer 162B, the nonlinear transformation layer 163B, the linear transformation layer 164B and the nonlinear transformation layer 165B shown have the same basic structure. 1 To linear transformation layer 176B 3 The jth linear transformation layer 176B in j The predicted time Tpj (j is an integer in the range of 1 to 3) at which the jth abnormal event may occur can be generated by performing a linear transformation using a weight matrix and a bias vector on the output vector of the nonlinear transformation layer 175B. The predicted times Tp1, Tp2, and Tp3 are output to the outside via the nodes of the output layer 66.

[0152] It should be noted that in Fig.21 In the example of FIG. 1 , a linear transformation layer 176A is configured in parallel in the sub-NN 135A to process the output of the nonlinear transformation layer 175A. 1 To 176A 3 The number of the linear transformation layer 176B is three, and the linear transformation layer 176B is configured in parallel in the sub-NN135B to process the output of the nonlinear transformation layer 175B. 1 To 176B 3The number is three, but is not limited to this. The configuration of sub-NN135A and sub-NN135B can also be appropriately changed so that the number of linear transformation layers configured in parallel to process the output of nonlinear transformation layer 175A is four or more, and the number of linear transformation layers configured in parallel to process the output of nonlinear transformation layer 175B is four or more. In this way, it is possible to construct an NN structure that can predict the occurrence of four or more abnormal events.

[0153] Fig. 22 NN structure 37 is a diagram showing another modified example of NN structure 35. NN structure 37 is configured to be able to predict the occurrence of three types of abnormal events.

[0154] Reference Fig. 22 NN structure 37 has: input layer 44, which is used to receive multidimensional array data MDy composed of third-order tensors; intermediate layer structure 57, which includes CNN130, CNN131, and RNN132 configured to process multidimensional array data MDy input via input layer 44; and output layer 66, which is configured to output three types of prediction information (Dr1, Tp1), (Dr2, Tp2), (Dr3, Tp3) based on the output of intermediate layer structure 57. The intermediate layer structure 57 further has six sub-NN136 configured in parallel with each other. 1 , sub NN136 2 , sub NN136 3 , sub NN136 4 , sub NN136 5 , sub NN136 6 The forward propagation type NN136 composed of sub-NN136 1 Up to 136 6 Each of is configured to process the output vector of the final output layer 153 in the RNN 132.

[0155] like Fig. 22 As shown, sub NN136 1 To NN136 3 The i-th child NN136 in i Having: Linear transformation layer (LL) 181 i , Normalization layer (NL) 182 i , Non-linear transformation layer (NLL) 183 i , Linear transformation layer (LL) 184 i , Non-linear transformation layer (NLL) 185 i , and a linear transformation layer (LL) 186 i(i is any integer in the range of 1 to 3). In addition to the values ​​of the parameter groups (e.g., the elements of the weight matrix and the elements of the bias vector) updated by the machine learning algorithm, these linear transformation layers 181 i , Normalization layer 182 i , nonlinear transformation layer 183 i , linear transformation layer 184 i , nonlinear transformation layer 185 i and linear transformation layer 186 i Respectively have Fig. 20 The linear transformation layer 161A, the normalization layer 162A, the nonlinear transformation layer 163A, the linear transformation layer 164A, the nonlinear transformation layer 165A and the linear transformation layer 166A shown in FIG. i The nonlinear transformation layer 185 can be i The output vector of is subjected to a linear transformation using a weight matrix and a bias vector to generate a scalar risk Dri indicating the degree of the i-th abnormal event. The risks Dr1, Dr2, and Dr3 are output to the outside via the nodes of the output layer 66.

[0156] like Fig. 22 As shown, sub NN136 4 To NN136 6 The jth child NN136 in j Having: Linear transformation layer (LL) 181 j , Normalization layer (NL) 182 j , Non-linear transformation layer (NLL) 183 j , Linear transformation layer (LL) 184 j , Non-linear transformation layer (NLL) 185 j , and a linear transformation layer (LL) 186 j (j is any integer in the range of 4 to 6). In addition to the values ​​of the parameter groups (e.g., the elements of the weight matrix and the elements of the bias vector) updated by the machine learning algorithm, these linear transformation layers 181 j , Normalization layer 182 j , nonlinear transformation layer 183 j , linear transformation layer 184 j , nonlinear transformation layer 185 j and linear transformation layer 186 j Respectively have Fig. 20 The linear transformation layer 161B, the normalization layer 162B, the nonlinear transformation layer 163B, the linear transformation layer 164B, the nonlinear transformation layer 165B and the linear transformation layer 166B shown in FIG. j The nonlinear transformation layer 185 can bej The output vector of is linearly transformed using a weight matrix and a bias vector to generate a predicted time at which the (j-3)th abnormal event may occur. The predicted times Tp1, Tp2, and Tp3 are output to the outside via the nodes of the output layer 66.

[0157] In addition, Fig. 22 In the example of FIG. 1 , sub-NN 136 is configured in parallel to process the output vector of RNN 132 1 To NN136 6 The number is six, but not limited to this. The structure of the forward propagation type NN136 can also be appropriately changed so that the number of sub-NNs configured in parallel to process the output vector of RNN132 is eight or more. In this way, it is possible to construct an NN structure that can predict the occurrence of four or more abnormal events.

[0158] Fig.23 is a diagram showing an example of a NN structure 38. Fig.23 , the NN structure 38 has: an input layer 44, which is used to accept multidimensional array data MDy composed of a third-order tensor; an intermediate layer structure, which is composed of a sub-intermediate layer structure 58A and a sub-intermediate layer structure 58B configured in parallel with each other; and an output layer 64, which is configured to output prediction information (Dr, Tp) based on the output of the intermediate layer structure.

[0159] The sub-intermediate layer structure 58A has: a convolutional neural network (CNN) 130A, a convolutional neural network (CNN) 131A, a recurrent neural network (RNN) 132A, and a forward propagation type NN 137A. In addition to the values ​​of the parameter groups (e.g., the elements of the weight matrix and the elements of the bias vector) updated by the machine learning algorithm, CNN 130A, CNN 131A, and RNN 132A have the same Fig.18 CNN130, CNN131 and CNN132 shown in FIG. have the same basic structure. Fig.23 As shown, the forward propagation type NN137A has: a linear transformation layer (LL) 191A, a normalization layer (NL) 192A, a nonlinear transformation layer (NLL) 193A, a linear transformation layer (LL) 194A, a nonlinear transformation layer (NLL) 195A, and a linear transformation layer (LL) 196A. In addition to the values ​​of the parameter group (e.g., the elements of the weight matrix and the elements of the bias vector) updated by the machine learning algorithm, these linear transformation layers 191A, normalization layers 192A, nonlinear transformation layers 193A, linear transformation layers 194A, nonlinear transformation layers 195A, and linear transformation layers 196A have the same Fig. 20The linear transformation layer 161A, the normalization layer 162A, the nonlinear transformation layer 163A, the linear transformation layer 164A, the nonlinear transformation layer 165A, and the linear transformation layer 166A shown have the same basic structure.

[0160] The linear transformation layer 196A can generate a scalar risk Dr indicating the degree of abnormality by performing a linear transformation using a weight matrix and a bias vector on the output vector of the nonlinear transformation layer 195A. The risk Dr is output to the outside via a node of the output layer 64 .

[0161] On the other hand, the sub-intermediate layer structure 58B has a convolutional neural network (CNN) 130B, a convolutional neural network (CNN) 131B, a recurrent neural network (RNN) 132B, and a forward propagation type NN 137B. In addition to the values ​​of the parameter groups (e.g., the elements of the weight matrix and the elements of the bias vector) updated by the machine learning algorithm, CNN 130B, CNN 131B, and RNN 132B have the same Fig.18 CNN130, CNN131 and CNN132 shown in FIG. have the same basic structure. Fig.23 As shown, the forward propagation type NN137B has: a linear transformation layer (LL) 191B, a normalization layer (NL) 192B, a nonlinear transformation layer (NLL) 193B, a linear transformation layer (LL) 194B, a nonlinear transformation layer (NLL) 195B, and a linear transformation layer (LL) 196B. In addition to the values ​​of the parameter group (e.g., the elements of the weight matrix and the elements of the bias vector) updated by the machine learning algorithm, these linear transformation layers 191B, normalization layers 192B, nonlinear transformation layers 193B, linear transformation layers 194B, nonlinear transformation layers 195B, and linear transformation layers 196B have the same Fig. 20 The linear transformation layer 161B, the normalization layer 162B, the nonlinear transformation layer 163B, the linear transformation layer 164B, the nonlinear transformation layer 165B and the linear transformation layer 166B shown have the same basic structure.

[0162] The linear transformation layer 196B can generate a predicted time Tp at which an abnormal event may occur by performing a linear transformation using a weight matrix and a bias vector on the output vector of the nonlinear transformation layer 195B. The predicted time Tp is output to the outside via a node of the output layer 64 .

[0163] As mentioned above, Fig.18 and Figure 20 to Figure 23Each of the NN structures 34, NN structure 35, NN structure 36, NN structure 37, and NN structure 38 shown is configured to accept only multidimensional array data MDy. As an alternative, the composition of these NN structures can also be modified to accept multidimensional array data MDy and attribute information of the target object Tgt. Specifically, in each of the NN structures 34, NN structure 35, NN structure 36, NN structure 37, and NN structure 38, by including a connection layer configured to connect the attribute information input via the input layer with the output of the RNN (recurrent neural network), a modified example of the NN structure can be obtained.

[0164] As the attribute information of the target object Tgt, for example, a scalar or vector consisting of inherent information of a living body such as a human being can be used. More specifically, for example, as the attribute information, numerical values ​​representing age, gender, body mass index (BMI), and the presence or absence of a specific disease such as diabetes can be used. In this case, Figure 1 The data reconstruction unit 25 or the inference calculation unit 27 shown has an internal memory (not shown) that stores attribute information in advance, and provides the attribute information read from the internal memory to the NN structure. Alternatively, the data reconstruction unit 25 or the inference calculation unit 27 may be configured to obtain attribute information from an external information source (not shown) and provide the obtained attribute information to the NN structure.

[0165] Fig.24 It means as Fig.23 FIG. 39 is a diagram of a modified example of the NN structure 38 shown. Fig.23 The NN structure 39 has: an input layer 49, which is used to receive multidimensional array data MDy composed of a third-order tensor and attribute information AD of the object Tgt; an intermediate layer structure, which is composed of a sub-intermediate layer structure 59A and a sub-intermediate layer structure 59B; and an output layer 64, which is configured to output prediction information (Dr, Tp) based on the output of the intermediate layer structure.

[0166] The sub-intermediate layer structure 59A has: CNN130A, CNN131A, RNN132A, connection layer 76A, and forward propagation type NN138A. The connection layer 76A can generate a connection vector by connecting the output vector of the RNN132A with the scalar value or vector of the attribute information AD input via the input layer 49. For example, the connection layer 76A can be configured to connect the 40-dimensional vector input from the RNN132A with the 10-dimensional vector of the attribute information AD to generate a 50-dimensional connection vector.

[0167] The forward propagation NN 138A is configured to process the connection vector input from the connection layer 76A, and has the same Fig.23The same basic structure as the forward propagation type NN137A shown in the figure. Fig.24 As shown, the forward propagation type NN138A has: a linear transformation layer (LL) 201A, a normalization layer (NL) 202A, a nonlinear transformation layer (NLL) 203A, a linear transformation layer (LL) 204A, a nonlinear transformation layer (NLL) 205A, and a linear transformation layer (LL) 206A. These linear transformation layer 201A, normalization layer 202A, nonlinear transformation layer 203A, linear transformation layer 204A, nonlinear transformation layer 205A, and linear transformation layer 206A have the same Fig.23 The linear transformation layer 191A, the normalization layer 192A, the nonlinear transformation layer 193A, the linear transformation layer 194A, the nonlinear transformation layer 195A and the linear transformation layer 196A in the forward propagation type NN 137A shown in FIG. The linear transformation layer 206A generates a scalar risk Dr by performing a linear transformation using a weight matrix and a bias vector on the output of the nonlinear transformation layer 205A. The risk Dr is output to the outside via the node of the output layer 64.

[0168] On the other hand, the sub-intermediate layer structure 59B has: CNN130B, CNN131B, RNN132B, connection layer 76B, and forward propagation type NN138B. The connection layer 76B can generate a connection vector by connecting the output vector of the RNN132B with the scalar value or vector of the attribute information AD input via the input layer 49. For example, the connection layer 76B can be configured to connect the 40-dimensional vector input from the RNN132B with the 10-dimensional vector of the attribute information AD to generate a 50-dimensional connection vector.

[0169] The forward propagation type NN138B is configured to process the connection vector input from the connection layer 76B, with Fig.23 The same basic structure as the forward propagation type NN137B shown in the figure. Fig.24 As shown, the forward propagation type NN138B has: a linear transformation layer (LL) 201B, a normalization layer (NL) 202B, a nonlinear transformation layer (NLL) 203B, a linear transformation layer (LL) 204B, a nonlinear transformation layer (NLL) 205B, and a linear transformation layer (LL) 206B. These linear transformation layers 201B, normalization layers 202B, nonlinear transformation layers 203B, linear transformation layers 204B, nonlinear transformation layers 205B, and linear transformation layers 206B have the same Fig.23The linear transformation layer 191B, the normalization layer 192B, the nonlinear transformation layer 193B, the linear transformation layer 194B, the nonlinear transformation layer 195B and the linear transformation layer 196B in the forward propagation type NN 138B shown in the figure have the same basic structure. The linear transformation layer 206B generates a scalar prediction time Tp by linearly transforming the output of the nonlinear transformation layer 205B using a weight matrix and a bias vector. The prediction time Tp is output to the outside via the node of the output layer 64.

[0170] As explained above, Fig.24 The NN structure 39 shown has, in addition to multidimensional array data MDy having measurement parameters and relative time values ​​assigned to these measurement parameters as array elements, also capable of receiving and processing attribute information AD of the object Tgt. Therefore, the inference operation unit 27 can calculate the prediction information (Dr, Tp) related to the object Tgt with higher accuracy based on the NN structure 39.

[0171] It should be noted that in Fig.23 and Fig.24 In the example, the number of sub-middle-layer structures configured in parallel to process the multi-dimensional array data MDy in parallel is two, but it is not limited to this. The components of the NN structure 38 and the NN structure 39 can also be appropriately changed so that the number of sub-middle-layer structures configured in parallel to process the multi-dimensional array data MDy in parallel is four or more. In this way, it is possible to construct an NN structure that can predict the occurrence of two or more abnormal events.

[0172] Next, a feedback control system using the above-mentioned prediction information will be described. Fig.25 It is a block diagram schematically showing the configuration of an abnormality prediction system 2 according to one embodiment of the present disclosure.

[0173] like Fig.25 As shown, the abnormal prediction system 2 includes: multiple devices E 0 ,…,E K-1 , each device is used to observe or process the target object Tgt; and an abnormality prediction device 11, which is configured to be based on the data from these devices E 0 ,…,E K-1 Provide multiple data time series D 0 ,…,D K-1 To predict the occurrence of at least one abnormal event. Here, K represents the device E 0 To E K-1 A positive integer of 3 or more. Equipment E 0 ,…,E K-1 Each has a measuring device S 0 ,…,S K-1 . Fig.25 The measuring devices S shown k Function and Figure 1 The measuring device S shown k It should be noted that device E 0 ,…,E K-1 The number of devices is not limited to three or more, and the structure of the abnormality prediction system 2 may be appropriately changed to include two devices.

[0174] The abnormal prediction device 11 and Figure 1 The abnormal prediction device 10 shown also includes: a data reconstruction system 20, which is configured to 0 ,…,E K-1 Provided data time series D 0 ,…,D K-1 reconstructing the multidimensional array data; and an inference operation unit 27, which is configured to calculate prediction information related to the abnormal event by performing an operation based on the NN structure 30 that takes the multidimensional array data as input. The abnormality prediction device 11 further includes: a feedback control unit 28, which is configured to modify the device E according to the prediction information calculated by the inference operation unit 27. 0 ,…,E K-1 Cyclic control of operating conditions.

[0175] The feedback control unit 28 can generate a method for adaptively changing the device E according to the prediction information. 0 ,…,E K-1 The control information CT of the operating conditions of the device E is provided 0 ,…,E K-1 . Equipment E 0 ,…,E K-1 The feedback control unit 28 is configured to perform operations corresponding to the control information CT. For example, when the risk Dr and the predicted time Tp match a predetermined condition (for example, when the risk Dr exceeds a predetermined threshold value), the feedback control unit 28 can generate the control information CT as a trigger. In this way, the actual occurrence of abnormal events can be prevented.

[0176] The above cycle control is in device E 0 ,…,E K-1 This is effective when the operation intensity depends on the measured parameters used to adjust the specific operation intensity. For example, speed parameters such as liquid delivery concentration and operating temperature belong to this category. In order for this cycle control to work effectively, it is desirable to predict the occurrence of abnormal events as early as possible (for example, the difference between the time when the prediction information Tp is generated and the predicted time Tp is 20 minutes or more). That is, a time margin is required to allow the device E to 0 ,…,EK-1 Changes made to the control system are reflected in the monitoring.

[0177] Next, refer to Fig.26 , the processing process of the above-mentioned abnormality prediction device 11 is explained below. Fig.26 This is a flowchart schematically showing an example of the processing procedure of the loop control. Fig.26 Steps S11 to S13 shown are similar to Figure 6 The steps S11 to S13 shown are the same.

[0178] After step S13, the feedback control unit 28 determines whether the generated prediction information (Dr, Tp) is consistent with the specified conditions (step S14). k The prediction information is determined to be consistent with the prescribed condition by determining whether the risk Dr of an abnormal event such as the operating temperature exceeding the allowable range exceeds a threshold. If the prediction information is inconsistent with the prescribed condition ("No" in step S14), the feedback control unit 28 returns the process to step S11.

[0179] On the other hand, when the prediction information matches the predetermined condition (“Yes” in step S14), the feedback control unit 28 generates a command for changing the device E based on the prediction information. 0 ,…,E K-1 The feedback control unit 28 receives the control information CT of the operating condition of at least one device in the control unit (step S15), and sends the control information CT (step S16). After that, when the feedback control unit 28 determines to continue the process ("Yes" in step S20), the process returns to step S11. Finally, when the feedback control unit 28 determines not to continue the process ("No" in step S20), the feedback control unit 28 ends the process.

[0180] As described above, the abnormality prediction system 3 includes the data reconstruction system 20, the inference calculation unit 27, and the feedback control unit 28, so that the abnormality event can be prevented from happening, and the device E can be 0 ,…,E K-1 Operate safely.

[0181] Next, an abnormality prediction system of another embodiment is described. In the abnormality prediction system 1 and the abnormality prediction system 2, the data reconstruction unit 25 in the abnormality prediction device 10 and the abnormality prediction device 11 has a function of assigning the relative time value τ to the measurement parameter. Alternatively, an embodiment in which the device has a function of assigning the relative time value τ to the measurement parameter may be used. Fig. 27 1 is a block diagram schematically showing the configuration of an abnormality prediction system 3 as one of such embodiments.

[0182] like Fig. 27 As shown, the abnormal prediction system 3 includes: multiple devices U0 , …, U K-1 , each device is used to observe or process the target object Tgt; and an abnormality prediction device 12, which is configured to be based on the abnormality prediction device U respectively from these devices 0 , …, U K-1 Provide multiple vector sequences G 0 , …, G K-1 To predict the occurrence of at least one abnormal event. Here, K represents the device U 0 toU K-1 The number of devices is a positive integer of 3 or more. 0 toU K-1 For example, it can be a medical device, a manufacturing device, or a monitoring device. 0 , …, U K-1 The number of devices is not limited to three or more, and the structure of the abnormality prediction system 3 may be appropriately changed to include two devices.

[0183] Equipment 0 , …, U K-1 Each has a measuring device S 0 ,…,S K-1 . Fig. 27 The measuring devices S shown k Function and Figure 1 The measuring device S shown k The functions of each measuring device S are the same. k is configured to physically (eg, optically, electrically or acoustically) measure the k Device U k The state of the object Tgt or the state of the target object Tgt, and output the measurement quantity D obtained from the measurement result. k (t) is a time series composed of data time series D k .

[0184] Furthermore, the device U 0 , …, U K-1 Each has a sequence generator V 0 ,…,V K-1 . Sequence Generator V 0 ,…,V K-1 is configured to be based on the measurement device S 0 ,…,S K-1 Input data time series D 0 ,…,D K-1 Generate vector sequences G in parallel 0 , …, G K-1 A vector sequence G kThe sequence generator V consists of a sequence of a plurality of vectors, each of which has at least one measurement parameter and a relative time value τ assigned to the at least one measurement parameter as a vector element. 0 ,…,V K-1 With the use of equipment U 0 , …, U K-1 The timer (not shown) in the sequence generator V assigns a relative time value τ based on a common time reference to the function of measuring the parameter. 0 ,…,V K-1 The generated vector sequence G 0 , …, G K-1 As long as Figure 7 or Figure 8 The vector sequence G shown 0 , …, G K-1 However, Figure 7 and Figure 8 In the example, the vector sequence G 0 , …, G K-1 The length of the sequences is the same value (=N), but it is not limited to this. When i and j are set to any different integers in the range of 1 to K (i≠j), the i-th sequence generator V i The generated vector sequence G i The sequence length and the j-th sequence generator V j The generated vector sequence G j The lengths of the sequences can be different.

[0185] The abnormality prediction device 12 communicates with the device U via a wired transmission path such as a transmission cable or a wireless transmission path. 0 , …, U K-1 Connect. Device U 0 , …, U K-1 Having will be generated by the sequence generator V 0 ,…,V K-1 The vector sequence G obtained respectively 0 , …, G K-1 The abnormality prediction device 12 has the function of transmitting the abnormality prediction device 12 to the abnormality prediction device 12 via a wired transmission path or a wireless transmission path. 0 , …, U K-1 Receive vector sequence G 0 , …, G K-1 function.

[0186] like Fig. 27 As shown, the abnormality prediction device 12 includes: a data input unit 24, which receives and temporarily stores data from the device U 0 , …, U K-1 Provided vector sequence G 0 , …, GK-1 ; Integration unit 26, the integration unit is configured to integrate the vector sequence G input via the data input unit 24 0 , …, G K-1 to generate multidimensional array data to be input to the input layer of the NN structure 30; and an inference operation unit 27, which is configured to calculate prediction information related to abnormal events by performing operations based on the NN structure 30 with the multidimensional array data as input. Fig. 27 The NN structure 30 shown has Figures 9 to 11 and Fig.16 The same structure can be any one of the NN structures 31 to 33 and 31B shown in the figure. 0 , …, G K-1 When there is a mismatch between Figure 8 In the case of dimensional mismatch as shown in the figure), the N structure N30 can also compensate for the mismatch, so the inference operation unit 27 can perform highly reliable operations based on the NN structure 30.

[0187] In the abnormal prediction system 3, if Fig. 27 As shown, through the sequence generator V 0 ,…,V K-1 The combination of the data input unit 24 and the integration unit 26 constitutes the data reconstruction system 21.

[0188] Next, refer to Fig.28 , the processing process of the above-mentioned abnormality prediction device 12 is explained below. Fig.28 is a flowchart schematically illustrating an example of a process for predicting the occurrence of an abnormal event.

[0189] Reference Fig.28 The integration unit 26 reads the vector sequence G from the data group temporarily stored in the data input unit 24. 0 , …, G K-1 (Step S11A), by integrating the vector sequence G 0 , …, G K-1 To generate multidimensional array data to be input to the inference operation unit 27 (ie, the input layer of the NN structure 30) (step S12A).

[0190] Next, the inference operation unit 27 calculates prediction information related to the abnormal event by performing operations based on the neural network structure 30 by taking the multidimensional array data as input (step S13).

[0191] After that, when the inference operation unit 27 determines to continue the process (“Yes” in step S20), the process returns to step S11A. At this time, the integration unit 26 reads a new vector sequence G from a set of data temporarily stored in the data input unit 24. 0 , …, G K-1 (Step S11A), according to the new vector sequence G 0 , …, G K-1 Generate multidimensional array data (step S12A). Next, the inference operation unit 27 calculates prediction information related to abnormal events by taking the multidimensional array data as input and performing operations based on the neural network structure 30 (step S13). In this way, the abnormality prediction device 12 can predict the occurrence of abnormal events one by one and in real time. Finally, when the inference operation unit 27 determines that the processing is not to be continued ("No" in step S20), the processing ends.

[0192] Next, a feedback control system which is a modified example of the abnormality prediction system 3 described above will be described. Fig.29 It is a block diagram schematically showing the configuration of an abnormality prediction system 4 according to one embodiment of the present disclosure.

[0193] like Fig.29 As shown, the abnormal prediction system 4 includes: multiple devices U 0 , …, U K-1 , each device is used to observe or process the target object Tgt; and an abnormality prediction device 13, which is configured to be based on the abnormality prediction device U from these devices respectively 0 , …, U K-1 Provide multiple vector sequences G 0 , …, G K-1 To predict the occurrence of at least one abnormal event. It should be noted that the device U 0 , …, U K-1 The number of devices is not limited to three or more, and the structure of the abnormality prediction system 4 may be appropriately changed to include two devices.

[0194] The abnormal prediction device 13 and Fig. 27 The abnormality prediction device 12 shown in the figure also includes a data input unit 24, an integration unit 26 and an inference calculation unit 27. 0 ,…,V K-1 The abnormality prediction device 13 further includes a feedback control unit 28 configured to modify the device U according to the prediction information calculated by the inference operation unit 27. 0 , …, U K-1 Cyclic control of operating conditions.

[0195] The feedback control unit 28 can generate a feedback signal for adaptively changing the device U according to the prediction information. 0 , …, U K-1 The control information CT of the operating conditions of the device U is provided 0 , …, U K-1 . Equipment U 0 , …, U K-1 The feedback control unit 28 is configured to perform operations corresponding to the control information CT. For example, when the risk Dr and the predicted time Tp match a predetermined condition (for example, when the risk Dr exceeds a predetermined threshold value), the feedback control unit 28 can generate the control information CT as a trigger. In this way, the actual occurrence of abnormal events can be prevented.

[0196] Next, refer to Fig.30 , the processing process of the above-mentioned abnormality prediction device 13 is explained below. Fig.30 This is a flowchart schematically showing an example of the processing procedure of the loop control. Fig.30 Steps S11A to S13 shown in FIG. Fig.28 Steps S11A to S13 are the same as shown. Fig.30 Steps S14 to S20 shown in FIG. Fig.26 Steps S14 to S20 shown are the same.

[0197] After step S13, the feedback control unit 28 determines whether the generated prediction information (Dr, Tp) is consistent with the specified conditions (step S14). When the prediction information is inconsistent with the specified conditions ("No" in step S14), the feedback control unit 28 returns the process to step S11A. On the other hand, when the prediction information is consistent with the specified conditions ("Yes" in step S14), the feedback control unit 28 generates a setting for changing the device U according to the prediction information. 0 , …, U K-1 The feedback control unit 28 receives the control information CT of the operating condition of at least one device in the control unit (step S15), and sends the control information CT (step S16). After that, when the feedback control unit 28 determines to continue the process ("Yes" in step S20), the process returns to step S11A. Finally, when the feedback control unit 28 determines not to continue the process ("No" in step S20), the feedback control unit 28 ends the process.

[0198] As described above, the abnormality prediction system 4 includes the data reconstruction system 21, the inference calculation unit 27, and the feedback control unit 28, so that the abnormality event can be prevented from occurring before it occurs, and the device U can be 0 , …, U K-1 Operate safely.

[0199] and Figure 1The same is true for the abnormal prediction device 10 shown in the figure. All or part of the constituent elements of each abnormal prediction device in the above-mentioned abnormal prediction device 11 to abnormal prediction device 13 can be implemented by a computer including one or more processors, or can also be implemented by multiple computers interconnected via a communication path. All or part of the constituent elements of each abnormal prediction device can be implemented by one or more processors, which contain one or more computing devices (Processing Units) that execute processing by software or firmware codes (instruction groups) read from non-volatile memory (computer-readable storage medium). Alternatively, all or part of the constituent elements of each abnormal prediction device can be implemented by one or more processors having semiconductor integrated circuits such as FPGA. Alternatively, all or part of the constituent elements of each abnormal prediction device can also be implemented by one or more processors that include a combination of semiconductor integrated circuits such as FPGA and computing devices such as CPU and GPU.

[0200] In the above, various embodiments of the present disclosure are described with reference to the accompanying drawings, but these embodiments are exemplary and various modes other than these embodiments may exist. Fig.18 and Figure 20 to Figure 24 In the example, the number of CNNs (convolutional neural networks) connected in series in each intermediate layer structure or each sub-intermediate layer structure is two, but it is not limited to this. Each intermediate layer structure or each sub-intermediate layer structure can also be appropriately changed so that the number of CNNs is one or three or more.

[0201] In addition, it can be Fig.18 and Figure 20 to Figure 24 The multi-dimensional array data MDy processed by the NN structures 34 to 39 shown are not limited to Fig.17 A and Fig.17 Multidimensional array data MD3 shown in B.

[0202] In addition, as mentioned above, Figure 1 and Fig.25 The abnormality prediction system 1 and the abnormality prediction system 2 are respectively configured to include a plurality of devices E 0 ,…,E K-1 However, as an alternative, the device E may be configured outside each of the abnormality prediction system 1 and the abnormality prediction system 2. 0 ,…,E K-1 In other words, the abnormality prediction system 1 and the abnormality prediction system 2 do not include the device E 0 ,…,E K-1 Ways to use device E 0 ,…,E K-1 The output of abnormal prediction system 1 and abnormal prediction system 2 are changed.

[0203] It should be understood that the above-mentioned embodiments may be appropriately modified, added, and improved without departing from the spirit and scope of the present invention. The scope of the present invention should be understood based on the description of the claims and should be understood to include equivalents thereof.

[0204] Industrial Applicability

[0205] Since the abnormality prediction device, method and computer program involved in the present disclosure can predict the occurrence of abnormal events with high accuracy, they can be applied to, for example, the prediction of the occurrence of defective products in industrial production, the prediction of the occurrence of equipment operation failures, the prediction of the occurrence of equipment failures in medical monitoring, and the prediction of the occurrence of adverse events in biological organisms in medical monitoring.

[0206] Description of Reference Numerals

[0207] 1 to 4: abnormality prediction system; 10 to 13: abnormality prediction device; 20, 21: data reconstruction system; 23, 24: data input unit; 25: data reconstruction unit; 26: integration unit; 27: inference operation unit; 28: feedback control unit; 30 to 39: neural network structure (NN structure); 40 to 42, 44, 48, 49: input layer; 50 to 57: intermediate layer structure; 58A, 58B, 59A, 59B: sub-intermediate layer structure; 60 to 64, 66: output layer; 70 0 Up to 70 K-1 : Recurrent Neural Network (RNN); 81, 81B, 81C, 82, 83: Forward Propagation Neural Network (Forward Propagation NN); 90, 90A, 90B, 90C, 100, 100A, 100B, 100C, 110, 120: Sub-Neural Network (Sub-NN); 130, 130A, 130B, 131, 131A, 131B: Convolutional Neural Network (CNN); 132, 132A, 132B: Recurrent Neural Network (RNN); 133, 134, 135, 136, 137A, 137B, 138A, 138B: Forward Propagation Neural Network (Forward Propagation NN; 134A, 134B, 135A, 135B, 136 1 Up to 136 6 : Sub-neural network (sub-NN); 300: information processing device; 301: processor; 302: RAM (Random Access Memory (RAM: Random Access Memory)); 303: non-volatile memory; 304: large-capacity storage; 305: input-output interface; 306: signal path.

Claims

1. An abnormality prediction system, wherein the abnormality prediction system predicts the occurrence of at least one abnormal event based on a plurality of data time series obtained by a plurality of measuring devices, the abnormality prediction system comprising: a data reconstruction system configured to reconstruct multidimensional array data according to the plurality of data time series, the multidimensional array data comprising a plurality of measurement parameters and a plurality of relative time values ​​respectively assigned to the plurality of measurement parameters as array elements; as well as An inference operation unit, wherein the inference operation unit is configured to calculate prediction information related to the abnormal event by performing operations based on a neural network structure, wherein the neural network structure has an input layer for accepting the multidimensional array data, an intermediate layer structure including one or more recursive neural networks, and an output layer.

2. The abnormality prediction system according to claim 1, wherein: The prediction information is information representing a set of a risk level indicating the degree of the abnormal event and a predicted time at which the abnormal event is likely to occur.

3. The abnormality prediction system according to claim 1, wherein: The data reconstruction system comprises: a data input unit that receives the plurality of data time series provided respectively from the plurality of measuring devices; and A data reconstruction unit is configured to reconstruct the multi-dimensional array data according to the plurality of data time series input via the data input unit.

4. The abnormality prediction system according to claim 3, wherein: The data reconstruction unit is configured to generate a plurality of vector sequences as the multi-dimensional array data according to the plurality of data time series. Each of the plurality of vector sequences has a dimension for specifying each of the plurality of measurement parameters and a dimension for specifying each of the plurality of relative time values, and The multiple recurrent neural networks are configured to respectively process the multiple vector sequences input via the input layer in parallel.

5. The abnormality prediction system according to claim 1, wherein: The data reconstruction system comprises: A plurality of sequence generators, each of which is configured to generate a plurality of vector sequences according to the plurality of data time series; a data input section receiving the plurality of vector sequences respectively provided from the plurality of sequence generators; and an integration unit configured to generate the multidimensional array data to be input to the input layer by integrating the plurality of vector sequences input via the data input unit, Each sequence generator of the plurality of sequence generators is configured to generate a vector sequence consisting of a plurality of vectors, each vector having at least one measurement parameter and a relative time value assigned to the at least one measurement parameter as a vector element, and The multiple recurrent neural networks are configured to respectively process the multiple vector sequences input via the input layer in parallel.

6. The abnormality prediction system according to claim 4 or 5, wherein: The intermediate layer structure comprises: a connection layer configured to connect outputs of the plurality of recurrent neural networks; and A forward propagation neural network is configured to process the output of the connection layer.

7. The abnormality prediction system according to claim 6, wherein: The forward propagation neural network comprises: A first sub-neural network, the first sub-neural network configured to process an output of the connection layer; and a second sub-neural network, wherein the second sub-neural network is configured to be configured in parallel with the first sub-neural network and processes the output of the connection layer, and The inference operation unit is configured to generate a risk indicating the degree of the abnormal event based on the output of the first sub-neural network, and to generate a predicted time when the abnormal event is likely to occur based on the output of the second sub-neural network.

8. The abnormality prediction system according to claim 4 or 5, wherein: At least one of the plurality of measuring devices is configured to measure a state of a target object, and The connection layer is configured to connect the attribute information of the target object input via the input layer and the outputs of the plurality of recurrent neural networks.

9. The abnormality prediction system according to claim 3, wherein: The intermediate layer structure comprises: one or more convolutional neural networks, wherein the one or more convolutional neural networks are configured to process the multidimensional array data input via the input layer; and The one or more recurrent neural networks are configured to process outputs of the one or more convolutional neural networks.

10. The abnormality prediction system according to claim 9, wherein: The multidimensional array data is composed of a third-order tensor.

11. The abnormality prediction system according to claim 9, wherein: The intermediate layer structure further comprises: a first sub-neural network configured to process an output of the recurrent neural network; and a second sub-neural network, the second sub-neural network being configured in parallel with the first sub-neural network and processing an output of the recurrent neural network, and The inference operation unit is configured to generate a risk indicating the degree of the abnormal event based on the output of the first sub-neural network, and to generate a predicted time when the abnormal event is likely to occur based on the output of the second sub-neural network.

12. The abnormality prediction system according to claim 9, wherein: The intermediate layer structure comprises: a first sub-intermediate layer structure, the first sub-intermediate layer structure being configured to process the multi-dimensional array data input via the input layer; and a second sub-middle layer structure, wherein the second sub-middle layer structure is configured to be configured in parallel with the first sub-middle layer structure and processes the multi-dimensional array data inputted via the input layer, The first sub-intermediate layer structure comprises: a first convolutional neural network, the first convolutional neural network being configured as at least one of the plurality of convolutional neural networks; and a first recursive neural network, the first recursive neural network being arranged as one of the plurality of recursive neural networks and being configured to process an output of the first convolutional neural network, The second sub-intermediate layer structure comprises: a second convolutional neural network, the second convolutional neural network being arranged as at least one other of the plurality of convolutional neural networks; and a second recursive neural network, the second recursive neural network being arranged as another one of the plurality of recursive neural networks and being configured to process an output of the second convolutional neural network, and The inference operation section is configured to generate a risk level indicating the degree of the abnormal event based on the output of the first sub-middle layer structure, and to generate a predicted time when the abnormal event is likely to occur based on the output of the second sub-middle layer structure.

13. The abnormality prediction system according to claim 9, wherein: At least one of the plurality of measuring devices is configured to measure a state of a target object, and The intermediate layer structure further includes: one or more connection layers, and the one or more connection layers are configured to connect the outputs of the one or more recurrent neural networks with the attribute information of the target object input via the input layer.

14. The abnormality prediction system according to claim 13, wherein: The intermediate layer structure comprises: a first sub-intermediate layer structure, the first sub-intermediate layer structure being configured to process the multi-dimensional array data input via the input layer; and a second sub-middle layer structure, wherein the second sub-middle layer structure is configured to be configured in parallel with the first sub-middle layer structure and processes the multi-dimensional array data inputted via the input layer, The first sub-intermediate layer structure comprises: A first convolutional neural network, wherein the first convolutional neural network is configured as at least one of the plurality of convolutional neural networks; a first recursive neural network, the first recursive neural network being set as one of the plurality of recursive neural networks and configured to process an output of the first convolutional neural network; a first connection layer, the first connection layer being set as one of the plurality of connection layers and configured to connect an output of the first recursive neural network with the attribute information; and a first forward propagation neural network, wherein the first forward propagation neural network is configured to process an output of the first connection layer, The second sub-intermediate layer structure comprises: a second convolutional neural network, the second convolutional neural network being arranged as at least one other of the plurality of convolutional neural networks; a second recursive neural network, the second recursive neural network being set as another one of the plurality of recursive neural networks and configured to process an output of the second convolutional neural network; a second connection layer, the second connection layer being set as another one of the plurality of connection layers and configured to connect an output of the second recurrent neural network with the attribute information; and A second forward propagation neural network, wherein the second forward propagation neural network is configured to process the output of the second connection layer.

15. The abnormality prediction system according to any one of claims 1 to 5 and 9 to 14, further comprising a feedback control unit that controls the change of operating conditions of a device including at least one of the plurality of measuring devices based on the prediction information.

16. A method for predicting the occurrence of at least one abnormal event based on a plurality of data time series obtained by a plurality of measuring devices, comprising: a step of reconstructing multidimensional array data according to the plurality of data time series, the multidimensional array data comprising a plurality of measurement parameters and a plurality of relative time values ​​respectively assigned to the plurality of measurement parameters as array elements; as well as A step of calculating prediction information related to the abnormal event by performing operations based on a neural network structure, wherein the neural network structure has an input layer for accepting the multi-dimensional array data, an intermediate layer structure including one or more recursive neural networks, and an output layer.

17. The method according to claim 16, wherein: The prediction information is information representing a set of a risk level indicating the degree of the abnormal event and a predicted time at which the abnormal event is likely to occur.

18. The method according to claim 16, wherein: The step of reconstructing the multidimensional array data comprises the step of generating a plurality of vector sequences as the multidimensional array data according to the plurality of data time series. Each of the plurality of vector sequences has a dimension for specifying each of the plurality of measurement parameters and a dimension for specifying each of the plurality of relative time values, and The multiple recurrent neural networks are configured to respectively process the multiple vector sequences input via the input layer in parallel.

19. The method according to claim 16, wherein: The step of reconstructing the multidimensional array data comprises: A step of generating a plurality of vector sequences according to the plurality of data time series; and The step of generating the multidimensional array data to be input to the input layer by integrating the plurality of vector sequences, Each vector sequence of the plurality of vector sequences is composed of a plurality of vectors, each vector having at least one measurement parameter and a relative time value assigned to the at least one measurement parameter as vector elements, and The multiple recurrent neural networks are configured to respectively process the multiple vector sequences input via the input layer in parallel.

20. The method according to claim 18 or 19, wherein: The intermediate layer structure comprises: a connection layer configured to connect outputs of the plurality of recurrent neural networks; and A forward propagation neural network is configured to process the output of the connection layer.

21. The method according to claim 20, wherein: The forward propagation neural network comprises: A first sub-neural network, the first sub-neural network configured to process an output of the connection layer; and a second sub-neural network, wherein the second sub-neural network is configured to be configured in parallel with the first sub-neural network and processes the output of the connection layer, and The step of calculating the prediction information includes: a step of generating a risk level indicating the degree of the abnormal event based on the output of the first sub-neural network; and a step of generating a predicted time when the abnormal event may occur based on the output of the second sub-neural network.

22. The method according to claim 18 or 19, wherein: At least one of the plurality of measuring devices is configured to measure a state of a target object, and The connection layer is configured to connect the attribute information of the target object input via the input layer and the outputs of the plurality of recurrent neural networks.

23. The method according to claim 16, wherein: The intermediate layer structure comprises: one or more convolutional neural networks, wherein the one or more convolutional neural networks are configured to process the multidimensional array data input via the input layer; and The one or more recurrent neural networks are configured to process outputs of the one or more convolutional neural networks.

24. The method according to claim 23, wherein: The multidimensional array data is composed of a third-order tensor.

25. The method according to claim 23, wherein: The intermediate layer structure further comprises: a first sub-neural network configured to process an output of the recurrent neural network; and a second sub-neural network, the second sub-neural network being configured in parallel with the first sub-neural network and processing an output of the recurrent neural network, and The step of calculating the prediction information includes: a step of generating a risk level indicating the degree of the abnormal event based on the output of the first sub-neural network; and a step of generating a predicted time when the abnormal event may occur based on the output of the second sub-neural network.

26. The method of claim 23, wherein: The intermediate layer structure comprises: a first sub-intermediate layer structure, the first sub-intermediate layer structure being configured to process the multi-dimensional array data input via the input layer; and a second sub-middle layer structure, wherein the second sub-middle layer structure is configured to be configured in parallel with the first sub-middle layer structure and processes the multi-dimensional array data inputted via the input layer, The first sub-intermediate layer structure comprises: a first convolutional neural network, the first convolutional neural network being configured as at least one of the plurality of convolutional neural networks; and a second recursive neural network, the second recursive neural network being set as one of the plurality of recursive neural networks and configured to process an output of the first convolutional neural network, The second sub-intermediate layer structure comprises: a second convolutional neural network, the second convolutional neural network being arranged as at least one other of the plurality of convolutional neural networks; and a second recursive neural network, the second recursive neural network being arranged as another one of the plurality of recursive neural networks and being configured to process an output of the second convolutional neural network, and The step of calculating the prediction information includes: generating a risk level indicating the degree of the abnormal event based on the output of the first sub-middle layer structure; and generating a predicted time when the abnormal event may occur based on the output of the second sub-middle layer structure.

27. The method according to claim 23, wherein: At least one of the plurality of measuring devices is configured to measure a state of a target object, and The intermediate layer structure further includes: one or more connection layers, and the one or more connection layers are configured to connect the outputs of the one or more recurrent neural networks with the attribute information of the target object input via the input layer.

28. The method according to claim 27, wherein: The intermediate layer structure comprises: a first sub-intermediate layer structure, the first sub-intermediate layer structure being configured to process the multi-dimensional array data input via the input layer; and a second sub-middle layer structure, wherein the second sub-middle layer structure is configured to be configured in parallel with the first sub-middle layer structure and processes the multi-dimensional array data inputted via the input layer, The first sub-intermediate layer structure comprises: A first convolutional neural network, wherein the first convolutional neural network is configured as at least one of the plurality of convolutional neural networks; a first recursive neural network, the first recursive neural network being set as one of the plurality of recursive neural networks and configured to process an output of the first convolutional neural network; a first connection layer, the first connection layer being set as one of the plurality of connection layers and configured to connect an output of the first recursive neural network with the attribute information; and a first forward propagation type neural network, wherein the first forward propagation type neural network is configured to process an output of the first connection layer, and The second sub-intermediate layer structure comprises: a second convolutional neural network, the second convolutional neural network being arranged as at least one other of the plurality of convolutional neural networks; a second recursive neural network, the second recursive neural network being set as another one of the plurality of recursive neural networks and configured to process an output of the second convolutional neural network; a second connection layer, the second connection layer being set as another one of the plurality of connection layers and configured to connect an output of the second recurrent neural network with the attribute information; and A second forward propagation neural network, wherein the second forward propagation neural network is configured to process the output of the second connection layer.

29. The method according to any one of claims 16 to 19, 23 to 28, further comprising: A step of controlling to change an operating condition of a device including at least one of the plurality of measuring devices based on the prediction information is performed.

30. A computer program, the computer program being read out from a non-volatile memory and executed by one or more processors, and being configured to cause the one or more processors to implement the method of any one of claims 16 to 19, 23 to 28.

31. A computer program, the computer program being read out from a non-volatile memory and executed by one or more processors, and being configured to cause the one or more processors to implement the method of claim 29.

Citation Information

Patent Citations

  • Unsupervised anomaly detection, diagnosis, and correction for multivariate time series data

    JP2021527906A

  • Unsupervised anomaly detection, diagnosis, and correction in multivariate time series data

    WO2020046806A1