Computer system, control system, and processing method

Abstract

To provide a technique that can synchronize signals received through different signal paths even if the signals are not provided with time information.SOLUTION: A computer system has: an image forming unit that forms an image on the basis of the waveforms, in a time frame in a predetermined period, of a plurality of input signals in which a plurality of time-series signals are delayed; a convolution neural network unit that causes a convolution neural network to extract the characteristics of the image; a synchronization determination unit that, on the basis of the characteristics, specifies an initial time relationship that is a time relationship before the delay of the plurality of input signals; and a synchronization reproduction unit that generates a plurality of time-series signals in which the time relationship between the plurality of input signals is returned to the initial time relationship.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a technique for processing a time-series signal collected using a communication means. [Background technology]

[0002] In recent years, attempts have been made to reduce the number of workers, automate operations, and remotely operate industrial and social infrastructure by utilizing the evolution of information and communication networks, as typified by the fifth-generation mobile communication system (5G).

[0003] One of these efforts is the digital twin in the manufacturing industry, which is a digital representation of the characteristics, status, and functions of real-world objects, such as production machines in a factory or aircraft engines, mapped into cyberspace.

[0004] Digital twins enable advanced engineering that links the real world and the cyber world. For example, by using digital twins to analyze and predict objects in cyberspace, it is possible to minimize design rework, verify advanced and diverse faults, and optimize maintenance cycles by predicting deterioration based on sensor information.

[0005] Until now, digital twins have been primarily studied in the manufacturing industry, and have mainly involved mapping objects installed in fixed locations within a factory in cyberspace. However, in the future, it is expected that a large number of people and things that share time and space information with the real world will be simultaneously turned into digital twins, and by handling a large number of digital twins in cyberspace in a coordinated manner, it will be possible to realize social innovation through production, transportation, and logistics in which people and things work together.

[0006] In general, many things that exist in the real world operate while sharing a common space and a common time. In order to reproduce these many things as digital twins in cyberspace and operate these many digital twins in cooperation, it is necessary to share a common time coordinate, i.e., synchronized time information, between the many digital twins. In this case, since various signals acquired by sensors in the real world have different delay amounts until they reach a server via a signal path, in order to reproduce various events that occurred in the real world in cyberspace in the correct order while maintaining the time relationship, it is required to restore the mutual time relationship changed by the delay amount generated in the real world before applying it to cyberspace. Regarding this, Patent Document 1 proposes a method of sharing synchronized time information between many digital twins by adding time information to signals transmitted by each device in the real world. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent Publication No. 2022-55721 Summary of the Invention [Problem to be solved by the invention]

[0008] However, as with the technique of Patent Document 1, it is not easy to provide a function for adding accurate time information to signals transmitted to a large number of devices in the real world.

[0009] One objective of the present disclosure is to provide a technique that allows signals received via different signal paths to be synchronized even if they are not provided with time information. [Means for solving the problem]

[0010] One embodiment of a computer system included in the present disclosure includes an image forming unit that forms an image based on waveforms of a plurality of input signals in which a plurality of time-series signals are respectively delayed over a predetermined time frame; a convolutional neural network unit that extracts features of the image using a convolutional neural network; a synchronization determination unit that identifies an initial time relationship, which is the time relationship before the plurality of input signals are delayed, based on the features; and a synchronization reproduction unit that generates a plurality of time-series signals in which the time relationship of the plurality of input signals is returned to the initial time relationship. Effect of the Invention

[0011] According to one aspect of a computer system included in the present disclosure, the mutual temporal relationship at the time of occurrence of multiple time-series signals is identified from an image based on the waveforms of the multiple time-series signals, making it possible to synchronize the multiple time-series signals. [Brief description of the drawings]

[0012] [Figure 1] 1 is a configuration block diagram of a time-series information processing system according to a first embodiment. [Diagram 2] FIG. 2 is a conceptual diagram showing image formation and synchronous reproduction. [Diagram 3] 1 is a flowchart of a process executed by the time-series information processing system. [Figure 4] FIG. 1 is a configuration block diagram of a time series information processing system that performs learning using a pseudo-delay signal. [Diagram 5] FIG. 1 is a configuration block diagram of a time-series information processing system capable of specifying a situation. [Figure 6] FIG. 13 is a configuration block diagram of a time-series information processing system according to a fourth embodiment. [Figure 7] FIG. 13 is a diagram illustrating the configuration of a smart factory according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The examples are illustrative for explaining the present invention, and are omitted and simplified as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0014] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0015] In the embodiment, a process performed by an information processing system (computer) executing a program may be described. Here, the computer may execute the program using a processor (e.g., GPU, etc.) and perform the process defined in the program while using a storage resource (e.g., memory) and an interface device (e.g., communication port), etc. Therefore, the subject of the process performed by executing the program may be the processor. Similarly, the subject of the process performed by executing the program may be a controller, device, system, computer, or node having a processor. The subject of the process performed by executing the program may be a calculation unit, and may include a dedicated circuit that performs a specific process. Here, the dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device), etc.

[0016] The program may be installed in the computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and a storage resource that stores the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in the embodiment, two or more programs may be realized as one program, and one program may be realized as two or more programs. EXAMPLES

[0017] <Configuration of time series information processing system> FIG. 1 is a block diagram showing the configuration of a time-series information processing system according to a first embodiment.

[0018] The time-series information processing system 100 is an information processing system that acquires time-series data related to a controlled object 900 by a sensor 101, analyzes the data, and generates a control signal 108 to the controlled object 900. The controlled object 900 is, for example, an infrastructure facility in which people and things existing in real space work together.

[0019] The time-series information processing system 100 includes a sensor 101, an image forming unit 102, a convolutional neural network unit 103, a synchronization determination unit 104, a synchronization reproduction unit 105, a recurrent neural network unit 106, and a learning unit 107. The image forming unit 102, the convolutional neural network unit 103, the synchronization determination unit 104, and the synchronization reproduction unit 105 are used to determine the synchronization of a plurality of input signals and reproduce a time-series signal in which the original time relationship has been restored. The recurrent neural network unit 106 and the learning unit 107 are used to generate a desired control signal for controlling the controlled object 900 from the time-series signal in which the synchronization has been reproduced. The configuration for reproducing the original time-series signal from the input signal and the configuration for generating a desired control signal from the time-series signal are realized as an appropriate combination.

[0020] A plurality of time-series signals simultaneously output from a plurality of sensors 101 are input to the image forming unit 102 as input signals via different signal paths. The sensor 101 may be any sensor that outputs a time-series signal, such as an image sensor, an acoustic sensor, a vibration sensor, an olfactory sensor, or a taste sensor, without any particular limitations. The signal path may be any path, such as a wired path, a wireless path, or in particular a 5G communication path. Since the signal paths have different delay times, the mutual time relationships of the time-series signals from the plurality of sensors 101 may have changed by the time they reach the image forming unit 102.

[0021] The image forming unit 102 forms an image based on the waveforms of a plurality of input signals in a time frame (time window) of a predetermined period (time width). Here, as an example, the image forming unit 102 cuts out the waveforms of a timing chart of a plurality of input signals, for example, with the horizontal axis representing time and the vertical axis representing signal strength, in a certain time frame and creates an image. In this embodiment, a 64-channel MSO58B mixed signal oscilloscope manufactured by Tektronix is ​​used as the image forming unit 102. The image formed by the image forming unit 102 is input to the convolutional neural network unit 103. "Tektronix" is a registered trademark. The time width of the time frame for cutting out the input signal in the image forming unit 102 may be adjustable. An appropriate time width may be set depending on the clock width of the time series signal transmitted from a sensor in the real world to which the time series information processing system 100 is applied, the delay amount of the signal path, the real-time property required for the control signal 108, and the like.

[0022] The convolutional neural network unit 103 extracts features from the image. Information on the features extracted by the convolutional neural network unit 103 is provided to the synchronization determination unit 104. Based on the provided features, the synchronization determination unit 104 classifies the image into waveforms of time-series signals having a time relationship at the time when the images are simultaneously output from the multiple sensors 101. As an example, the convolutional neural network unit 103 and the synchronization determination unit 104 are neural networks that have been trained in advance to classify an image into a class representing multiple time-series signals having a time relationship before the change (initial time relationship) based on image features based on the waveforms of multiple time-series signals whose mutual time relationship has changed.

[0023] The convolutional neural network unit 103 and the synchronization determination unit 104 are trained in advance using time-series signals simultaneously output from the multiple sensors 101 and input signals from these time-series signals input to the image formation unit 102 via signal paths as training data. The convolutional neural network unit 103 and the synchronization determination unit 104 extract features from images and classify the images into classes based on the extracted features. The determination result of the synchronization determination unit 104 in classifying the images into classes is provided to the synchronization playback unit 105.

[0024] The synchronous reproduction unit 105 receives the determination result of the synchronization determination unit 104, and reproduces and outputs the time-series signals at the time when they are simultaneously output from the multiple sensors 101. The time-series signals reproduced by the synchronous reproduction unit 105 are input to the recurrent neural network unit 106.

[0025] The recurrent neural network unit 106 is an intermediate layer of an appropriately weighted recurrent neural network, which receives a time series signal from the synchronous reproduction unit 105 and inputs the time series information that has passed through the recurrent neural network to the learning unit 107. The recurrent neural network unit 106 is preferably a reservoir for reservoir calculation. In the reservoir calculation, the node connection strength of the reservoir, i.e., the weight of the neural network, is randomly determined.

[0026] The learning unit 107 is a learning device that has been trained in advance to output a control signal for performing appropriate control on the controlled object 900, and receives the output of the recurrent neural network unit 106 to generate a control signal 108. The control signal 108 is provided to the controlled object 900 and used for control.

[0027] Here, it is desirable for the configuration of the convolutional neural network unit 103, the synchronization determination unit 104, the recurrent neural network unit 106, and the learning unit 107 to be capable of performing neural network matrix calculations at high speed, and in this embodiment, a GPU (Graphics Processing Unit) Jetson AGX Xavier manufactured by NVIDIA, Inc. "NVIDIA" is a registered trademark.

[0028] In addition, the synchronization playback unit 105 uses the same Tektronix 64-channel MSO58B mixed signal oscilloscope as the image formation unit 102, and by correcting the synchronization shift of the time-series input signals recorded during image formation, it plays back and outputs the time-series signal waveforms at the time when they are output simultaneously from the multiple sensors 101.

[0029] FIG. 2 shows a conceptual diagram of image formation and synchronous playback.

[0030] The time series signals 201 output from the multiple sensors 101 are two-value digital signals of square waves taking values ​​of 1 or 0. Although a binary signal is exemplified here, the present invention is not limited thereto, and a multi-value digital signal taking multiple discrete values ​​can be handled in the same manner. When output from the sensor 101, each time series signal is synchronized, and the rising and falling times of each time series signal are the same. The time series signals 201 output simultaneously from the multiple sensors 101 pass through signal paths 202 with different signal delay amounts, causing a synchronization error, and reach the image forming unit 102. The image forming unit 102 forms an image 204 by cutting out the waveform of the input signal that has passed through the signal path 202 and has a synchronization error in a certain image acquisition time frame 203.

[0031] The convolutional neural network unit 103 extracts the features of the image 204, and the synchronization determination unit 104 identifies the time-series signal before the synchronization loss. Based on the features of the image 204, the convolutional neural network unit 103 and the synchronization determination unit 104 classify the image into a class that represents a binary combination of multiple time-series signals. The synchronization reproduction unit 105 reproduces the time-series signal 205 before the synchronization loss and outputs it to the recurrent neural network unit 106. FIG. 2 illustrates an example in which the image 204 is correctly classified into the class (1, 1, 0, 1, 1, 0, 1, 0, 1) and the time-series signal 205 in which the time-series signal 201 is correctly reproduced is generated.

[0032] As described above, the recurrent neural network unit 106 and the learning unit 107 generate the desired control signal 108 from the reproduced time-series signal 205. In this embodiment, one of VGG11, VGG16, RepVGG, and ResNet18 was selected and used as the convolutional neural network. In addition, the recurrent neural network unit 106 used a reservoir calculation with 1000 connection nodes.

[0033] <Time series information processing system> FIG. 3 shows a flowchart of the process executed by the time-series information processing system.

[0034] 3, first, the image forming unit 102 cuts out a plurality of input signals in an image acquisition time frame 203, forms an image 204, and outputs it to the convolutional neural network unit 103 (S301). The convolutional neural network unit 103 extracts features of the image 204 (S302).

[0035] The synchronization determination unit 104 determines the time-series signal before the synchronization loss from the features extracted from the image 204, and outputs the determination result to the synchronization reproduction unit 105 (S303). The synchronization reproduction unit 105 reproduces the time-series signal 205 before the synchronization loss based on the determination result of the synchronization determination unit 104, and outputs it to the recurrent neural network unit 106 (S304). The recurrent neural network unit 106 and the learning unit 107 generate a desired control signal 108 from the reproduced time-series signal 205 before the synchronization loss (S305). It is desirable that the convolutional neural network unit 103, the synchronization determination unit 104, the recurrent neural network unit 106, and the learning unit 107 are capable of performing matrix calculations required for the neural network at high speed, and in addition to the Jetson AGX Xavier, which is a GPU manufactured by NVIDIA, Jetson Orin and Jetson Orin Nano can be used.

[0036] According to the present embodiment described above, the mutual time relationships at the time of occurrence of multiple time series signals are identified from an image based on the waveforms of those signals, so that multiple time series signals can be synchronized even if time information is not attached.

[0037] In this embodiment, an example has been shown in which the time relationship of the time series signals acquired from the controlled object 900 is restored and analyzed, and a control signal for the controlled object 900 is generated, but other applications are also possible. For example, the behavior of the controlled object 900 can be simulated by restoring the time relationship of the time series signals acquired from the controlled object 900 in the real world and applying it to the digital twin of the controlled object 900, and generating a control signal to control the digital twin.

[0038] In this embodiment, the image forming unit 102, the convolutional neural network unit 103, the synchronization determination unit 104, the synchronization playback unit 105, the recurrent neural network unit 106, and the learning unit 107 included in the time-series information processing system 100 may be realized by a processor (e.g., a CPU or a GPU) executing a program stored in a volatile or non-volatile memory. They may also be realized by electronic circuit devices such as FPGA, ASIC, and CPLD. EXAMPLES

[0039] In the first embodiment, a configuration is exemplified in which the learning process for constructing the convolutional neural network unit 103 that performs image classification and the synchronization determination unit 104 is performed in advance using time-series signals simultaneously output from a plurality of sensors 101 and input signals input to the image forming unit 102 from the time-series signals via a signal path as learning data. In the second embodiment, a time-series information processing system having a function for executing the learning process for constructing the convolutional neural network that performs image classification and the synchronization determination unit is exemplified. The time-series information processing system of the second embodiment performs the learning process using pseudo-delayed time-series signals as learning data.

[0040] In order to improve the accuracy of feature extraction by the convolutional neural network unit 103 and the accuracy of judgment by the synchronization judgment unit 104, it is necessary to train the convolutional neural network unit 103 and the synchronization judgment unit 104 using a large amount of data before and after delays occur in the signal paths of time-series signals that reach the image forming unit 102 via the signal paths from the multiple sensors 101 as training data. However, it is not easy to obtain training data using the actual multiple sensors 101 and communication paths. In this regard, the time-series information processing system 100 of this embodiment can generate training data by itself and train the convolutional neural network unit 103 and the synchronization judgment unit 104, so that the accuracy of feature extraction by the convolutional neural network unit 103 and the accuracy of judgment by the synchronization judgment unit 104 can be easily improved.

[0041] FIG. 4 shows a configuration block diagram of a time series information processing system that performs learning using a pseudo-delay signal.

[0042] The time series information processing system 100 of the second embodiment is obtained by adding a learning processing function to the time series information processing system 100 of the first embodiment. Therefore, the time series information processing system 100 of the third embodiment includes all the configurations of the time series information processing system 100 of the first embodiment shown in Fig. 1. The time series information processing system 100 of the second embodiment differs from the time series information processing system 100 of the first embodiment only in the parts related to the learning processing, and therefore Fig. 4 illustrates only the parts related to the learning processing of the time series information processing system 100 of the second embodiment.

[0043] 4, the time-series information processing system 100 of the second embodiment is configured by adding a pseudo delay generating unit 401 and a learning processing unit 402 to the time-series information processing system 100 of the first embodiment.

[0044] The pseudo delay generation unit 401 is provided between the multiple sensors 101 and the image forming unit 102. The pseudo delay generation unit 401 generates learning data by generating delays of various delay times in place of the signal path for the time-series signal from the sensor 101. For example, a delay of a random amount within a given range may be generated.

[0045] The learning processing unit 402 performs learning of the convolutional neural network unit 103 and the synchronization determination unit 104 using the time-series signal before and after the delay is applied by the pseudo delay generation unit 401. That is, the learning processing unit 402 learns the convolutional neural network from the image formed by the image forming unit 102 based on the time-series signal after the delay so that the results determined by the convolutional neural network unit 103 and the synchronization determination unit 104 match the time-series signal before the delay is applied, and applies the learning result to the convolutional neural network unit 103 and the synchronization determination unit 104.

[0046] This makes it possible to generate a large amount of learning data in a shorter time than if learning data were generated using actual multiple sensors 101 and communication paths, improving the accuracy of judgment in the synchronization judgment unit 104 and enabling the synchronization reproduction unit 105 to reproduce a time series signal with high accuracy.

[0047] In this embodiment, the pseudo delay generation unit 401 is a Tektronix 64-channel MSO58B mixed signal oscilloscope, the same as that used in the image forming unit 102. Specifically, as a result of applying this embodiment to a plurality of vibration sensors, the number of pieces of learning data can be increased from about 100 to about 10,000, and the probability that the original time series signal is correctly reproduced by the synchronous reproduction unit 105 can be improved from about 85% to 98%. In addition, while it takes about one week to generate about 100 pieces of learning data using the actual plurality of sensors 101 and communication paths, by using the pseudo delay generation unit 401 having the above configuration, it is possible to generate about 10,000 pieces of learning data in about one hour, thereby shortening the time required for generating the learning data. EXAMPLES

[0048] In the first embodiment, a time series information processing system 100 is illustrated that generates a control signal 108 to a controlled object 900 from a time series signal related to the controlled object 900 acquired by a sensor 101. In the third embodiment, a time series information processing system is illustrated that generates a control signal to a controlled object by using a situation designation signal representing a static situation in addition to the time series signal acquired by a sensor. In the real world, there are not only things that change from moment to moment, but also things that exhibit several static states. According to the time series information processing system of this embodiment, by designating a static situation, it is possible to generate a control signal to be given to a controlled object in that situation.

[0049] FIG. 5 is a configuration block diagram of a time-series information processing system capable of specifying a situation.

[0050] The time series information processing system 100 of the third embodiment is obtained by adding a function of specifying a situation to the system of the first embodiment, and using information on the situation to generate a control signal. Therefore, the time series information processing system 100 of the third embodiment includes all the configurations of the time series information processing system 100 of the first embodiment shown in FIG. 1. The time series information processing system 100 of the third embodiment further includes a situation specification unit 501 in addition to the system of the first embodiment, and a situation specification signal 502 from the situation specification unit 501 is input to the recurrent neural network unit 106. The time series information processing system 100 switches the control signal to be output in response to the specification by the situation specification signal.

[0051] The situation designation unit 501 is configured to allow a user to select a plurality of situations, and outputs a situation designation signal 502 that indicates the situation selected by the user from among the plurality of situations.

[0052] In this embodiment, the recurrent neural network unit 106 constitutes a reservoir in the reservoir calculation. In addition to the time-series signal reproduced by the synchronous reproduction unit 105, a situation designation signal 502 from a situation designation unit 501 is input to the recurrent neural network unit 106.

[0053] For example, a multi-value signal expressing a value according to the number of situations can be used as the situation designation signal 502. Specifically, if the number of selectable situations is three, a DC signal with three different values ​​of height represented by signal strength such as voltage can be used.

[0054] Reservoir calculation learning unit 107, which has a reservoir in recurrent neural network unit 106, learns learning data such that if situation designation signal 502 is different, control signal 108 output corresponding to the time-series signal from synchronous playback unit 105 will be different. In this way, even if the time-series signal 205 from synchronous playback unit 105 is the same, it is possible to output a different control signal 108 corresponding to situation designation signal 502 by switching situation designation signal 502.

[0055] As a concrete example, five different control signals can be switched and output using a situation designation signal 502 capable of designating five different situations for a reservoir calculation with 1000 connected nodes. EXAMPLES

[0056] In the first embodiment, the reservoir of the reservoir calculation is exemplified as a suitable example of the recurrent neural network unit 106. In the reservoir calculation, the node connection strength of the reservoir, which is the intermediate layer, is set to a random value of normal distribution. In this regard, in the fourth embodiment, an example is shown in which the node connection strength of the reservoir is set to a random value of power distribution.

[0057] FIG. 6 is a block diagram showing the configuration of a time-series information processing system according to a fourth embodiment.

[0058] 6, the time-series information processing system 100 of the fourth embodiment has a configuration basically similar to that of the first embodiment shown in Fig. 1. However, in the fourth embodiment, as shown in a weight frequency distribution graph 601, a random value of the power-law distribution is set as the node connection strength in the reservoir of the recurrent neural network unit 106.

[0059] By using a power-law distribution with a narrower tail than a normal distribution as the node coupling (weight) distribution for reservoir calculation, it becomes possible to stabilize the operation against a decrease in the reproduction accuracy of synchronization in the synchronous reproduction unit 105. For example, even if the reproduction accuracy of the synchronous reproduction unit 105 decreases and the synchronization of the time-series signal 205 temporarily deviates, it becomes possible to maintain and stably generate the desired control signal 108. EXAMPLES

[0060] As a fifth embodiment, a smart factory to which the time-series information processing system shown in the first embodiment etc. is applied will be described.

[0061] FIG. 7 shows the configuration of a smart factory according to the fifth embodiment.

[0062] 7 shows things that constitute a real space 701 in a smart factory 700 and a time-series information control system that realizes a cyberspace 702. The real space 701 contains components such as people, robots, mobile objects, and devices that exist in an actual factory, and each component is equipped with a sensor 101. Time-series signals from the multiple sensors 101 reach the image forming unit 102 via a signal path 202, which is, for example, a 5G communication path. When the time-series signals that were synchronized in the real space 701 reach the image forming unit 102, they are out of sync due to different delays.

[0063] An image 204 formed by the image forming unit 102 is input to a pre-trained convolutional neural network unit 103. The convolutional neural network unit 103 extracts features from the image 204. The features extracted by the convolutional neural network unit 103 are input to a synchronization determination unit 104. The synchronization determination unit 104 classifies the image into classes based on the features. The classification results are input to a synchronization reproduction unit 105. The synchronization reproduction unit 105 reproduces a synchronized time series signal based on the classification results. The synchronized time series signal is input to a recurrent neural network unit 106. The recurrent neural network unit 106 and the learning unit 107 generate a control signal 108 based on the time series signal.

[0064] The control signal 108 output from the learning unit 107 is fed back via a feedback path 703 to humans, robots, moving objects, devices, etc. in the real space to control each of them.

[0065] As described above, in order to reproduce the synchronization of time-series signals in real space in cyberspace, convert the time-series signals into control signals, and feed the control signals back to real space to perform real-time control, it is necessary to shorten as much as possible the time required for the synchronization and reproduction of time-series signals in cyberspace and the conversion into control signals.

[0066] In this embodiment, a convolutional neural network and a synchronization judgment unit that have been unstructured and pruned are used as the convolutional neural network unit 103 and the synchronization judgment unit 104. Specifically, RepVGG is used as the convolutional neural network, and the RepVGG is unstructured pruned to a pruning rate of 98% using the Synapyic low method, and then compressed by the unstructured pruning compression compiler SparseRT that is incorporated into the structural pruning compression compiler TensorRT and runs on a GPU, and then implemented on the NVIDIA GPU Jetson AGX Xavier. As a result, the time required to generate a control signal from a time-series signal in the cyberspace 702 is reduced to 5 milliseconds or less. This enables real-time feedback control that emphasizes people, robots, moving objects, and devices in the smart factory, and allows people, robots, moving objects such as automated product transport vehicles to work smoothly in the factory without colliding with each other, improving the operating rate of the smart factory.

[0067] The above-described embodiments include the following items. However, the items included in each embodiment are not limited to those listed below.

[0068] [Matter 1] A computer system comprising: an image forming unit that forms an image based on the waveforms of a plurality of input signals, each of which is delayed, in a time frame of a predetermined period, a convolutional neural network unit that extracts features of the image using a convolutional neural network, a synchronization determination unit that specifies an initial time relationship, which is a time relationship before the delay of the plurality of input signals, based on the features, and a synchronization reproduction unit that generates a plurality of time-series signals in which the time relationship of the plurality of input signals is restored to the initial time relationship. This makes it possible to synchronize the plurality of time-series signals, since the mutual time relationship at the time of their generation is specified from an image based on the waveforms of the plurality of time-series signals.

[0069] [Matter 2] In the computer system described in item 1, the image forming unit forms, as the image, a timing chart that represents signal strengths of the multiple input signals against a time axis in the time frame. With this, the timing chart that represents the waveforms of the multiple input signals is used as an image to extract the features, so that it is possible to capture the mutual time relationship of the multiple input signals as features.

[0070] [Matter 3] In the computer system described in item 1, the convolutional neural network unit and the synchronization determination unit are neural networks that classify images into classes representing a plurality of time-series signals in the initial time relationship based on image features based on the waveforms of a plurality of time-series signals whose mutual time relationships have changed, and classify the images into the classes based on features extracted from images based on the waveforms of a plurality of the input signals. This makes it possible to easily identify the original time relationships of the plurality of time-series signals by a classification problem of classifying images into classes.

[0071] [Matter 4] In the computer system described in item 3, the time series signal is a digital signal expressing multiple discrete values ​​based on signal strength, and the convolutional neural network unit and the synchronization determination unit classify the image into a class representing a combination of multiple discrete values ​​of the multiple time series signals based on the features of the image.

[0072] [Matter 5] In the computer system described in item 1, the computer system further includes a recurrent neural network unit that inputs multiple time series signals generated by the synchronous playback unit into a recurrent neural network in which node connection strengths are determined in advance, and a learning unit that has been pre-trained to generate a desired control signal from the output of the recurrent neural network.

[0073] [Matter 6] In the computer system according to item 5, the recurrent neural network is a reservoir of reservoir calculation in which node connection strengths are randomly determined. This makes it easy to learn the recurrent neural network by using reservoir calculation.

[0074] [Matter 7] In the computer system according to item 6, the reservoir is a reservoir that is randomly determined so that the node connection strength is a power-law distribution. By making the node connection strength of the reservoir a power-law distribution, it is possible to generate a control signal with tolerance to the accuracy of reproduction of the time relationship of a time-series signal.

[0075] [Matter 8] In the computer system according to item 1, the image forming unit is capable of adjusting the duration of the time frame.

[0076] [Matter 9] The computer system according to item 1 further includes a pseudo-delay generating unit that adds a pseudo-delay to each of a plurality of time-series signals, and a learning processing unit that learns the convolutional neural network unit and the synchronization judgment unit so that the time relationship of the pseudo-delayed time-series signals can be identified based on an image formed by the image forming unit from the pseudo-delayed time-series signals, using a combination of a pseudo-delayed time-series signal that is a time-series signal before the pseudo-delay is added and a pseudo-delayed time-series signal that is a time-series signal after the pseudo-delay is added as learning data. This makes it possible to improve the accuracy of reproducing the time-series signals in a short time.

[0077] [Matter 10] In the computer system described in item 5, the recurrent neural network unit inputs the reproduced multiple time-series signals and a situation designation signal that selectively designates multiple static situations to the recurrent neural network.

[0078] [Matter 11] A control system comprising: a computer system according to item 5; a control object controlled by the control signal output from the computer system; and a sensor that acquires time series data related to the control object and transmits it to the computer system as the time series signal.

[0079] [Matter 12] A processing method comprising the steps of: forming an image based on waveforms of a plurality of input signals, each of which is delayed in a predetermined time frame, extracting features of the image using a convolutional neural network; specifying an initial time relationship, which is a time relationship before the delay of the plurality of input signals, based on the features; and generating a plurality of time-series signals in which the time relationship of the plurality of input signals is restored to the initial time relationship. [Explanation of symbols]

[0080] 100...time series information processing system, 101...sensor, 102...image forming unit, 103...neural network unit, 104...synchronization determination unit, 105...synchronization playback unit, 106...recurrent neural network unit, 107...learning unit, 108...control signal, 201...time series signal, 202...signal path, 203...image acquisition time frame, 204...image, 205...time series signal, 401...pseudo delay generation unit, 402...learning processing unit, 501...situation designation unit, 502...situation designation signal, 601...frequency distribution graph, 700...smart factory, 701...real space, 702...cyber space, 703...feedback path, 900...control target

Claims

1. an image forming unit that forms an image based on waveforms of a plurality of input signals in which a plurality of time series signals are delayed respectively, in a time frame of a predetermined period; a convolutional neural network unit that extracts features of the image using a convolutional neural network; a synchronization determination unit that determines an initial time relationship, which is a time relationship before the plurality of input signals are delayed, based on the characteristics; a synchronous reproduction unit that generates a plurality of time-series signals by returning the time relationship of the plurality of input signals to the initial time relationship; A computer system having:

2. 2. The computer system of claim 1, the image forming unit forms, as the image, a timing chart representing signal strengths of the plurality of input signals relative to a time axis in the time frame; Computer system.

3. 2. The computer system of claim 1, The convolutional neural network unit and the synchronization determination unit include: a neural network that classifies an image into a class representing a plurality of time-series signals having an initial time relationship based on a feature of the image based on waveforms of the plurality of time-series signals whose mutual time relationship has changed, classifying the images into the classes based on features extracted from the images based on the waveforms of the plurality of input signals; Computer system.

4. 4. The computer system according to claim 3, the time series signal is a digital signal that expresses a plurality of discrete values ​​according to signal strength, the convolutional neural network unit and the synchronization determination unit classify the image into classes representing combinations of a plurality of discrete values ​​of a plurality of the time-series signals based on the features of the image; Computer system.

5. 2. The computer system of claim 1, a recurrent neural network unit that inputs the plurality of time series signals generated by the synchronous reproduction unit to a recurrent neural network in which node connection strengths are determined in advance; a learning unit that has been trained in advance to generate a desired control signal from an output of the recurrent neural network; Further comprising Computer system.

6. 6. The computer system according to claim 5, The recurrent neural network is a reservoir of reservoir calculations in which node connection strengths are randomly determined. Computer system.

7. 7. The computer system according to claim 6, The reservoir is a randomly determined reservoir such that the node connection strengths are distributed according to a power law distribution. Computer system.

8. 2. The computer system of claim 1, The image forming unit is capable of adjusting the duration of the time frame. Computer system.

9. 2. The computer system of claim 1, a pseudo-delay generating unit that applies a pseudo-delay to each of the plurality of time-series signals; a learning processing unit that learns the convolutional neural network unit and the synchronization determination unit so that a time relationship of the pseudo pre-delay time series signal can be identified based on an image formed by the image forming unit from the pseudo post-delay time series signal, using a combination of a pseudo pre-delay time series signal, which is a time series signal before the pseudo delay is applied, and a pseudo post-delay time series signal, which is a time series signal after the pseudo delay is applied, as learning data; The computer system further comprises:

10. 6. The computer system according to claim 5, the recurrent neural network unit inputs the reproduced time-series signals and a situation designation signal that selectively designates a plurality of static situations to the recurrent neural network; Computer system.

11. A computer system according to claim 5 ; a control object that is controlled by the control signal output from the computer system; a sensor that acquires time-series data related to the controlled object and transmits the time-series data to the computer system; A control system having

12. forming an image based on waveforms of a plurality of input signals each having a delay in a predetermined time period; Extracting features of the image using a convolutional neural network; determining an initial time relationship, the initial time relationship being a time relationship before the delay of the plurality of input signals, based on the characteristics; generating a plurality of time-series signals by returning the time relationship of the plurality of input signals to the initial time relationship; Processing methods.

Images