Processing system, program, and processing method

Abstract

To provide a processing system and the like that can use machine learning to infer the state of an object from a spectrogram related to the phase of vibration information.SOLUTION: A processing system 100 includes an acquisition unit 111, a calculation unit 112, a storage unit 120, and an inference unit 113. The acquisition unit 111 acquires information on vibration of an object 10. The calculation unit 112 calculates a spectrogram related at least to a phase. The storage unit 120 stores a learned model 130 that is trained to infer, from the spectrogram, a state related to at least one of condition monitoring, quality control, and predictive maintenance. The inference unit 113 inputs the spectrogram to the learned model 130 to infer the state.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a processing system, a program, a processing method, and the like.

Background Art

[0002] Patent Document 1 discloses an inspection method for inspecting an automotive part having an operating part such as a motor as an inspection object. In the inspection method, waveform data regarding the operating sound of the inspection object is acquired by a microphone arranged near the inspection object, a short-time Fourier transform is performed on the waveform data to generate a complex spectrogram, a predetermined phase feature amount is calculated based on the complex spectrogram, a group delay is calculated by performing a differential operation on the phase feature amount in the frequency direction, a smoothing filter process is performed on the group delay, and the intensity level of the abnormal sound component is calculated based on the group delay after the smoothing filter process.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, a group delay is calculated from a phase feature amount calculated based on a complex spectrogram, and the intensity level of an abnormal sound component is calculated based on the group delay. Patent Document 1 does not disclose or suggest a configuration for inferring the state of an object from a spectrogram regarding the phase of vibration information using machine learning.

Means for Solving the Problems

[0005] One aspect of the present disclosure relates to a processing system including an acquisition unit that acquires vibration information of an object, a calculation unit that calculates a spectrogram related to at least a phase based on the vibration information, a storage unit that stores a learned model learned to infer a state related to at least one of state monitoring, quality control, and predictive maintenance from the spectrogram, and an inference unit that infers the state by inputting the spectrogram into the learned model.

[0006] Another aspect of the present disclosure relates to a program that causes a computer to function as an acquisition unit that acquires vibration information of an object, a calculation unit that calculates a spectrogram related to at least a phase based on the vibration information, and an inference unit that infers the state by inputting the spectrogram into a learned model learned to infer a state related to at least one of state monitoring, quality control, and predictive maintenance with respect to the spectrogram.

[0007] Still another aspect of the present disclosure relates to a processing method that acquires vibration information of an object, calculates a spectrogram related to at least a phase based on the vibration information, and infers the state by inputting the spectrogram into a learned model learned to infer a state related to at least one of state monitoring, quality control, and predictive maintenance with respect to the spectrogram.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0009] Hereinafter, preferred embodiments of the present disclosure will be described in detail. Note that the embodiments described below do not unduly limit the content described in the claims, and not all of the configurations described in these embodiments are essential constituent elements.

[0010] 1. Processing system FIG. 1 is an explanatory diagram of a sensor that detects the vibration of an object. The object 10 includes, for example, a vibration source 11 that generates vibration by mechanical operation. The sensor 140 detects the vibration of the object 10 generated by the vibration source 11. The vibration source 11 is, as an example, a motor, an engine, a turbine, or the like. Further, the object 10 may not include the vibration source 11, and vibration may be applied to the outside of the object 10, and the sensor 140 may detect the vibration.

[0011] The object 10 is, for example, the vibration source 11 itself, that is, a motor, an engine, a turbine, or the like. Alternatively, the object 10 is a machine, device, or apparatus including the vibration source 11, such as a home appliance or industrial equipment such as a printer, an air conditioner, a robot, a pump, a belt conveyor, or a processing device, a moving body such as an automobile or an airplane, or an industrial facility such as a generator or a manufacturing plant. Alternatively, the object 10 may be a structure that vibrates by an external force, such as a building, a road, or a bridge.

[0012] Sensor 140 detects acceleration, velocity, displacement, angular acceleration, angular velocity, or an angle, and outputs a signal indicating the detected physical quantity as vibration information. Sensor 140 may be a sensor that detects one type of physical quantity, or a sensor that detects multiple types of physical quantities. Also, Sensor 140 may be a sensor that detects a physical quantity on one axis, or alternatively, a sensor that detects a physical quantity on two or more axes. Sensor 140 outputs vibration information from one or more channels. One channel means a channel that outputs a signal of one type of physical quantity on one axis.

[0013] Sensor 140 is an acceleration sensor or a gyro sensor that uses a crystal oscillator as a detection element, or an acceleration sensor or a gyro sensor that uses MEMS as a detection element, etc. Also, Sensor 140 may be an IMU in which an acceleration sensor and a gyro sensor are combined and unitized. Sensor 140 may detect velocity or displacement by integrating the acceleration detected by the detection element, or may use a detection element that detects velocity or the like. Sensor 140 may detect angular acceleration or an angle by differentiating or integrating the angular velocity detected by the detection element, or may use a detection element that detects angular acceleration or the like. An example of an acceleration sensor is a sensor that utilizes the change in vibration frequency according to the stress applied to a crystal oscillator and detects the acceleration by measuring the vibration frequency. An example of a gyro sensor is a sensor that detects the angular velocity by detecting the Coriolis force applied to a crystal oscillator. Another example of an acceleration sensor or a gyro sensor is a sensor that is composed of a mass part and electrodes by MEMS and detects the acceleration or the angular velocity by detecting the capacitance between the electrodes that changes according to the inertial force applied to the mass part.

[0014] Although the sensor 140 is assumed to be attached so as to contact the object 10, it is not limited thereto, and it is sufficient that vibration is transmitted from the object 10 to the sensor 140. One sensor unit may be attached to the object 10, or a plurality of sensor units may be attached to the object 10. The plurality of sensor units may be attached to different positions of the object 10, or may be attached to the same position of the object 10.

[0015] FIG. 2 is a configuration example of the processing system. The processing system 100 infers the state of the object by analyzing the vibration information of the object. Details of the state will be described later. The processing system 100 includes a processing unit 110, a storage unit 120, a sensor 140, and a presentation unit 150. Note that the sensor 140 may be provided outside the processing system 100 and connected to the processing system 100 via a cable, a network, or the like.

[0016] The processing unit 110 includes an acquisition unit 111, a calculation unit 112, and an inference unit 113.

[0017] The acquisition unit 111 acquires the vibration information of the object from the sensor 140 by receiving the physical quantity signal output by the sensor 140. The sensor 140 may output either an analog signal or digital data. When receiving an analog signal, the acquisition unit 111 may include an A / D converter that converts the analog signal into digital data. The acquisition unit 111 outputs vibration data, which is the vibration information of the object, to the calculation unit 112. The vibration data is time-series data of acceleration, velocity, displacement, angular acceleration, angular velocity, or angle detected by the sensor 140.

[0018] The calculation unit 112 calculates a spectrogram related to the phase of the vibration from the vibration data. The calculation unit 112 may further calculate a spectrogram related to the amplitude or power of the vibration from the vibration data.

[0019] Fig. 3 shows an example of a spectrogram. A spectrogram is a representation of the temporal evolution of the spectrum of a time series signal, obtained by successively shifting the time window of the signal and computing the spectrum at each position. In general, the horizontal axis represents time, the vertical axis represents frequency, and the signal intensity is represented by color or shading. A spectrum is the distribution of a time series signal decomposed into its frequency components according to a given rule such as Fourier transform, and represented as a function of frequency.

[0020] More specifically, a spectrogram is a two-dimensional data set in which data is arranged at each point on the time axis and the frequency axis. A column of data along the frequency axis at a given time represents the spectrum of the vibration at that time. A spectrogram is generated by acquiring spectra in a time series. In an amplitude spectrogram, the data at each point is amplitude data, and in a phase spectrogram, the data at each point is phase data. When multiple spectrograms are acquired, for example, when using a multi-axis sensor, the multiple spectrograms may be stacked in the depth direction to form three-dimensional data.

[0021] Fig. 3 shows a phase spectrogram as an example of a spectrogram related to phase. A phase spectrogram represents the frequency characteristics of the phase in one-channel vibration data. A spectrogram related to phase may be a cross-phase spectrogram or a cross-power spectrogram. A cross-phase spectrogram represents the frequency characteristics of the phase difference between channels of multiple channels as an angle. A cross-power spectrogram represents the frequency characteristics of the phase difference and magnitude (amplitude) between channels of multiple channels. It is obtained by calculating the product of the Fourier transform of the vibration data of one channel and the complex conjugate of the Fourier transform of the vibration data of the other channel.

[0022] FIG. 3 shows an amplitude spectrogram as an example of a spectrogram related to amplitude or power. The amplitude spectrogram represents the frequency characteristics of the amplitude in one-channel vibration data. The spectrogram related to amplitude or power may be a power spectrogram or a cross-power spectrogram. The power spectrogram represents the frequency characteristics of the power in one-channel vibration data. The cross-power spectrogram represents the frequency characteristics of the power correlation between channels of multiple channels.

[0023] In addition, as a spectrogram obtained from multiple channels, there is a coherence spectrogram other than the above. The coherence spectrogram is obtained by normalizing the cross-power spectrogram with the product of the powers of the respective signals.

[0024] Details of the method for calculating each of the above-mentioned various spectrograms from vibration data will be described later.

[0025] The calculation unit 112 converts the spectrogram into image data using a color map or the like, and outputs the image data to the inference unit 113. In the spectrogram converted into image data, each point on the time axis and the frequency axis corresponds to a pixel. The color map is a map that associates data values in the spectrogram with color data.

[0026] The number of pixels in the frequency direction of the spectrogram may be larger than the number of pixels in the time direction. For example, the calculation unit 112 may generate a spectrogram in which the number of pixels in the frequency direction is the same as the number of pixels in the time direction, and compress the spectrogram in the time direction. Alternatively, the calculation unit 112 may generate a spectrogram in which the number of pixels in the frequency direction is larger than the number of pixels in the time direction from the vibration data. It is assumed that more information on the vibration state exists in the frequency direction than in the time direction. Therefore, it is desirable that the resolution in the frequency direction of the spectrogram is higher than the resolution in the time direction.

[0027] When converting a spectrogram related to phase into image data, the arithmetic unit 112 may use a cyclic color map. A cyclic color map is a color map in which the colors are the same at the starting point 0 and the ending point 2π of the phase range, and the colors are continuously connected at the starting point 0 and the ending point π. Although the phases close to 0 and those close to 2π have values that are far apart, they actually represent similar phases. By using a cyclic color map, it is possible to represent the phases close to 0 and those close to 2π with similar colors.

[0028] When calculating the amplitude spectrogram, the arithmetic unit 112 may use the amplitude values represented by logarithmic values. By using the logarithmically transformed amplitude, the inference accuracy of the vibration state can be improved as compared with the case where non-logarithmic amplitude values are used.

[0029] Note that the arithmetic unit 112 may output the spectrogram itself to the inference unit 113, regarding the spectrogram as image data without converting it with a color map. Further, the arithmetic unit 112 may convert the spectrogram related to phase into image data using a non-cyclic color map. Each pixel of the image data after conversion by the color map generally has three elements. However, when the value of the spectrogram is a complex number, it may be converted regarding the values corresponding to the real part and the imaginary part as image data having two elements. Alternatively, two grayscale images each having corresponding values may be used. Since a complex number includes phase information, the phase can be handled without explicitly calculating the phase θ. Since the number of elements is reduced, memory usage is saved.

[0030] The inference unit 113 receives at least a spectrogram related to phase from the calculation unit 112. The inference unit 113 infers the vibration state from the spectrogram using the learned model 130. The vibration state is a state related to at least one of state monitoring, quality control, and predictive maintenance of the object. Details of these states will be described later. The inference unit 113 inputs the spectrogram to the learned model 130 and causes the learned model 130 to output an inference result. The learned model 130 is, for example, a classifier that classifies the vibration state of the object into a plurality of vibration states, or a detector that detects an abnormality or a failure of the object. The learned model 130 may be the whole or a part of the model learned as a classifier or a detector. The inference result of the classifier is the probability of each vibration state, or a flag indicating which vibration state it is. The inference result of the detector is a flag indicating whether an abnormality or a failure has been detected.

[0031] The processing unit 110 outputs presentation information based on the inference result to the presentation unit 150. The presentation unit 150 presents the presentation information to the user. The presentation unit 150 is, for example, a display, a speaker, a lamp, or a vibrator. The presentation information is a numerical value, a character, a color, an image, a sound, a light, or a vibration. Note that the output unit 114 may store the classification result of the vibration state, the detection result of the vibration state, or the presentation information generated from them in the memory or the storage. The memory or the storage may be common to the following storage unit 120.

[0032] The memory unit 120 stores the learned model 130. The learned model 130 is generated by the learning system pre-training the model using the teacher data. The generated learned model 130 is stored in the memory unit 120. The learning system is a computer or a cloud system in which a plurality of computers are connected by a network or the like. The teacher data includes a plurality of spectrograms and the correct labels for each spectrogram. The correct label in the classifier is, for example, a flag indicating which of a plurality of vibration states it corresponds to. The correct label in the detector is, for example, a flag indicating whether the object is abnormal or faulty. Note that the learning system may perform fine-tuning using spectrograms after performing pre-training using general images. Further, fine-tuning may be performed when new data and labels are additionally obtained.

[0033] The learned model 130 is a neural network for image recognition using deep learning. The neural network for image recognition is, for example, a CNN or a ViT as an example. CNN is an abbreviation for Convolutional Neural Network. ViT is an abbreviation for Vision Transformer. Although Transformer is used in models in various fields, what uses it for image recognition is collectively called ViT.

[0034] As the hardware configuration of the processing system 100, various configurations may be adopted. The processing system 100 is, for example, a computer or a cloud system in which a plurality of computers are connected by a network or the like. The computer is not limited to a general-purpose one such as a personal computer, and may be a dedicated one for performing the vibration analysis of the present embodiment or one incorporated in a specific device or the like.

[0035] The processing unit 110 is, as an example, a processor. The processor includes, for example, one or more of a CPU, GPU, microcomputer, DSP, ASIC, or FPGA, etc. CPU is the abbreviation of Central Processing Unit. GPU is the abbreviation of Graphics Processing Unit. DSP is the abbreviation of Digital Signal Processor. ASIC is the abbreviation of Application Specific Integrated Circuit. FPGA is the abbreviation of Field Programmable Gate Array. The storage unit 120 stores a program in which the functions of each part of the processing unit 110 are described. The processor realizes the functions of each part of the processing unit 110 as processing by executing the program.

[0036] The processing unit 110 is not limited to the software processing as described above, and may be a circuit that hardware-implements the functions of each part. In that case, the storage unit 120 may not store a program.

[0037] The storage unit 120 is a memory or a register. The memory is a volatile memory such as a RAM, or a non-volatile memory such as an OTP memory or an EEPROM. RAM is the abbreviation of Random Access Memory. OTP is the abbreviation of One Time Programmable. EEPROM is the abbreviation of Electrically Erasable Programmable Read Only Memory.

[0038] Note that a non-temporary information storage medium, which is a computer-readable medium, may store the learned model 130 and the above program. The information storage medium is, for example, an optical disk, a memory card, a hard disk drive, or a non-volatile semiconductor memory.

[0039] 2. Processing Flow Figure 4 is an example of the first flow of the processes executed by the processing unit. In step S1, the acquisition unit 111 acquires vibration information of one channel from the sensor 140. In step S2, the calculation unit 112 calculates a spectrogram of the relative phase from the vibration information. In step S3, the inference unit 113 infers the vibration state by inputting the spectrogram of the relative phase into the learned model 130.

[0040] Note that the acquisition unit 111 may acquire vibration information of a plurality of channels at the same time. The calculation unit 112 may acquire a plurality of spectrograms of the relative phase by calculating the spectrogram of the relative phase from the vibration information of each channel. The inference unit 113 may infer the vibration state by inputting the spectrograms of the plurality of relative phases into the learned model 130.

[0041] In the flow of Figure 4, the spectrogram of the relative phase corresponds to the "spectrogram regarding the phase". Figure 5 is a diagram for explaining the relative phase.

[0042] The left figure of Figure 5 shows the signal SA1 and the signal SB1 included in the vibration data before being converted into the relative phase. The frequency of the signal SA1 is the same reference frequency as the vibration frequency of the vibration source. The frequency of the signal SB1 is a frequency other than the reference frequency, and here it is, for example, a frequency of 1 / 2 of the reference frequency.

[0043] When the calculation unit 112 generates a spectrogram from the vibration data, it performs Fourier transform or the like while shifting a short-time window in the time direction. The signals SA1 and SB1 are signals included in a certain short-time window, and the start point of the short-time window is set to time 0. In the example of the left figure of Figure 5, the signal SA1 is sin(2X - π), and the signal SB1 is (3 / 2)×sin(X). X is a parameter in the time direction. The phase π of the signal SA1 is an example and changes as the short-time window moves. The same applies to the phase 0 of the signal SB1.

[0044] Figure 5 shows signals SA2 and SB2 included in the vibration data after being converted into relative phases. The calculation unit 112 shifts signals SA1 and SB1 to signals SA2 and SB2 such that the phase of signal SA2 at time 0 becomes 0 while maintaining the phase relationship between signal SA1 and signal SB1. Signal SA2 is sin(2X), and signal SB2 is (3 / 2)×sin(X + π / 2). In this way, the phases of each frequency are relativized so that the phase of the reference signal becomes 0.

[0045] The following formula (1) shows the calculation formula for relative phase. CurrentPhase, RelativePhase, and fcurrent are parameters related to the signal to be converted into relative phase. CurrentPhase is the phase before conversion, RelativePhase is the relative phase after conversion, and fcurrent is the frequency of the signal. BasePhase and fbase are parameters related to the signal of the reference frequency. BasePhase is the phase before conversion, and fbase is the frequency of the signal.

[0046]

Equation

[0047] Applying the example of Figure 5 to the above formula (1) gives 0 - π×(1 / 2) = -π / 2, which matches the relative phase of signal SB2.

[0048] The learning stage of the learned model 130 used in the flow of Figure 4 will be described. The teacher data includes the spectrogram of the relative phase and the correct label. The learning system inputs the spectrogram of the relative phase into the model and feeds back to the model so that the error between the inference result output by the model and the correct label becomes small. The learning system trains the model by repeating this for a large number of spectrograms.

[0049] FIG. 6 is a second flow example of the process executed by the processing unit. In steps S11 and S12, the acquisition unit 111 acquires the first vibration information of the first channel and the second vibration information of the second channel from the sensor 140. The first vibration information and the second vibration information are vibration information at the same time. In step S13, the calculation unit 112 calculates a cross-phase spectrogram or a cross-power spectrogram from the first vibration information and the second vibration information. In step S14, the inference unit 113 infers the vibration state by inputting the cross-phase spectrogram or the cross-power spectrogram into the learned model 130. In the flow of FIG. 6, the cross-phase spectrogram or the cross-power spectrogram corresponds to the "spectrogram related to phase".

[0050] Note that the sensor 140 may include one sensor unit having a first channel and a second channel. Alternatively, the sensor 140 may include a first sensor unit having a first channel and a second sensor unit having a second channel. The first sensor unit and the second sensor unit may be provided at different positions of the object.

[0051] Note that the acquisition unit 111 may acquire vibration information of three or more channels. The calculation unit 112 may calculate a cross-phase spectrogram or the like from any two pieces of vibration information among the vibration information of three or more channels. The inference unit 113 may infer the vibration state from a plurality of cross-phase spectrograms or the like.

[0052] The learning stage of the learned model 130 used in the flow of FIG. 6 will be described. The teacher data includes a cross-phase spectrogram or a cross-power spectrogram and a correct label. The learning system inputs the cross-phase spectrogram or the cross-power spectrogram into the model, and feeds back to the model so that the error between the inference result output by the model and the correct label becomes small. The learning system trains the model by repeating this for a large number of spectrograms.

[0053] Figure 7 is a third flow example of the processing executed by the processing unit. In step S21, the acquisition unit 111 acquires one-channel vibration information from the sensor 140. In step S22, the calculation unit 112 calculates a spectrogram of the relative phase from the vibration information. This is referred to as the first spectrogram. Also, in step S23, the calculation unit 112 calculates an amplitude spectrogram or a power spectrogram from the vibration information. This is referred to as the second spectrogram. In step S24, the inference unit 113 infers the state of the vibration by inputting the first spectrogram and the second spectrogram into the learned model 130.

[0054] The learning stage of the learned model 130 used in the flow of Figure 7 will be described. The teacher data includes a spectrogram of the relative phase, an amplitude spectrogram or a power spectrogram, and a correct label. The learning system inputs the spectrogram of the relative phase and the amplitude spectrogram or the power spectrogram into the model, and performs feedback on the model so that the error between the inference result output by the model and the correct label is reduced. The learning system trains the model by repeating this for a large number of spectrograms.

[0055] Figure 8 is a fourth flow example of the processing executed by the processing unit. In steps S31 and S32, the acquisition unit 111 acquires the first vibration information of the first channel and the second vibration information of the second channel from the sensor 140. The first vibration information and the second vibration information are vibration information at the same time. In step S33, the calculation unit 112 calculates a spectrogram of the relative phase from the first vibration information. This is referred to as the first spectrogram. In step S34, the calculation unit 112 calculates a cross-power spectrogram, a cross-phase spectrogram, or a coherence spectrogram from the first vibration information and the second vibration information. This is referred to as the second spectrogram. In step S35, the inference unit 113 infers the state of the vibration by inputting the first spectrogram and the second spectrogram into the learned model 130.

[0056] In step S33, the arithmetic unit 112 may calculate a cross-phase spectrogram or a cross-power spectrogram from the first vibration information and the second vibration information as a first spectrogram related to the phase. In this case, in step S34, the arithmetic unit 112 calculates a second spectrogram different from the first spectrogram among the cross-power spectrogram, the cross-phase spectrogram, or the coherence spectrogram.

[0057] The arithmetic unit 112 may calculate a spectrogram of the relative phase from the second vibration information. This is set as the third spectrogram. The inference unit 113 may infer the vibration state by inputting the first to third spectrograms into the learned model 130.

[0058] The learning stage of the learned model 130 used in the flow of FIG. 8 will be described. The teacher data includes a spectrogram of the relative phase, a cross-power spectrogram, a cross-phase spectrogram, or a coherence spectrogram, and a correct label. The learning system inputs the spectrogram of the relative phase, the cross-power spectrogram, the cross-phase spectrogram, or the coherence spectrogram into the model, and feeds back to the model so that the error between the inference result output by the model and the correct label becomes small. The learning system trains the model by repeating this for a large number of spectrograms.

[0059] FIG. 9 is a configuration example of a two-stream learned model. In FIG. 7 or FIG. 8 described above, first and second spectrograms of different types are input into the learned model 130. In such a case, the inference accuracy can be improved by using a two-stream learned model.

[0060] The learned model 130 in FIG. 9 includes a first network 131, a second network 132, a combination processing unit 135, and a combination layer 136.

[0061] The inference unit 113 inputs the first spectrogram SPG1 to the first network 131 and inputs the second spectrogram SPG2 to the second network 132. The first network 131 outputs a first inference result NQ1 for the first spectrogram SPG1. The second network 132 outputs a second inference result NQ2 for the second spectrogram SPG2. Each network includes an input layer to which a spectrogram is input, an intermediate layer that performs calculations based on the output of the input layer, and an output layer that outputs an inference result based on the output of the intermediate layer. The inference results output from each network are intermediate information such as, for example, the feature amounts of an image. Each of the first network 131 and the second network 132 is a neural network for image recognition such as a CNN or a ViT.

[0062] The combination processing unit 135 combines the first inference result NQ1 and the second inference result NQ2 and outputs the combination result CPQ. Specifically, the combination processing unit 135 outputs CPQ = α × NQ1 + β × NQ2. α and β are real numbers.

[0063] The combination layer 136 outputs a state inference result STQ according to the input combination result CPQ. That is, the combination layer 136 is a network that converts the combination result CPQ, which is the feature amount of an image or the like, into the state inference result STQ. The inference result STQ becomes the output of the inference unit 113.

[0064] The teacher data in the learning stage includes the first spectrogram and the second spectrogram at the same time, and the correct label corresponding to these spectrograms. The learning system inputs the first spectrogram of the teacher data into the first network of the two-stream model, and inputs the second spectrogram of the teacher data into the second network of the two-stream model. The learning system performs feedback to the two-stream model based on the error between the inference result output by the two-stream model and the correct label of the teacher data. At this time, the parameters of the first network, the parameters of the second network, the parameters α and β of the combination process, and the parameters of the combination layer are updated. The learning system trains the two-stream model by repeating the above based on a large number of teacher data.

[0065] In the above, the learned model of the two-stream model has been described. However, it is also possible to configure the first and second spectrograms to be input into the learned model of one stream.

[0066] The processing system 100 of the present embodiment described above includes an acquisition unit 111, a calculation unit 112, a storage unit 120, and an inference unit 113. The acquisition unit 111 acquires vibration information of the object 10. The calculation unit 112 calculates at least a spectrogram related to the phase based on the vibration information. The storage unit 120 stores a learned model 130 that is learned to infer a state related to at least one of state monitoring, quality control, and predictive maintenance from the spectrogram. The inference unit 113 infers the state by inputting the spectrogram into the learned model 130.

[0067] In conventional vibration analysis, an amplitude spectrogram or a power spectrogram is often used, but it is impossible to detect a vibration state in which no change or only a small change occurs in the amplitude or power. According to the present embodiment, by inputting a spectrogram related to the phase into the learned model 130 to infer the vibration state, it becomes possible to detect also a vibration state in which no change or only a small change occurs in the amplitude or power. Thereby, the inference accuracy of the vibration state in state monitoring, quality control, and predictive maintenance can be improved. For example, it becomes possible to monitor a state that could not be detected conventionally, improve the accuracy of quality control, or predict and notify in advance before a failure or the like occurs.

[0068] Also, as described with reference to FIG. 5 and the like, the calculation unit 112 may calculate a spectrogram of the relative phase at each frequency based on the phase at a predetermined frequency as a spectrogram related to the phase.

[0069] When a phase spectrogram is created from one vibration data, since the phase of the signal within the window depends on the position of the window, it is basically random information. For this reason, the accuracy of inferring the vibration state from the phase spectrogram may be reduced. According to the present embodiment, since the phase of the signal within the window is relativized with respect to the reference, the vibration state is likely to appear in the phase spectrogram, and the accuracy of inferring the vibration state from the phase spectrogram is improved. This effect is also apparent from the comparison between (1) and (3) of FIG. 11 described later.

[0070] Also, as described with reference to FIG. 6 and the like, the acquisition unit 111 may acquire the first vibration information of the first channel and the second vibration information of the second channel. The calculation unit 112 may calculate a cross-phase spectrogram or a cross-power spectrogram of the first vibration information and the second vibration information as a spectrogram related to the phase.

[0071] The first vibration information of the first channel and the second vibration information of the second channel differ in the detection axis, detection position, or type of physical quantity. By using the cross-phase or cross-power of such two vibration information, it becomes possible to detect vibration states in which no change or only a small change occurs in the amplitude or power. Thereby, the inference accuracy of the vibration state in condition monitoring, quality control, and predictive maintenance can be improved.

[0072] Also, as described with reference to FIG. 7 and the like, the calculation unit 112 may calculate the amplitude spectrogram or power spectrogram of the vibration information as the second spectrogram. The learned model 130 may be learned to infer the state with respect to the spectrogram related to the phase and the second spectrogram. The inference unit 113 may infer the state by inputting the spectrogram related to the phase and the second spectrogram to the learned model 130.

[0073] According to the present embodiment, in addition to the spectrogram related to the phase, the amplitude spectrogram or power spectrogram is input to the learned model 130. Thereby, the inference accuracy of the vibration state can be further improved as compared with the case where only the spectrogram related to the phase is used.

[0074] Also, in the present embodiment, the intensity value in the second spectrogram may be logarithmic.

[0075] According to the present embodiment, the amplitude spectrogram or power spectrogram using the logarithmic intensity value is input to the learned model 130. By using the logarithmic intensity value, it becomes easy to detect a relative change even for a value with a small number of digits compared to the maximum value, and thereby the inference accuracy of the state is improved. This effect is also apparent from the comparison between (1) and (2) of FIG. 11 described later.

[0076] Also, as described with reference to FIG. 8 and the like, the acquisition unit 111 may acquire the first vibration information of the first channel and the second vibration information of the second channel. The calculation unit 112 calculates a spectrogram of the relative phase based on the first vibration information or the second vibration information, and calculates a second spectrogram different from the above spectrogram among the cross-power spectrogram, the cross-phase spectrogram, or the coherence spectrogram of the first vibration information and the second vibration information. The learned model 130 may be learned to infer a state with respect to the spectrogram of the relative phase and the second spectrogram. The inference unit 113 may infer a state by inputting the spectrogram of the relative phase and the second spectrogram to the learned model 130.

[0077] According to the present embodiment, in addition to the spectrogram of the relative phase, the second spectrogram obtained from the two pieces of vibration information is input to the learned model 130. Thereby, compared with the case of using only the spectrogram of the relative phase, the inference accuracy of the vibration state can be further improved.

[0078] Also, as described with reference to FIG. 9 and the like, the learned model 130 may include a first network 131 to which a spectrogram related to phase is input, a second network 132 to which the second spectrogram is input, and a combination layer 136 that outputs an inference result of a state based on a combination result of an output of the first network 131 and an output of the second network 132.

[0079] According to the present embodiment, two spectrograms of different types, such as a spectrogram related to phase and a spectrogram related to amplitude, are input to different networks, respectively. Since each network performs inference with only one type of spectrogram input, the inference accuracy can be improved compared to the case where a plurality of types of spectrograms are mixed and input to one network. This effect is also apparent from the comparison between "ResNet18" and "2-st-res18" in FIG. 12 described later.

[0080] Also, in this embodiment, the spectrogram is image data. The trained model 130 may be a CNN or a ViT.

[0081] Since the spectrogram is two-dimensional data, it can be treated as image data as it is or by being converted using a color map or the like. Thereby, it is possible to input the spectrogram into a deep learning neural network for image recognition and infer the vibration state.

[0082] Also, in this embodiment, the image data as the spectrogram may have more pixels in the frequency direction than in the time-axis direction.

[0083] It is assumed that there is more vibration state information in the frequency direction than in the time direction. For this reason, when the resolution in the frequency direction of the spectrogram is higher than the resolution in the time direction, the estimation accuracy is improved compared to the case where the resolution in the frequency direction and the resolution in the time direction are the same.

[0084] Also, in this embodiment, the vibration information may be acceleration, velocity, displacement, angular acceleration, angular velocity, or angle.

[0085] When the object 10 vibrates, the acceleration, velocity, displacement, angular acceleration, angular velocity, or angle changes. That is, by creating a spectrogram from these vibration states and inputting it into the trained model 130, the vibration state can be inferred.

[0086] Also, in this embodiment, the inference unit 113 may infer a classification result regarding the state.

[0087] Regarding each of state monitoring, quality control, and predictive maintenance, it is assumed that the object has a plurality of states. According to this embodiment, it is possible to classify which of the plurality of states the object belongs to from the spectrogram.

[0088] Further, this embodiment may be implemented as a program that causes a computer to function as the acquisition unit 111, the calculation unit 112, and the inference unit 113. The acquisition unit 111 acquires vibration information of an object. The calculation unit 112 calculates a spectrogram related at least to the phase based on the vibration information. The inference unit 113 infers the state by inputting the spectrogram into the learned model 130 that is learned to infer the state related to at least one of state monitoring, quality control, and predictive maintenance for the spectrogram.

[0089] Further, this embodiment may be implemented as a processing method. The processing method acquires vibration information of an object. The processing method calculates a spectrogram related at least to the phase based on the vibration information. The processing method infers the state by inputting the spectrogram into the learned model 130 that is learned to infer the state related to at least one of state monitoring, quality control, and predictive maintenance for the spectrogram. The processing method causes a computer to perform these steps, for example.

[0090] 3. Experimental Example FIGS. 10 to 12 show experimental examples of vibration analysis using the processing system of this embodiment.

[0091] In this example, the processing system 100 classifies the six states A to F shown in FIG. 10. FIG. 10 shows a part of the device that is the object. The device includes a rotation axis along the z-axis direction and a rotating body fixed to the rotation axis and rotating in the xy plane. Although not shown, the device further includes a bearing that receives the rotation axis, a motor that rotates the rotation axis, and a housing that holds the bearing and the motor. The sensor may be mounted inside the device, for example, on the bearing, the motor, or the housing.

[0092] Weights can be attached to the rotating body. The weights a and b have different weights. The states A to F differ in the weight of the weight and the attachment position of the weight. The states A to F cause slightly different vibrations from each other, and the processing system 100 infers the state based on the difference.

[0093] Figure 11 compares the analysis results when the phase spectrogram and the amplitude spectrogram are input into the two-stream model for various analysis methods. The "Res" in the model name refers to the ResNet model of CNN. ResNet is the abbreviation of Residual Neural Networks. The number following Res indicates that the larger the number, the larger the scale of the network. The numbers in each column represent the correct rate as a percentage. For example, when the true state is A, it is considered correct if the processing system 100 can correctly estimate state A. The correct rate when this is done for multiple cases is shown.

[0094] (1) is the inference result when using the spectrogram of the relative phase and the amplitude spectrogram with the amplitude represented as a logarithmic value. (2) is the inference result when the amplitude in the amplitude spectrogram is not made logarithmic. (3) is the inference result when the phase of the phase spectrogram is not made relative. (4) is the inference result when the resolution in the frequency direction in the image data of the spectrogram is resized to half. From these results, it can be seen that the method of (1) has the highest classification accuracy. Also, from the comparison of (1) and (3), it can be seen that making the phase relative is very effective.

[0095] Figure 12 compares the analysis results when using the one-stream model and the two-stream model. "ResNet18" indicates the ResNet model of CNN. "2-st-res18" indicates the two-stream model using "ResNet18". "PRETRAIN" indicates having pre-training, and "NO-PRETRAIN" indicates no pre-training. From these results, it can be seen that the two-stream model has a higher classification accuracy than the one-stream model. Also, it can be seen that having pre-training results in a higher classification accuracy than not having pre-training.

[0096] 4. Examples of Vibration Analysis for Condition Monitoring, Quality Control, and Predictive Maintenance Hereinafter, as an example where the processing system 100 of the present embodiment can be applied, examples of vibration analysis related to condition monitoring, quality control, and predictive maintenance are shown.

[0097] Condition Monitoring refers to the technology of continuously or periodically monitoring the current operating state or performance of a machine or equipment. The main purpose of condition monitoring is to monitor specific parameters such as vibration, sound, temperature, or pressure, and evaluate the soundness of the machine or equipment. Examples of systems for performing condition monitoring include, for example, the following (a) to (f).

[0098] (a) A system that attaches sensors to industrial motors and issues an alarm when abnormal vibration or a temperature rise is detected.

[0099] (b) A system that installs vibration sensors on each part of a building and monitors the vibration during an earthquake or strong wind to monitor the soundness of the building structure.

[0100] (c) A system that has a vibration sensor attached to monitor the performance of a gas turbine and warns the operator when abnormal vibration is detected.

[0101] (d) A system that has a vibration sensor attached to monitor the soundness of heavy machinery in a mine or quarry and helps detect wear or failure at an early stage.

[0102] (e) A system that has a vibration sensor attached to monitor the soundness of each machine on a factory production line and issues an alarm when abnormal vibration is detected.

[0103] (f) A system that uses vibration sensors attached to the blades or gearboxes of wind power turbines to detect whether abnormal wear or damage has occurred.

[0104] Quality Control refers to the process of verifying whether a product or service meets the defined quality standards or requirements. Examples of systems for performing quality control include the following (g) to (k).

[0105] (g) A system that uses cameras or sensors on the production line to check the quality of products in real time, such as dimensions, color, or shape, and automatically rejects products that deviate from the standards.

[0106] (h) A system that checks the vibration pattern when a product is operated using a vibration sensor after the assembly process of a product composed of multiple parts. If abnormal vibration is detected, defects in assembly or malfunctions of parts are suspected.

[0107] (i) A system that operates a motor or generator after production and measures its vibration during operation using a vibration sensor. If vibration exceeding specific standards is detected, internal imbalance or damage is considered.

[0108] (j) A system that installs an electronic device on a vibration test bench and monitors the impact of vibration using a vibration sensor to verify whether a newly manufactured electronic device has the set vibration resistance.

[0109] (k) A system that monitors the vibration during operation using a vibration sensor to check whether a new type of railway vehicle or aircraft has the designed operating performance. If abnormal vibration is detected, the cause is identified and used to improve quality.

[0110] Predictive Maintenance refers to an approach that collects and analyzes the operation data or status data of equipment or machinery. The purpose of predictive maintenance is to predict the risk of future failures or performance degradation. Examples of systems for performing predictive maintenance include the following (l) to (q).

[0111] (l) A system that attaches sensors to industrial robots, analyzes the sensor data when the industrial robots are operating and the sensor data when they malfunctioned in the past, and detects signs indicating precursors of specific component failures, thereby predictively scheduling the replacement of those components.

[0112] (m) A system that uses sensors attached to the wheels of railway vehicles to monitor the degree of wear and predicts the replacement time of the wheels based on data indicating excessive wear.

[0113] (n) A system that monitors the condition of pipes or valves in oil plants using sensors and predicts the likelihood of future leaks.

[0114] (o) A system that analyzes sensor data when elevators or escalators are operating and predicts the risk of failure before component replacement becomes necessary.

[0115] (p) A system that monitors the efficiency of industrial cooling devices or air conditioners based on sensor data and predicts component wear or failure.

[0116] (q) A system that collects and analyzes sensor data when agricultural machinery is operating, predicts component wear or failure, and supports scheduling appropriate maintenance activities.

[0117] 5. Calculation methods for spectrograms As a first method, calculation methods for various spectrograms using Fourier transform are shown. Hereinafter, a method for calculating a spectrum from vibration data within a window is shown. A spectrogram is obtained by acquiring spectra in time series while moving the window in the time direction. For example, the short-time Fourier transform (STFT) using a window with a fixed width is used.

[0118] The power spectrum PWxy(f) is shown in the following formula (2). t is time, and x(t) is a signal as vibration information. f is frequency, and X(f) is the Fourier transform of x(t).

[0119]

Number

[0120] The cross-power spectrum CRxy(f) is shown in the following formula (3). x(t) is a signal as the first vibration information, and y(t) is a signal as the second vibration information. X(f) is the Fourier transform of x(t), and Y(f) is the Fourier transform of y(t). * indicates complex conjugate.

[0121]

Number

[0122] The above formula (3) can be rewritten as the following formula (4). The phase angle θ(f) of CRxy(f) represents the cross-phase spectrum. θ(f) is the phase difference between the signals x(t) and y(t) at frequency f. j is the imaginary unit.

[0123]

Number

[0124] The coherence spectrum CHxy(f) is shown in the following formula (5). <> represents an appropriate smoothing operation, for example, averaging in the time direction, frequency direction, or both time and frequency directions.

[0125]

Number

[0126] As a second method, an operation method of various spectrograms using wavelet transform is shown.

[0127] The following equation (6) shows the wavelet transform of the signal x(t). Wx(a,b) represents the wavelet transform of the signal x(t) at scale a and time point b. Ψ(t) represents the Morlet wavelet function. Ψ * represents the complex conjugate of the wavelet function. a is the scale parameter or dilation parameter, which controls the stretching and shrinking of the wavelet and corresponds to the frequency. b is the position parameter or transformation parameter of the transform, which controls the position of the wavelet and corresponds to time.

[0128]

Equation

[0129] The following equation (7) shows the Morlet wavelet function Ψ(t). ω0 specifies the center frequency. As an example, ω0 = 6, but it is not limited to this. Note that various functions such as the Haar wavelet function or the Daubechies wavelet function may be used as the wavelet function.

[0130]

Equation

[0131] The following equation (8) shows the arithmetic expression of the wavelet power spectrogram WPWx(a,b) corresponding to the "power spectrogram".

[0132]

Equation

[0133] The following equation (9) shows the arithmetic expression of the cross-wavelet spectrum WCRxy(a,b) corresponding to the "cross-power spectrogram". Wy(a,b) represents the wavelet transform of the signal y(t) at scale a and time point b.

[0134]

Equation

[0135] The above formula (9) can be rewritten as the following formula (10). The phase angle θ(a, b) of WCRxy(a, b) represents the cross-phase spectrum. θ(a, b) is the phase difference between the signals x(t) and y(t) at scale a and time point b.

[0136]

Equation

[0137] The following formula (11) shows the arithmetic expression of the wavelet coherence spectrogram WCH(a, b) corresponding to the "coherence spectrogram".

[0138]

Equation

[0139] Although the present embodiment has been described in detail as above, those skilled in the art will easily understand that many modifications can be made without substantially departing from the novel matters and effects of the present disclosure. Therefore, all such modified examples are intended to be included within the scope of the present disclosure. For example, in the specification or drawings, terms that have been described at least once together with broader or synonymous different terms can be replaced with those different terms at any location in the specification or drawings. Also, all combinations of the present embodiment and modified examples are included within the scope of the present disclosure. Further, the configurations or operations such as the processing unit, storage unit, learned model, sensor, processing system, object, vibration information, and spectrogram are not limited to those described in the present embodiment, and various modifications can be made.

Explanation of Reference Numerals

[0140] 10…Object, 11…Vibration source, 100…Processing system, 110…Processing unit, 111…Acquisition unit, 112…Calculation unit, 113…Inference unit, 120…Memory unit, 130…Trained model, 131…First network, 132…Second network, 135…Combination processing unit, 136…Combination layer, 140…Sensor, 150…Presentation unit

Claims

1. An acquisition unit that acquires vibration information of an object; An arithmetic unit that calculates at least a spectrogram related to phase based on the vibration information; A storage unit that stores a learned model learned to infer a state related to at least one of condition monitoring, quality control, and predictive maintenance from the spectrogram; An inference unit that infers the state by inputting the spectrogram into the learned model; A processing system characterized by including:

2. In the processing system according to Claim 1, The arithmetic unit calculates a spectrogram of relative phases at respective frequencies based on a phase at a predetermined frequency as the spectrogram related to the phase. A processing system characterized by this.

3. In the processing system according to Claim 1, The acquisition unit acquires first vibration information of a first channel and second vibration information of a second channel, The arithmetic unit calculates a cross-phase spectrogram or a cross-power spectrogram of the first vibration information and the second vibration information as the spectrogram related to the phase. A processing system characterized by this.

4. In the processing system according to Claim 1, The arithmetic unit calculates an amplitude spectrogram or a power spectrogram of the vibration information as a second spectrogram, The learned model is learned to infer the state with respect to the spectrogram and the second spectrogram, The inference unit infers the state by inputting the spectrogram and the second spectrogram into the learned model. A processing system characterized by this.

5. In the processing system according to Claim 4, A processing system characterized in that intensity values in the second spectrogram are logarithmic.

6. In the processing system according to Claim 1, The acquisition unit acquires first vibration information of a first channel and second vibration information of a second channel, The arithmetic unit calculates the spectrogram based on the first vibration information or the second vibration information, and calculates a second spectrogram different from the spectrogram among a cross-power spectrogram, a cross-phase spectrogram, or a coherence spectrogram of the first vibration information and the second vibration information. The learned model is trained to infer the state for the spectrogram and the second spectrogram. The inference unit infers the state by inputting the spectrogram and the second spectrogram into the learned model. A processing system characterized by this. **Claim 7** In the processing system according to any one of claims 4 to 6, The learned model is A first network into which the spectrogram is input, A second network into which the second spectrogram is input, A combination layer that outputs an inference result of the state based on a combination result of an output of the first network and an output of the second network, A processing system characterized by including this. **Claim 8** In the processing system according to claim 1, The spectrogram is image data, The learned model is a CNN or a ViT. A processing system characterized by this. **Claim 9** In the processing system according to claim 8, The image data is characterized in that the number of pixels in the frequency direction is larger than the number of pixels in the time axis direction. A processing system characterized by this. **Claim 10** In the processing system according to claim 1, The vibration information is acceleration, velocity, displacement, angular acceleration, angular velocity, or angle. A processing system characterized by this. **Claim 11** In the processing system according to claim 1, The inference unit infers a classification result regarding the state. A processing system characterized by this. **Claim 12** An acquisition unit that acquires vibration information of an object, An arithmetic unit that calculates at least a spectrogram related to the phase based on the vibration information, An inference unit that infers the state by inputting the spectrogram into a learned model trained to infer a state related to at least one of state monitoring, quality control, and predictive maintenance for the spectrogram, A program that causes a computer to function as this. **Claim 13** Acquire the vibration information of the object, Based on the vibration information, calculate at least a spectrogram related to the phase, Infer the state by inputting the spectrogram into a learned model trained to infer a state related to at least one of state monitoring, quality control, and predictive maintenance for the spectrogram A processing method characterized by this.

Images