Generation device, generation method, and generation program

Abstract

To improve the accuracy of explanations of reasons when an abnormality is detected in input sound. [Solution] The generation device has a memory unit that stores a set of training data sets that are combinations of audio signals that indicate a state and explanatory text that explains the state in text strings; a signal encoding unit that encodes the audio signal based on first learning parameters to generate an audio feature vector; a language encoding unit that encodes the explanatory text based on second learning parameters to generate a language feature vector; a language decoding unit that decodes the audio feature vector into text that indicates the state based on third learning parameters; and an update unit that updates the first learning parameters and second learning parameters through contrastive learning using the combination of the audio feature vector and the language feature vector, and updates the third learning parameters based on the difference between the text that indicates the state decoded by the language decoding unit and the explanatory text.

Description

[Technical Field]

[0001] The present invention relates to a generating device, a generating method, and a generating program for generating a character string. [Background technology]

[0002] It is important to generate a string of text in natural language that explains what has changed in the signals obtained under two different conditions as the conditions change. For example, abnormalities and signs of abnormalities in equipment or machinery can be automatically detected from operating sounds. However, simply displaying the presence or absence of abnormalities or signs does not tell the user what to focus on in subsequent detailed manual inspections, which requires a lot of work.

[0003] On the other hand, if it were possible to automatically and clearly present in natural language how the normal sounds measured in the past differ from the current sounds determined to be abnormal, this would provide clues for the inspector user to carry out a detailed inspection, further reducing labor costs.

[0004] Non-Patent Document 1 below discloses a method for generating text that explains in natural language what has changed between two optical images. Non-Patent Document 1 states, "We present a novel Dual Dynamic Attention Model (DUDA) to perform robust Change Captioning. Our model learns to distinguish distractors from semantic changes, localize the changes via Dual Attention over "before" and "after" images, and accurately describe them in natural language via Dynamic Speaker, by adaptively focusing on the necessary visual inputs (e.g., "before" or "after" image)." [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Dong Huk Park, Trevor Darrell, and Anna Rohrbach, “Robust Change Captioning,” in arxiv, 17 April 2019. Summary of the Invention [Problem to be solved by the invention]

[0006] In Non-Patent Document 1, optical images are the subject of the detection, and therefore local changes in limited pixel regions, such as object movement, are the primary detection target. Therefore, it is relatively clear from the image what changes should be noted. Meanwhile, training data is created by a human being called annotator manually assigning correct explanatory strings to two images, one before and one after the change. In the case of optical images, as described above, the changes that should be noted are relatively clear, so the annotator can assign appropriate explanatory strings. Therefore, appropriate training data can be created, and a generative model for string generation can be trained based on that training data, enabling highly accurate string generation.

[0007] However, when dealing with general signals, especially sounds and vibrations from equipment and machinery, their components are not limited to a specific time-frequency region; rather, changes in volume, pitch, and the appearance or disappearance of new sound sources occur across the entire signal. Because countless changes occur between the signal before and after a change, it is unclear which changes should be focused on. Therefore, unless an annotator knows what to focus on among the countless changes, they will be unable to provide a desirable description. Therefore, even if appropriate training data cannot be created and a generative model for string generation is trained based on that training data, accurate string generation will not be achieved.

[0008] The present invention aims to learn, from signals obtained under two different conditions, what has changed between the signals due to a change in conditions in a way that can be explained in natural language.The present invention also aims to learn, from signals obtained under two different conditions, what has changed between the signals due to a change in conditions in a way that can be explained in natural language. [Means for solving the problem]

[0009] A generation device according to one aspect of the invention disclosed in the present application comprises: a memory unit that stores a set of training datasets, each of which is a combination of a speech signal indicating a state and a description that describes the state in the form of a string of characters; a signal encoding unit that encodes the speech signal based on first learning parameters to generate a speech feature vector; a language encoding unit that encodes the description based on second learning parameters to generate a language feature vector; a language decoding unit that decodes the speech feature vector into text indicating the state based on third learning parameters; and an update unit that updates the first learning parameters and the second learning parameters through contrastive learning using a combination of the speech feature vector generated by the signal encoding unit and the language feature vector generated by the language encoding unit, and updates the third learning parameter based on a difference between the text indicating the state decoded by the language decoding unit and the description. [Effects of the Invention]

[0010] According to the exemplary embodiment of the present invention, it is possible to improve the accuracy of the explanation of the reason when an abnormality in the input sound is detected. Problems, configurations, and effects other than those described above will become clear from the description of the following examples. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating an example of the hardware configuration of a generating device. [Figure 2] FIG. 2 is a block diagram of an example of a functional configuration of the generating device according to the first embodiment. [Figure 3] FIG. 3 is a block diagram of an example of a functional configuration of the learning unit according to the first embodiment. [Figure 4] FIG. 4 is a flowchart of an example of a learning process procedure of the learning unit according to the first embodiment. [Figure 5] FIG. 5 is a block diagram of an example of a functional configuration of the generating unit according to the first embodiment. [Figure 6] FIG. 6 is a flowchart of an example of a generating process procedure of the generating unit according to the first embodiment. [Figure 7] FIG. 7 is a block diagram of an example of a functional configuration of a learning unit according to the second embodiment. [Figure 8] FIG. 8 is a flowchart of an example of a learning process procedure of the learning unit according to the second embodiment. [Figure 9] FIG. 9 is a block diagram of an example of a functional configuration of a generating unit according to the second embodiment. [Figure 10] FIG. 10 is a flowchart of an example of a generating process procedure of the generating unit according to the second embodiment. [Figure 11] FIG. 11 is a block diagram of an example of a functional configuration of a learning unit according to the third embodiment. [Figure 12] FIG. 12 is a flowchart of an example of a learning procedure of the learning unit according to the third embodiment. [Figure 13] FIG. 13 is a block diagram of an example of a functional configuration of a generating unit according to the third embodiment. [Figure 14] FIG. 14 is a flowchart of an example of a generation process procedure of the generation unit according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION [Example]

[0012] <Figure 1 Example of hardware configuration of generation device> FIG. 1 is a block diagram showing an example of the hardware configuration of a generating device. The generating device 100 includes a processor 101, a storage device 102, an input device 103, an output device 104, and a communication interface (communication IF) 105. The processor 101, the storage device 102, the input device 103, the output device 104, and the communication IF 105 are connected via a bus 106. The processor 101 controls the generating device 100. The storage device 102 serves as a working area for the processor 101. The storage device 102 is a non-transitory or temporary recording medium that stores various programs and data. Examples of the storage device 102 include a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), and a flash memory. The input device 103 inputs data. Examples of the input device 103 include a keyboard, a mouse, a touch panel, a numeric keypad, a scanner, a microphone, and a sensor. The output device 104 outputs data. The output device 104 includes, for example, a display, a printer, and a speaker. The communication IF 105 connects to a network and transmits and receives data.

[0013] <Figure 2: Example of functional configuration of generation device 100> 2 is a block diagram illustrating an example of a functional configuration of the generating device 100 according to the first embodiment. The generating device 100 includes a training dataset DB 201, a learning unit 202, a generative model 203, and a generating unit 204. Specifically, the training dataset DB 201 is stored in, for example, the storage device 102 illustrated in FIG. 1 or another computer that can communicate with the generating device 100. Specifically, the learning unit 202, the generative model 203, and the generating unit 204 are realized by, for example, causing the processor 101 to execute a program stored in the storage device 102 illustrated in FIG. 1.

[0014] The training dataset DB 201 is a database that stores one or more training datasets. A training dataset is a combination of training data and ground truth data. The training dataset DB 201 has, as a training dataset, a set of triples u {triplet 1, , triple u, , triple U}, each of which is made up of a prior signal time waveform set 2u1, a posterior signal time waveform set 2u2, and an explanatory sentence 2u3.

[0015] The prior signal time waveform set 2u1 is a set of prior signal time waveforms. A prior signal time waveform is training data that indicates the time waveform of a prior signal. A prior signal is a signal under a condition before a certain state change, such as a steady sound, periodic sound, or non-periodic sound from the equipment under test. In the case of sound, a time waveform is a waveform over time whose elements are the sound pressure value at each time.

[0016] The posterior signal time waveform set 2u2 is a set of posterior signal time waveforms. The posterior signal time waveforms are training data that indicate the time waveforms of posterior signals. The posterior signals are signals under conditions after a certain state change, such as abnormal sounds of the equipment under test that have changed from the steady sound before the change.

[0017] When there is no distinction between the pre-signal time waveform and the post-signal time waveform, they are referred to as signal time waveforms.

[0018] The explanatory text 2u3 is a variable-length text that includes onomatopoeia that expresses the change between the pre-signal time waveform and the post-signal time waveform.

[0019] An example of an explanatory sentence that describes a change in the bearings of a rotating body from normal to abnormal is as follows: The character strings enclosed in quotation marks are onomatopoeia added by the annotator.

[0020] "The sound changed from 'bo' to 'woo' and the pitch became higher." "The 'win win' sound has disappeared." "The buzzing and hissing sounds have become higher in pitch and volume."

[0021] Here, the annotator is instructed to focus on the changes, create an explanatory text that expresses the changes, and use this as ground truth data for the generative model 203, which enables the generative model 203 to explain how the steady sound differs from the sound currently determined to be abnormal.

[0022] The assignment of onomatopoeia by an annotator is extremely important in order to provide information that will serve as clues for detailed inspection by an inspector. This is because if an annotator were given only the post-signal time waveform and asked to answer questions such as "What sound is that?" or "What kind of sound is that?", they would only be able to obtain answers that are independent of the change, such as simply "the sound of a bearing."

[0023] Furthermore, there is a problem that explanations using only the original text that do not include onomatopoeia cannot express in detail what sounds have changed and how they have changed. In other words, without onomatopoeia expressions, not only cannot the generative model 203 express them in detail, but also the annotator cannot provide good explanations, making it impossible to create a training dataset to be used for learning the generative model 203.

[0024] Therefore, by creating explanatory text containing onomatopoeia in the annotation, the annotator can describe in detail what sounds have changed and how they have changed. Using the explanatory text created in this way, a generative model 203 that can express changes in detail can be realized.

[0025] It is also possible to have the annotator answer not just with onomatopoeia but also with a classification of the sound (for example, "the sound of a bearing"). However, such expressions tend to increase the vocabulary as the number of usage scenes increases, but this increased vocabulary is not used in different scenes, making it difficult to acquire a general-purpose model that spans multiple scenes. Therefore, by focusing on the fact that onomatopoeia can be used generically across multiple scenes and having the annotator create explanatory text that includes onomatopoeia in the annotation, it is possible to acquire a general-purpose generative model 203 that spans multiple scenes.

[0026] The learning unit 202 randomly selects a triplet u from a set of U triplets {triplet 1, , triplet u, , triplet U}. As described above, the triplet u is composed of a prior signal time waveform set 2u1, a posterior signal time waveform set 2u2, and an explanatory sentence 2u3. Furthermore, the learning unit 202 randomly selects one element from the prior signal time waveform set 2u1 of the triplet to set it as a prior signal time waveform 301, and randomly selects one element from the posterior signal time waveform set 2u2 to set it as a posterior signal time waveform 302, sets the combination of the prior signal time waveform 301 and the posterior signal time waveform 302 as an explanatory variable, and sets the explanatory sentence 2u3 as the objective variable, that is, the explanatory sentence 303.

[0027] The learning unit 202 uses the triplet u to learn the generative model 203. Specifically, for example, the learning unit 202 calculates the value of a loss function based on the difference between the output data output as a result of inputting the set explanatory variables into the generative model 203 and the explanatory text 303, and updates the learning parameters of the generative model 203 so that the value of the loss function is minimized.

[0028] The generation model 203 is a language model that outputs an explanatory sentence when a signal time waveform is input. The generation model 203 is trained by the training unit 202, and the generation unit 204 generates the summary basis explanatory sentence 243.

[0029] The generation unit 204 uses a generation model 203 to input a reference signal time waveform 241 and a target signal time waveform 242 to the generation model 203 and output a summary basis explanation 243. The reference signal time waveform 241 is a time waveform of a reference audio signal (hereinafter referred to as a reference signal) that serves as a reference for the test target device that is the target of anomaly detection. The reference signal time waveform 241 may be a prior signal time waveform in the prior signal time waveform set 211, or may be a prior signal time waveform different from the prior signal time waveform in the prior signal time waveform set 211.

[0030] The target signal time waveform 242 is a time waveform of an audio signal (hereinafter referred to as the target signal) emitted by the test target device that is the target of anomaly detection. The target signal time waveform 242 may be a posterior signal time waveform in the posterior signal time waveform set 212, or may be a posterior signal time waveform different from the posterior signal time waveform in the posterior signal time waveform set 212.

[0031] <Figure 3: Example of functional configuration of learning unit 202> 3 is a block diagram illustrating an example of a functional configuration of the learning unit 202 according to the first embodiment. The learning unit 202 includes frame division units 311 and 312, window function multiplication units 321 and 322, frequency domain signal generation units 313 and 323, signal encoding units 351 and 352, a feature difference calculation unit 353, a feature combination unit 354, a language decoding unit 355, an onomatopoeia-to-phoneme conversion unit 331, an onomatopoeia-to-subword generation unit 332, and an updating unit 356.

[0032] The training unit 202 also includes a language encoding unit 333 , a language linear projection unit 334 , a signal linear projection unit 344 , and a dimension adjustment unit 345 .

[0033] The frame division units 311 and 312 divide the signal time waveform into frames, and each of the divided signal time waveforms is called a frame division signal.

[0034] The window function multiplication units 321 and 322 perform window function multiplication on the frame division signals to convert each of the frame division signals into a window function multiplied signal.

[0035] The frequency domain signal generators 313 and 323 perform a short-time Fourier transform on each of the window function-multiplied signals to convert them into time-frequency domain signals. The frequency domain signal generators 313 and 323 can also use a frequency transform method such as a constant Q transform (CQT) instead of the short-time Fourier transform.

[0036] The signal encoding units 351 and 352 are neural networks that calculate feature vectors from frequency-domain signals based on learning parameters of the signal encoding units 351 and 352. The signal encoding units 351 and 352 are typically neural network encoders in which multiple convolutional layers, activation functions, and pooling layers are stacked, with skip connections sandwiched between them. The signal encoding units 351 and 352 may also be recurrent neural networks having layers such as a known Transformer model, a Long-Short-Term Memory (LSTM), a bidirectional LSTM, a Gated Recurrent Unit (GRU), or a bidirectional GRU.

[0037] The feature difference calculation unit 353 calculates a difference vector, which is the difference between the feature vector from the signal encoding unit 351 and the feature vector from the signal encoding unit 352. The difference vector is a feature that emphasizes changes, and by learning using this feature, it is possible to generate a generative model 203 that generates explanatory text that emphasizes changes.

[0038] The feature quantity combining unit 354 combines the feature quantity vector from the signal encoding unit 351, the feature quantity vector from the signal encoding unit 352, and the difference vector to generate a combined vector.

[0039] The signal linear projection unit 344 is a neural network that linearly projects the combined vector from the feature combination unit 354 to generate a signal feature vector with N dimensions and outputs it to the dimension adjustment unit 345. This neural network is, for example, a fully connected layer, or a combination of a fully connected layer and an appropriate activation nonlinear function such as ReLU. Note that the signal linear projection unit 344 may output the signal feature vector with N dimensions directly to the update unit 356, rather than to the dimension adjustment unit 345.

[0040] The dimension adjustment unit 345 is a neural network that adjusts the dimensions of an input vector to the dimensions of a text embedding vector with a dimension number of P. Specifically, for example, the dimension adjustment unit 345 converts a signal feature vector with a dimension number of N from the signal linear projection unit 344 into a text embedding vector with a dimension number of P. Also, the dimension adjustment unit 345 converts a language feature vector with a dimension number of M (≠N) from the language linear projection unit 334 into a text embedding vector with a dimension number of P. In this way, the dimension adjustment unit 345 converts multiple vectors with different dimensions into vectors with the same dimension number. This neural network is, for example, a fully connected layer, or a combination of a fully connected layer and an appropriate activation nonlinear function such as ReLU.

[0041] The onomatopoeia phoneme conversion unit 331 extracts character strings enclosed in parentheses from the explanatory sentence 303 as onomatopoeia, converts the extracted onomatopoeia into a phoneme string, and generates onomatopoeia phoneme-converted text. For example, if the onomatopoeia is "kankan," the phoneme string will be / ka N ka N / . Also, if the onomatopoeia is "katakata don," the phoneme string will be / katakatado: N / . Therefore, the example of the explanatory sentence 303 mentioned above, "The pitch of the sounds 'boon' and 'sha' has increased, and the volume has become louder," is converted to "The pitch of the sounds / bu: N / and / sh a: / has increased, and the volume has become louder."

[0042] The onomatopoeia subword generating unit 332 generates subwords from the onomatopoeia phoneme-converted text. Specifically, for example, the onomatopoeia subword generating unit 332 outputs partial character strings extracted for each predetermined number of characters n (number of grams) while shifting the target range by one character for the onomatopoeia phoneme string in the onomatopoeia phoneme-converted text.

[0043] For example, if the onomatopoeia is "katakatadon," the original phoneme ( / katakatado: N / ) is converted into the following phoneme subwords (assuming n=4):

[0044] / kata / / atak / / taka / / akat / / kata / / atad / / tado: / / ado: N /

[0045] Thereafter, each line of n=4 characters is treated as one word, thereby generating a subword explanation sentence 343 in which only onomatopoeia words are converted into phoneme subwords.

[0046] We will now discuss the effect of subwording onomatopoeia. Compared to regular words, onomatopoeia have a sparse frequency of occurrence. For example, "katakatadon" rarely appears in other scenes. Therefore, if it is input directly into a language model, similar onomatopoeia will be distinguished as completely separate words, resulting in insufficient training data per word and making it impossible to train. Subwording has the effect of preventing a lack of training data by breaking down "katakatadon" into high-frequency phoneme sequences such as "kata" and "taka."

[0047] The language encoding unit 333 calculates a language feature vector from the subworded explanation sentence 343 based on the learning parameters of the language encoding unit 333. The language encoding unit 333 is typically a neural network encoder in which multiple convolutional layers, activation functions, and pooling layers are stacked, with skip connections sandwiched between them. The language encoding unit 333 may also be a recurrent neural network having layers such as a known Transformer model, a Long-Short-Term-Memory (LSTM), a bidirectional LSTM, a Gated Recurrent Unit (GRU), or a bidirectional GRU.

[0048] The linguistic linear projection unit 334 is a neural network that linearly projects the linguistic feature vector from the linguistic encoding unit 333 to generate a linguistic feature vector with M dimensions and outputs it to the dimension adjustment unit 345. This neural network is, for example, a fully connected layer, or a combination of a fully connected layer and an appropriate activation nonlinear function such as ReLU. Note that the linguistic linear projection unit 334 may output the linguistic feature vector with M dimensions directly to the update unit 356, rather than to the dimension adjustment unit 345.

[0049] The language decoding unit 355 receives as input a text embedding vector with dimension P from the dimension adjustment unit 345 derived from the signal linear projection unit 344, and generates a variable-length text in which the onomatopoeic phonemes are subworded, similar to the subword-converted explanation 343 described below. The language decoding unit 355 is typically a decoder of the well-known Transformer model, which is a type of neural network. The language decoding unit 355 may also be a recurrent neural network having layers such as a long-short-term memory (LSTM), a bidirectional LSTM, a gated recurrent unit (GRU), or a bidirectional GRU.

[0050] The update unit 356 executes a learning process to update the learning parameters of the neural network.

[0051] Specifically, for example, the update unit 356 compares the variable-length text generated by the language decoding unit 355 (however, like the sub-word-converted explanation sentence 343, the onomatopoeia phonemes are sub-converted) with the sub-word-converted explanation sentence 343 generated by the onomatopoeia sub-word generation unit 332, and calculates the cross entropy L of the following formula (1).

[0052]

number

[0053] Here, K_u is the total number of elements belonging to the prior signal time waveform set 2u1, and k is a number that uniquely identifies that element. I_u is the total number of elements belonging to the posterior signal time waveform set 2u2, and i is a number that uniquely identifies that element. T is the number of words that appear in the subword-converted explanation 343. t is a number that uniquely identifies that word. w(t) is the probability that the t-th word is correctly estimated, and can be calculated by comparing the variable-length text generated by the language decoding unit 355 (however, like the subword-converted explanation 343, the onomatopoeic phonemes are subworded) with the subword-converted explanation 343. w_1:t-1 represents the sequence of words from t=1 to t=t-1. X is a connection vector. Optimization can be performed using known optimization algorithms such as SGD, Momentum SGD, AdaGrad, RMSProp, AdaDelta, and Adam.

[0054] In addition, the update unit 356 calculates contrastive learning, i.e., a contrastive loss of a contrast matrix composed of the cosine similarity between the text embedding vectors of dimension P from the dimension adjustment unit 345 derived from the signal linear projection unit 344 (which may be a signal feature vector of dimension N from the signal linear projection unit 344), and the text embedding vector of dimension P from the dimension adjustment unit 345 derived from the language linear projection unit 334 (which may be a language feature vector of dimension M from the language linear projection unit 334).

[0055] The update unit 356 uses the sum of the cross entropy L and the contrast loss as a loss function, and updates the learning parameters of each neural network so that the loss function becomes smaller.

[0056] Specifically, for example, the update unit 356 updates the learning parameters of the neural networks of the language decoding unit 355 and the dimension adjustment unit 345 so that the cross entropy L becomes smaller.

[0057] In addition, the update unit 356 updates the learning parameters of each neural network in the signal coding units 351, 352, the language coding unit 333, the signal linear projection unit 344, and the language linear projection unit 334 so that the contrast loss becomes small, i.e., so that the diagonal terms of the contrast matrix become large and the off-diagonal terms become small.

[0058] Specifically, for example, in a PxP comparison matrix consisting of both text embedding vectors, each element represents the similarity between both text embedding vectors. The similarity is expressed as the Euclidean distance or cosine similarity between both text embedding vectors. The smaller the Euclidean distance and the closer the cosine similarity is to 1, the higher the similarity between both text embedding vectors.

[0059] The update unit 356 updates the learning parameters of each neural network in the signal coding units 351, 352, the language coding unit 333, the signal linear projection unit 344, and the language linear projection unit 334 so that the similarity between the two text embedding vectors indicated by the diagonal components of the contrast matrix increases and the similarity between the two text embedding vectors indicated by the off-diagonal components decreases.

[0060] The combination of the neural network (encoding model) of the signal encoding units 351, 352, the language encoding unit 333, the signal linear projection unit 344, and the language linear projection unit 334, whose learning parameters have been updated, and the neural network (decoding model) of the language decoding unit 355 and the dimension adjustment unit 345, whose learning parameters have been updated, becomes the generative model 203.

[0061] The pre-trained signal linear projection unit 344 contains general-purpose acoustic knowledge, so the use of the signal linear projection unit 344 can significantly reduce the amount of data that needs to be additionally trained, making additional training data unnecessary.

[0062] Similarly, the pre-trained linguistic linear projection unit 334 also contains general-purpose knowledge about natural languages. Therefore, by using the linguistic linear projection unit 334, the amount of data to be additionally trained can be significantly reduced, and additional training data becomes unnecessary.

[0063] <Figure 4 Learning process procedure of learning unit 202> FIG. 4 is a flowchart of an example of a learning process procedure of the learning unit 202 according to the first embodiment.

[0064] (Step S401) The update unit 356 determines whether the value of the loss function converges. Specifically, for example, the update unit 356 determines whether a convergence condition is satisfied or whether the number of iterations C1 is greater than a threshold value ThC. The convergence condition is, for example, a condition that the convergence determination function becomes smaller than a predetermined threshold value.

[0065] If the convergence condition is not satisfied or the number of iterations C1 is not greater than the threshold value ThC (step S401: No), proceed to step S402. If the convergence condition is satisfied or the number of iterations C1 is greater than the threshold value ThC (step S401: Yes), it is determined that the value of the loss function has converged, and proceed to step S424.

[0066] (Step S402) The learning unit 202 randomly selects a triplet u from the training dataset DB 201. As described above, the triplet u consists of a prior signal time waveform set 2u1, a posterior signal time waveform set 2u2, and an explanatory sentence 2u3. Furthermore, the learning unit 202 randomly selects one element from 2u1 of the triplet to set it as a prior signal time waveform 301, and randomly selects one element from 2u2 to set it as a posterior signal time waveform 302, sets the combination of the prior signal time waveform 301 and the posterior signal time waveform 302 as an explanatory variable, and sets the explanatory sentence 2u3 as the objective variable, that is, the explanatory sentence 303.

[0067] (Step S403) The onomatopoeia-phoneme conversion unit 331 extracts onomatopoeia from the explanatory text 303, converts them into a phoneme string, and generates onomatopoeia-phoneme converted text.

[0068] (Step S404) The onomatopoeia subword generating unit 332 generates subwords from the onomatopoeia phoneme converted text converted by the onomatopoeia phoneme converting unit 331 , and generates a subword-converted explanation sentence 343 .

[0069] (Step S405) The frame dividing unit 311 divides the preliminary signal time waveform into frames. The frame divided signal from the frame dividing unit 311 is called a preliminary frame divided signal.

[0070] (Step S406) The window function multiplication unit 321 performs window function multiplication on the pre-frame divided signals to convert each of the pre-frame divided signals into a window function multiplied signal. This window function multiplied signal is referred to as a pre-window function multiplied signal.

[0071] (Step S407) The frequency domain signal generator 313 performs a short-time Fourier transform on each of the pre-window function multiplied signals to convert them into time-frequency domain signals, which are referred to as pre-time-frequency domain signals.

[0072] (Step S408) The signal encoding unit 351 calculates a feature vector from the a priori frequency domain signal based on the learning parameters of the signal encoding unit 351. This feature vector is called an a priori feature vector.

[0073] (Step S409) The frame dividing unit 312 divides the post-signal time waveform into frames. The frame divided signal from the frame dividing unit 312 is referred to as a post-frame divided signal.

[0074] (Step S410) The window function multiplication unit 322 performs window function multiplication on the post-frame division signals to convert each of the post-frame division signals into a window function multiplied signal, which is referred to as a post-window function multiplied signal.

[0075] (Step S411) The frequency domain signal generator 323 performs a short-time Fourier transform on each of the post-window function multiplied signals to convert them into time-frequency domain signals, which are referred to as post-time-frequency domain signals.

[0076] (Step S412) The signal encoding unit 352 calculates a feature vector from the posterior frequency domain signal based on the learning parameters of the signal encoding unit 352. This feature vector is called a posterior feature vector.

[0077] Note that steps S409 to S412 may be executed in parallel with steps S405 to S408.

[0078] (Step S413) The feature amount difference calculation unit 353 calculates a difference vector, which is the difference between the pre-feature amount vector and the posterior feature amount vector.

[0079] (Step S414) The feature combining unit 354 combines the a priori feature vector, the posterior feature vector, and the difference vector to generate a combined vector.

[0080] (Step S415) The signal linear projection unit 344 linearly projects the combined vector generated in step S414 to generate a signal feature vector with N dimensions.

[0081] (Step S416) The dimension adjustment unit 345 converts the signal feature vector with N dimensions generated in step S415 into a text embedding vector with P dimensions.

[0082] (Step S417) The language decoding unit 355 receives the text embedding vector of dimension P generated in step S416 from the signal linear projection unit 344 as input, and generates a variable-length text in which the onomatopoeic phonemes are sub-worded, similar to the sub-worded explanation sentence 343.

[0083] (Step S418) The language encoding unit 333 calculates a language feature vector from the subword-decomposed explanatory sentence 343 generated in step S404 based on the learning parameters of the language encoding unit 333.

[0084] (Step S419) The linguistic linear projection unit 334 linearly projects the linguistic feature vector generated in step S418 to generate a linguistic feature vector with M dimensions.

[0085] (Step S420) The dimension adjustment unit 345 converts the language feature vector with the M-dimensionality generated in step S419 into a text embedding vector with the P-dimensionality.

[0086] (Step S421) The update unit 356 performs contrastive learning based on the text embedding vector of dimension P from the signal linear projection unit 344 generated in step S415 and the text embedding vector of dimension P from the language linear projection unit 334 generated in step S419, and updates the learning parameters of each neural network in the signal encoding units 351 and 352, the language encoding unit 333, the signal linear projection unit 344, and the language linear projection unit 334 so that the symmetric loss matrices of both text embedding vectors become smaller.

[0087] In addition, the update unit 356 compares the variable-length text generated in step S417 (however, like the sub-worded explanation sentence 343, the onomatopoeic phonemes have been sub-worded) with the sub-worded explanation sentence 343 generated in step S404, and updates the learning parameters of each neural network model of the language decoding unit 355 and the dimension adjustment unit 345 so as to minimize the cross entropy L in the above equation (1).

[0088] (Step S422) The update unit 356 calculates the convergence conditions.

[0089] (Step S423) The update unit 356 increments the number of iterations C1, and then the process returns to step S401.

[0090] (Step S424) If step S401: Yes, the update unit 356 saves the learning parameters updated in step S420 in the storage device 102 as learning parameters of the generative model 203. This completes the learning process of the learning unit 202.

[0091] <Fig. 5: Example of functional configuration of the generation unit 204> 5 is a block diagram illustrating an example of a functional configuration of the generation unit 204 according to the first embodiment. The generation unit 204 includes frame division units 311 and 312, window function multiplication units 321 and 322, frequency domain signal generation units 313 and 323, signal encoding units 351 and 352, a feature difference calculation unit 353, a feature combination unit 354, a signal linear projection unit 344, a dimension adjustment unit 345, and a language decoding unit 355. That is, a part of the configuration of the learning unit 202 also functions as the generation unit 204. The generation unit 204 also includes an anomaly detection unit 501 and a summarization unit 502.

[0092] The anomaly detection unit 501 detects statistical outliers based on the feature vectors from the signal encoding unit 351 and the feature vectors from the signal encoding unit 352, and outputs an anomaly detection result 510. Specifically, for example, the anomaly detection unit 501 calculates the average value (average distance) of K distances between the K nearest neighbor feature vectors from the signal encoding unit 351 and the feature vector from the signal encoding unit 352 using the K nearest neighbor method. The anomaly detection unit 501 determines an anomaly if the average distance is equal to or greater than a threshold, and determines a normality if not.

[0093] The language decoding unit 355 receives as input a text embedding vector of dimension P for the reference signal and generates K nearest neighbor basis explanation sentences 520, which are variable-length texts in which the phonemes of onomatopoeia are sub-worded, similar to the sub-word explanation sentences 343 described below.

[0094] The summarizing unit 502 generates a prompt requesting the generation of a summary basis explanation sentence 243 for the K basis explanation sentences 520 generated by the language decoding unit 355, and outputs the prompt to a generating AI (Artificial Intelligence) not shown. The generating AI may be implemented inside the generating device 100, or may be implemented in an external computer capable of communicating with the generating device 100. The summarizing unit 502 acquires the summary basis explanation sentence 243 from the generating AI. The acquired summary basis explanation sentence 243 is displayed, for example, on a display.

[0095] Generative AI uses a dataset to train a language model through natural language processing, and generates sentences using the language model. A language model is a type of probabilistic model used in natural language processing, and is a model for probabilistically predicting how likely a given word or sentence is to occur in natural language. Specifically, in the field of natural language processing, it is a mathematical model that learns language patterns and grammatical rules to generate and understand natural language.

[0096] For example, generative AI uses a language model to calculate the probability of occurrence of a given word sequence or sentence, or to compare the probabilities of occurrence of multiple word sequences or sentences, and when predicting the next word or sentence, automatically generates the most likely word or sentence based on the context. In this way, when generative AI receives a query called a prompt, it outputs an answer to that query using a language model that has been trained on a huge amount of data set.

[0097] <Fig. 6 Generation processing procedure of the generation unit 204> FIG. 6 is a flowchart of an example of a generating process procedure of the generating unit 204 according to the first embodiment.

[0098] (Step S601) The generator 204 reads the generative model 203 .

[0099] (Step S602) The generator 204 reads out a reference signal from the reference data set DB 240 and inputs it as a reference signal time waveform 241 .

[0100] (Steps S603 to S605) Generator 204 performs the same processes as steps S405 to S407 on input reference signal time waveform 241. This generates a frequency domain signal of the reference signal (reference time-frequency domain signal). If there are multiple reference signals, a reference time-frequency domain signal is generated for each reference signal.

[0101] (Step S606) The signal encoding unit 351 calculates a reference feature vector based on the reference time-frequency domain signal from step S605 using the generative model 203. If there are multiple reference time-frequency domain signals, a reference feature vector is generated for each reference time-frequency domain signal.

[0102] (Steps S607 to S609) The generating unit 204 performs the same processes as steps S409 to S411 on the input target signal time waveform 242. As a result, a frequency domain signal of the target signal (target time-frequency domain signal) is generated.

[0103] (Step S610) The signal encoding unit 352 uses the generative model 203 to calculate a target feature vector based on the target time-frequency domain signal from step S609.

[0104] (Step S611) The anomaly detection unit 501 detects statistical outliers based on the feature vector generated in step S606 from the signal encoding unit 351 and the feature vector generated in step S610 from the signal encoding unit 352, and outputs an anomaly detection result 510.

[0105] (Step S612) The feature difference calculation unit 353 calculates a difference vector, which is the difference between the reference feature vector generated in step S606 and the target feature vector generated in step S610. The difference vector is a feature that emphasizes changes, and by performing inference using this feature, it is possible to generate an explanation that emphasizes changes. If there are K nearest reference feature vectors, a difference vector is generated for each reference feature vector.

[0106] (Step S613) The feature combining unit 354 combines the reference feature vector, the target feature vector, and the difference vector to generate a combined vector. If there are K nearest reference feature vectors, a combined vector is generated for each reference feature vector.

[0107] (Step S614) The signal linear projection unit 344 linearly projects the combined vector generated in step S613 to generate a signal feature vector with dimension number N. If there are K nearest combined vectors, a signal feature vector with dimension number N is generated for each combined vector.

[0108] (Step S615) The dimension adjustment unit 345 converts the signal feature vector with dimension N generated in step S614 into a text embedding vector with dimension P. If there are K nearest neighbor signal feature vectors with dimension N, a text embedding vector with dimension P is generated for each signal feature vector with dimension N.

[0109] (Step S616) The language decoding unit 355 uses the generative model 203 and the text embedding vectors with dimension P generated in step S615 as input to generate variable-length text including onomatopoeia subword sequences. If there are K nearest neighbor text embedding vectors with dimension P, variable-length text is generated for each text embedding vector with dimension P.

[0110] (Step S617) The language decoding unit 355 converts the onomatopoeia subword string in the variable-length text generated in step S616 back into onomatopoeia text via the onomatopoeia phoneme string. If there are K nearest variable-length texts, onomatopoeia text is generated for each variable-length text.

[0111] Here, a method for inversely converting an onomatopoeia sub-word string of a variable-length text into an onomatopoeia phoneme string will be described. In the onomatopoeia phoneme conversion during learning (step S403), when converting an onomatopoeia phoneme string into an onomatopoeia sub-word string, the onomatopoeia sub-word generating unit 332 creates an onomatopoeia sub-word string by shifting each character while overlapping the range by n-1 characters.

[0112] Therefore, in the inverse conversion, the language decoding unit 355 extracts phonemes as follows for an onomatopoeia subword sequence S=[s_1, ..., s_M] consisting of M onomatopoeia subwords s_m (m=1, ..., M): for m=1, ..., M-1, only the first character v_m1 is extracted for the m-th subword s_m=[v_m1, ..., v_mn].

[0113] The language decoding unit 355 extracts the entire character string s_M=[v_M1, ..., v_Mn] for s_M of the last sub-word m=M. That is, it generates [v_11, v_21, v_31, ..., v_{M-1}1, v_M1, ..., v_Mn] as the onomatopoeia phoneme string.

[0114] Next, in the reverse conversion of the onomatopoeia phoneme string to onomatopoeia text, the language decoding unit 355 can convert according to the correspondence table since there is a one-to-one relationship between phonemes and katakana characters, thereby restoring the onomatopoeia text.

[0115] (Step S618) The summarizing unit 502 generates a prompt requesting the generation of summary basis explanations 243 for the nearest K basis explanations 520 generated in step S617, outputs the generated prompt to the generation AI, and acquires the summary basis explanations 243 from the generation AI. The acquired summary basis explanations 243 are displayed, for example, on a display. This completes the generation process of the generation unit 204.

[0116] In this way, by using the same feature space for acoustics and language, it becomes possible to provide appropriate linguistic explanations from the same perspective as the basis for anomaly detection.

[0117] In the above-described configuration, the generation device 100 may be configured not to use the signal linear projection unit 344, the language linear projection unit 334, and the dimension adjustment unit 345. Even when the signal linear projection unit 344, the language linear projection unit 334, and the dimension adjustment unit 345 are not used, contrastive learning must still be performed. Therefore, the number of dimensions of the combined vector output from the feature combination unit 354 and the number of dimensions of the language feature vector output from the language encoding unit 333 are set to be the same. [Example]

[0118] A description will be given of Example 2. Example 2 has a configuration in which the feature combining unit 354 is removed from the configuration of Example 1. In Example 2, differences from Example 1 will be mainly described, and therefore descriptions of common parts with Example 1 will be omitted.

[0119] <Figure 7: Example of functional configuration of the learning unit 202> 7 is a block diagram illustrating an example of a functional configuration of the learning unit 202 according to the second embodiment. Since the feature combining unit 354 is not provided, the signal linear projection unit 344 is a neural network that linearly projects the difference vector from the feature difference calculation unit 353 to generate a signal feature vector with a dimension number N. This neural network is, for example, a fully connected layer, or a combination of a fully connected layer and an appropriate activation nonlinear function such as ReLU.

[0120] <Fig. 8 Learning process procedure of the learning unit 202> 8 is a flowchart illustrating an example of a learning process procedure of the learning unit 202 according to the second embodiment. In the second embodiment, step S414 is not executed.

[0121] (Step S415) The signal linear projection unit 344 linearly projects the difference vector generated in step S413 to generate a signal feature vector with N dimensions.

[0122] <Fig. 9 Example of functional configuration of the generation unit 204> 9 is a block diagram illustrating an example of the functional configuration of the generation unit 204 according to the second embodiment. Since the feature combining unit 354 is not provided, the signal linear projection unit 344 is a neural network that linearly projects the difference vector from the feature difference calculation unit 353 to generate a signal feature vector with a dimension number N. This neural network is, for example, a fully connected layer, or a combination of a fully connected layer and an appropriate activation nonlinear function such as ReLU.

[0123] <Fig. 10 Generation process procedure of the generation unit 204> 10 is a flowchart illustrating an example of a generating process procedure of the generating unit 204 according to the second embodiment. In the second embodiment, step S616 is not executed.

[0124] (Step S614) The signal linear projection unit 344 linearly projects the difference vector generated in step S612 to generate a signal feature vector with dimension N. If there are K nearest neighbor combined vectors, a signal feature vector with dimension N is generated for each difference vector.

[0125] According to the second embodiment, the feature combining unit 354 is not used, and therefore the signal linear projection unit 344 performs linear projection using a difference vector having fewer dimensions than the combined vector, thereby increasing the processing speed of the signal linear projection unit 344. [Example]

[0126] A description will be given of Example 3. Example 3 has a configuration in which the feature amount difference calculation unit 353 and the feature amount combination unit 354 are excluded from the configuration of Example 1. In Example 3, differences from Examples 1 and 2 will be mainly described, and therefore descriptions of common parts with Examples 1 and 2 will be omitted.

[0127] <Fig. 11 Example of functional configuration of the learning unit 202> 11 is a block diagram illustrating an example of a functional configuration of the learning unit 202 according to the third embodiment. Since the difference calculation and combination of feature vectors are not performed, the learning unit 202 acquires the posterior signal time waveform 302 and performs frame division, window function multiplication, frequency-dependent signal generation, and signal encoding.

[0128] In addition, the explanatory text 2u3 is a variable-length text including onomatopoeia that expresses the change between the pre-signal time waveform and the post-signal time waveform, but in the case of Example 3, it is a variable-length text including onomatopoeia that expresses the acquired post-signal time waveform.

[0129] The signal linear projection unit 344 is a neural network that linearly projects the feature vector from the posterior signal encoding unit 352 to generate a signal feature vector with a dimension number of N. This neural network is, for example, a fully connected layer, or a combination of a fully connected layer and an appropriate activation nonlinear function such as ReLU.

[0130] <Fig. 12 Learning process procedure of the learning unit 202> 12 is a flowchart illustrating an example of a learning process procedure of the learning unit 202 according to the third embodiment. In the third embodiment, steps S413 and S414 are not executed.

[0131] (Step S415) The signal linear projection unit 344 linearly projects the feature vector generated in step S418 or step S412 to generate a signal feature vector with N dimensions.

[0132] <Fig. 13 Example of functional configuration of the generation unit 204> 13 is a block diagram illustrating an example of a functional configuration of the generation unit 204 according to the third embodiment. Because the feature difference calculation unit 353 and the feature combination unit 354 are not present, the signal linear projection unit 344 linearly projects the feature vector for the reference signal from the signal encoding unit 352 to generate K nearest signal feature vectors for the reference signal with the number of dimensions N. Furthermore, the signal linear projection unit 344 linearly projects the feature vector for the target signal from the signal encoding unit 352 to generate a signal feature vector for the target signal with the number of dimensions M.

[0133] The dimension adjustment unit 345 converts the signal feature vector for the reference signal with dimension N from the signal linear projection unit 344 into a text embedding vector with dimension P. Also, the dimension adjustment unit 345 converts the signal feature vector for the target signal with dimension M from the signal linear projection unit 344 into a text embedding vector with dimension P.

[0134] The language decoding unit 355 receives as input a text embedding vector of dimension P for the reference signal and generates explanatory sentences 1320 that explain the characteristics of each of the K reference signals, which are variable-length texts in which the phonemes of onomatopoeia have been sub-worded, similar to the sub-word explanatory sentences 343 described below.

[0135] The language decoding unit 355 receives as input a text embedding vector of dimension P for the target signal and generates an explanation 1330 that explains the characteristics of the target signal, which is a variable-length text in which the phonemes of onomatopoeia are subworded, similar to the subword-converted explanation 343 described below.

[0136] The summarizing unit 502 generates a prompt requesting the generation of a summary basis explanation 243 for the explanation 1320 and explanation 1330 generated by the language decoding unit 355, and outputs the prompt to a generation AI (not shown). The summarizing unit 502 generates a prompt such as the following, for example:

[0137] The following are descriptions of the normal signal characteristics: 1st: (Description of the first reference signal is inserted) Second: (Description of the second reference signal is inserted here) ... Kth: (Description of the Kth reference signal is inserted) In response, the signal characteristics changed as shown in the description below, and an anomaly was detected. ... (Description of the target signal 1330 is inserted) Please explain how the signal characteristics changed compared to normal times, which led to the detection of an abnormality.

[0138] The summarization unit 502 acquires the summary basis explanation 243 from the generation AI. The acquired summary basis explanation 243 is displayed on a display, for example.

[0139] <Fig. 14 Generation processing procedure of the generation unit 204> 14 is a flowchart illustrating an example of a generating process procedure of the generating unit 204 according to the third embodiment. In the third embodiment, step S616 is not executed.

[0140] (Step S1414) After step S611, the signal linear projection unit 344 linearly projects the feature vectors generated in step S606 to generate signal feature vectors with a dimension number of N. For the K nearest neighbor feature vectors generated in step S606, signal feature vectors with a dimension number of N for the reference signal are generated.

[0141] Furthermore, the signal linear projection unit 344 linearly projects the feature vector generated in step S610 to generate a signal feature vector with M dimensions for the target signal.

[0142] (Step S1415) The dimension adjustment unit 345 converts the signal feature vector of dimension N for the reference signal generated in step S614 into a text embedding vector of dimension P. If there are K nearest signal feature vectors of dimension N, a text embedding vector of dimension P is generated for each signal feature vector of dimension N.

[0143] Furthermore, the dimension adjustment unit 345 converts the signal feature vector of dimension M for the target signal generated in step S614 into a text embedding vector of dimension P.

[0144] (Step S1416) The language decoding unit 355 uses the generative model 203 and the P-dimensional text embedding vector generated in step S615 as input to generate variable-length text including onomatopoeia subword sequences. If there are K nearest P-dimensional text embedding vectors, variable-length text is generated for each P-dimensional text embedding vector. Variable-length text is also generated for the P-dimensional text embedding vector for the target signal.

[0145] (Step S1417) The language decoding unit 355 inversely converts the onomatopoeia subword string in the variable-length text generated in step S616 into onomatopoeia text via the onomatopoeia phoneme string. If there are K nearest variable-length texts for the reference signal, the onomatopoeia text is inversely converted into onomatopoeia text for each variable-length text for the reference signal. This onomatopoeia text is the K number of explanations 1320. The variable-length text for the target signal is also inversely converted into onomatopoeia text. This onomatopoeia text is the explanation 1330 for the target signal.

[0146] (Step S1418) The summarization unit 502 generates a prompt requesting the generation AI to generate a summary basis explanation 243 for the K nearest explanatory sentences 1320 and 1330 generated in step S617, outputs the generated prompt to the generation AI, and acquires the summary basis explanation 243 from the generation AI. The acquired summary basis explanation 243 is displayed, for example, on a display. This completes the generation process of the generation unit 204.

[0147] According to the third embodiment, learning can be performed using only texts expressing the features of individual signals, since the feature difference calculation unit 353 is not used. Therefore, when texts expressing the features of individual signals are easier to collect than texts expressing the differences between signals, this can be implemented at low cost.

[0148] Furthermore, since the feature quantity combining unit 354 is not used, the signal linear projection unit 344 performs linear projection using a difference vector having fewer dimensions than the combined vector, thereby increasing the processing speed of the signal linear projection unit 344.

[0149] <Switching models depending on the type of signal you are focusing on> Signals observed as prior signals and posterior signals are roughly classified into three types: stationary signals, periodic signals, and non-periodic signals. Depending on the type of signal, different models are suitable as models (hereinafter referred to as coding models) for the signal coding units 351 and 352. For example, for stationary signals, a network equipped with a spatial attention mechanism is suitable, and has better accuracy than a Transformer.

[0150] Transformer is suitable for periodic and non-periodic signals, and provides better accuracy than networks equipped with a spatial attention mechanism. Furthermore, providing separate coding models for stationary, periodic, and non-periodic signals provides better accuracy. Therefore, the generating device 100 constructs these three types of coding models as the generative model 203 as follows. As a prerequisite, a training dataset DB 201 is prepared for each type of signal.

[0151] The training dataset DB201 for stationary signals is composed of a set of prior signal time waveforms 211 of stationary signals, a set of posterior signal time waveforms 212 of stationary signals, and a set of explanatory text 213 in which an annotator explains the changes between them. Furthermore, by instructing the annotator to provide explanations "focusing on stationary signals," a training dataset DB201 specialized for stationary signals can be constructed. Then, by preparing a network equipped with a spatial attention mechanism suitable for stationary signals as an encoding model, as described above, the learning unit 202 executes learning of the generative model 203.

[0152] The training dataset DB201 for periodic signals is composed of a set of prior signal time waveforms 211 of periodic signals, a set of posterior signal time waveforms 212 of periodic signals, and a set of explanations 213 in which an annotator explains the changes between them. Furthermore, by instructing the annotator to provide explanations "focusing on periodic signals," a training dataset DB201 specialized for periodic signals can be constructed. Then, by providing a Transformer suitable for periodic signals as an encoding model, as described above, the learning unit 202 executes learning of the generative model 203.

[0153] The training dataset DB201 for aperiodic signals is composed of a set of prior signal time waveforms of aperiodic signals, a set of posterior signal time waveforms of aperiodic signals, and a set of explanations 213 in which an annotator explains the changes between them. Furthermore, by instructing the annotator to provide explanations "focusing on aperiodic signals," a training dataset DB201 specialized for aperiodic signals can be constructed. Then, by providing a Transformer, which is suitable for aperiodic signals as described above, as an encoding model, the learning unit 202 executes learning of the generative model 203.

[0154] The generator 204 switches between the above three types of generative models 203. For example, in a usage scenario where it is known that attention should be paid to a specific type of signal, by specifying and executing that type of generative model 203, it is possible to generate a summary basis explanation 243 that is specialized for the specified type of signal with high accuracy and without being adversely affected by other noises.

[0155] The generation unit 204 may simultaneously use the three types of generative models 203 in parallel. For example, the generation unit 204 may add the string "For stationary signals," to the beginning of the summary basis explanation 243 output from the generative model 203 for stationary signals, add the string "For periodic signals," to the beginning of the summary basis explanation 243 output from the generative model 203 for periodic signals, and add the string "For non-periodic signals," to the beginning of the summary basis explanation 243 output from the generative model 203 for non-periodic signals, and then output these three explanations in a distinct manner. This allows the user to simultaneously read explanations from multiple perspectives corresponding to the three types of generative models 203, which facilitates comparison between the perspectives and facilitates insight.

[0156] Although the example described here is one in which the generation unit 204 executes three types of generative models 203 in parallel, the generation unit 204 may execute two types of generative models 203 in parallel out of the three types of generative models 203. Furthermore, if there are signals other than the three types described above, the generation unit 204 may execute four or more types of generative models 203 in parallel.

[0157] As explained above, let A and B be the two conditions, and for sets S_A and S_B consisting of one or more sample signals corresponding to condition A (e.g., normal (before)) and condition B (e.g., abnormal (after)), respectively, an annotator is asked to assign a string C that explains the difference between sets S_A and S_B, and the triplet set of set S_A, set S_B, and string C is used as the training dataset.

[0158] During training, the generating device 100 inputs each element of the set S_A and the set S_B and trains the generative model 203 to output a character string C. During inference, the generating device 100 uses the generative model 203 to generate an inference explanation sentence 234 from the signals of the condition A and the condition B.

[0159] Annotators who do not know what to focus on among the countless changes tend to explain only outliers rather than significant changes when comparing signal samples one-to-one.In contrast, in this embodiment, even if annotators do not have specialized knowledge, by comparing samples between multiple different pairs under different conditions, significant changes between conditions A and B can be found and assigned character string C regardless of outliers.

[0160] Therefore, for example, the learning unit 202 repeatedly selects combinations of prior signal time waveforms and posterior signal time waveforms from the triplet u selected as the training data set thus created, and uses the joint vector and explanation 2u3 repeatedly generated for each selected combination to learn the generative model 203. This enables the generation unit 204 to generate an inference explanation 234 that focuses on significant changes due to changes in condition A and condition B when making an inference using the generative model 203.

[0161] In this way, the above-described generating device 100 can learn and infer, as if explaining in natural language, what has changed between signals due to changes in conditions, from signals obtained under two different conditions, even if the annotator does not know in advance what to focus on among the countless changes. This allows the annotator to easily identify what to focus on among the countless changes.

[0162] In the above-described embodiment, the generating device 100 has a learning unit 202 and a generating unit 204, but the generating device 100 may have either the learning unit 202 or the generating unit 204, and the other may be provided by another computer that can communicate with the generating device 100.

[0163] Furthermore, while the above-described embodiment has been described using an example of a sound signal, the same means can be used to implement the same processing for an ultrasonic sensor signal. Furthermore, the same configuration can be used to implement general time-series signals, such as the time waveform of an acceleration sensor or a displacement sensor, the time waveform of a current sensor, and financial indicators such as stock prices and exchange rates. In the case of the time waveform of a current sensor, financial indicators such as stock prices and exchange rates, and the like, they are not "onomatopoeia" but "mimetic words," and such onomatopoeia and mimetic words can be used as onomatopoeia in the onomatopoeia-phoneme conversion unit 331 and the onomatopoeia subword conversion unit 332.

[0164] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added to, deleted from, or replaced with other configurations.

[0165] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0166] Information such as programs, tables, and files that realize each function can be stored in storage devices such as memory, hard disks, and SSDs (Solid State Drives), or on recording media such as IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs).

[0167] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines that are necessary for implementation. In reality, it can be considered that almost all components are interconnected. [Explanation of symbols]

[0168] 100 generator 101 processors 102 Storage Devices 201 Training Dataset DB 202 Learning Department 203 Generative Model 204 Generation part 211 Pre-signal time waveform set 212 Post-signal time waveform set 213 Description Set 234 Inference Explanation 241 Pre-signal time waveform 242 Post-signal time waveform 243 Inference Explanation 303 Description 311,312 Frame division section 312,322 Window function multiplication section 313,323 Frequency domain signal generator 331 Onomatopoeia Phoneme Conversion Unit 332 Onomatopoeia Subword Generation Unit 333 Language encoding section 334 Linguistic linear projection part 343 Subword Description 344 Signal Linear Projection Section 345 Dimensional adjustment section 351,352 Signal encoder 353 Feature Difference Calculation Unit 354 Feature Combination Unit 355 Language Decoding Unit 356 Update Department

Claims

1. a storage unit that stores a set of training data sets that are combinations of voice signals that indicate states and explanatory texts that explain the states using character strings; a signal encoding unit that encodes the speech signal based on a first learning parameter to generate a speech feature vector; a language encoding unit that encodes the explanatory text based on second learning parameters to generate a language feature vector; a language decoding unit that decodes the speech feature vector into text indicating the state based on a third learning parameter; an updating unit that updates the first learning parameters and the second learning parameters through contrastive learning using a combination of the speech feature vector generated by the signal encoding unit and the language feature vector generated by the language encoding unit, and updates the third learning parameters based on a difference between the text indicating the state decoded by the language decoding unit and the explanatory text; A generating device comprising:

2. The generating device of claim 1 , the storage unit stores, as the audio signal, a pre-event signal indicating a state before a change in the state and a post-event signal indicating a state after a change in the state, and the explanatory sentence is a sentence that explains the state before and after the change in a character string; the signal encoding unit includes a first signal encoding unit and a second signal encoding unit; the first signal encoding unit encodes the prior signal based on a fourth learning parameter to generate a prior speech feature vector; the second signal encoding unit encodes the posterior signal based on a fifth learning parameter to generate a posterior speech feature vector; the language decoding unit decodes a first difference vector between the prior speech feature vector and the posterior speech feature vector into text indicating the state based on the third learning parameter; the update unit updates the fourth learning parameter, the fifth learning parameter, and the second learning parameter through contrastive learning using a combination of the first difference vector and the language feature vector, and updates the third learning parameter based on a difference between the text indicating the state and the explanatory text. A generating device characterized by:

3. 3. The generating device of claim 2, the language decoding unit decodes a first combined vector obtained by combining the prior speech feature vector, the posterior speech feature vector, and the first difference vector based on the third learning parameter into text indicating the state; the update unit updates the fourth learning parameter, the fifth learning parameter, and the second learning parameter through contrastive learning using a combination of the first combined vector and the linguistic feature vector, and updates the third learning parameter based on a difference between the text indicating the state and the explanatory text. A generating device characterized by:

4. The generating device of claim 1 , an anomaly detection unit that detects an anomaly in an anomaly detection target; the signal encoding unit encodes a reference speech signal that serves as a reference when the state of the anomaly detection target is normal, based on the first learning parameter, to generate a reference speech feature vector; and encodes a target signal emitted by the anomaly detection target, based on the first learning parameter, to generate a target speech feature vector; the anomaly detection unit detects an anomaly in the anomaly detection target based on the reference speech feature vector and the target speech feature vector. A generating device characterized by:

5. 5. The generating device of claim 4, a summarizing unit that generates a summary indicating the reason why the anomaly is detected by the anomaly detection unit; the language decoding unit decodes, based on the third learning parameter, the reference speech feature vector based on the anomaly detection result by the anomaly detection unit, into a first reason explanation sentence indicating a reason for the anomaly detection, and decodes, based on the third learning parameter, the target speech feature vector into a second reason explanation sentence indicating a reason for the anomaly detection; the summarizing unit generates the summary based on the first basis explanation sentence and the second basis explanation sentence. A generating device characterized by:

6. 3. The generating device of claim 2, an anomaly detection unit that detects an anomaly in an anomaly detection target; the first signal encoding unit encodes a reference speech signal that serves as a reference when the state of the anomaly detection target is normal, based on the fourth learning parameter, to generate a reference speech feature vector; the second signal encoding unit encodes a target signal emitted by the anomaly detection target based on the fifth learning parameter to generate a target speech feature vector; the anomaly detection unit detects an anomaly in the anomaly detection target based on a second difference vector between the reference speech feature vector and the target speech feature vector. A generating device characterized by:

7. 7. The generating device of claim 6, a summarizing unit that generates a summary indicating the reason why the anomaly is detected by the anomaly detection unit; the language decoding unit decodes, based on the third learning parameter, the second difference vector based on the anomaly detection result by the anomaly detection unit, into a first basis explanation sentence indicating a basis for the anomaly detection; the summarizing unit generates the summary sentence based on the first basis explanation sentence. A generating device characterized by:

8. 4. The generating device of claim 3, an anomaly detection unit that detects an anomaly in an anomaly detection target; the first signal encoding unit encodes a reference speech signal that serves as a reference when the state of the anomaly detection target is normal, based on the fourth learning parameter, to generate a reference speech feature vector; the second signal encoding unit encodes a target signal emitted by the anomaly detection target based on the fifth learning parameter to generate a target speech feature vector; the anomaly detection unit detects an anomaly in the anomaly detection target based on a second combined vector obtained by combining the reference speech feature vector, the target speech feature vector, and a second difference vector between the reference speech feature vector and the target speech feature vector. A generating device characterized by:

9. 9. The generating device of claim 8, a summarizing unit that generates a summary indicating the reason why the anomaly is detected by the anomaly detection unit; the language decoding unit decodes, based on the third learning parameter, the second combined vector based on the anomaly detection result by the anomaly detection unit, into a first basis explanation sentence indicating a basis for the anomaly detection; the summarizing unit generates the summary sentence based on the first basis explanation sentence. A generating device characterized by:

10. A generation method executed by a generation device having a processor that executes a program and a storage device that stores the program, the generation device having access to a set of training data sets that are combinations of audio signals that indicate states and explanatory text that explains the states in text, the method comprising: The processor: a signal encoding process for encoding the speech signal based on a first learning parameter to generate a speech feature vector; a language encoding process for encoding the description sentence based on the second learning parameters to generate a language feature vector; a language decoding process for decoding the speech feature vector into text indicating the state based on a third learning parameter; an updating process for updating the first learning parameters and the second learning parameters through contrastive learning using a combination of the speech feature vector generated by the signal encoding process and the language feature vector generated by the language encoding process, and updating the third learning parameters based on a difference between the text indicating the state decoded by the language decoding process and the explanatory text; A generating method characterized by executing the following.

11. A generating device has a processor that executes a program and a storage device that stores the program, and the processor has access to a set of training data sets that are combinations of voice signals that indicate states and explanatory text that explains the states in text, a signal encoding process for encoding the speech signal based on a first learning parameter to generate a speech feature vector; a language encoding process for encoding the description sentence based on the second learning parameters to generate a language feature vector; a language decoding process for decoding the speech feature vector into text indicating the state based on a third learning parameter; an updating process for updating the first learning parameters and the second learning parameters through contrastive learning using a combination of the speech feature vector generated by the signal encoding process and the language feature vector generated by the language encoding process, and updating the third learning parameters based on a difference between the text indicating the state decoded by the language decoding process and the explanatory text; A generating program characterized by executing the above.

Images