Sound signal retrieval device, sound signal retrieval method, data retrieval device, data retrieval method, and program

Abstract

To provide a technology for retrieving, by using a natural language expression regarding a desired sound signal as a query, a sound signal corresponding to the natural language expression by using a data base that can be used for text base sound signal retrieval that is appropriately and efficiently generated.SOLUTION: The sound signal retrieval device includes: a storage unit which records a sound signal database which is constituted with a record including the sound signal and natural language expression corresponding to the sound signal, the natural language expression being generated by using a data generation model being a sound signal encoder which generates, from the sound signal, latent variables corresponding to the sound signal and a natural language expression decoder for generating a natural language expression corresponding to the sound signal from the latent variable and an index of a natural language expression desired to be generated as a natural language expression corresponding to the sound signal; and a retrieval unit which determines the sound signal corresponding to the natural language expression as a retrieval unit from a natural language expression being an input, herein after referred to as an input natural language expression, by using the sound signal database.SELECTED DRAWING: Figure 19

Description

[Technical field]

[0001] The present invention relates to a technique for searching an acoustic signal. [Background technology]

[0002] In recent years, a huge amount of audio signals has been accumulated, and the demand for technology to efficiently search for target audio signals (hereinafter referred to as audio signal search technology) is increasing. For example, when conveying audio information to others, selecting similar sounds from an audio signal database and using them for explanation enables efficient information transmission in various situations such as facility maintenance and inspection, security, and help desk work. In addition, selecting appropriate sound effects from a sound effect database plays an important role in the production of movies, games, music, etc.

[0003] Acoustic signal search techniques include a search method that uses an acoustic signal as a query and a search method that uses text data as a query. In the latter search method that uses text data as a query, a search is performed by matching a classification tag or description given to an acoustic signal with a query. As one of such searches using text data, a search using onomatopoeia as a query has been proposed. By using onomatopoeia used by humans in daily life as a query, more natural human-computer interaction is realized. For example, Non-Patent Document 1 proposes a text-based acoustic signal search based on the text similarity between onomatopoeia tags given in advance to an acoustic signal and an onomatopoeia query as a search using onomatopoeia as a query. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Kaori Okamoto, Yoshinori Yamanishi, Mitsunori Matsushita, “Development of SERVA: An exploratory sound effect search system based on multiple perspectives and user observations”, DEIM Forum 2016, E3-6, 2016. Summary of the Invention [Problem to be solved by the invention]

[0005] However, text-based acoustic signal retrieval using onomatopoeia as a query has the following problems.

[0006] (Problem) It is necessary to assign onomatopoeia tags to all acoustic signals in the database to be searched. If onomatopoeia tags are assigned manually, a large amount of human resources is required to search a large database. If tagging is performed using automatic onomatopoeia tagging, it is not always possible to assign a sufficient number of onomatopoeia tags while maintaining sufficient accuracy.

[0007] Although the above describes text-based acoustic signal retrieval using onomatopoeia as a query, the above problems also apply to text-based acoustic signal retrieval in general when an acoustic signal is tagged with text describing the characteristics of the acoustic signal. In other words, it is difficult to accurately and efficiently generate pairs of sentences describing the characteristics of an acoustic signal and the acoustic signal to be included in a record of a database used in text-based acoustic signal retrieval.

[0008] Therefore, an object of the present invention is to provide a technology for searching for an acoustic signal corresponding to a natural language expression related to a desired acoustic signal, using a database that is accurately and efficiently generated and can be used for text-based acoustic signal search, using the natural language expression as a query. [Means for solving the problem]

[0009] One aspect of the present invention includes an acoustic signal encoder that generates, from an acoustic signal, a latent variable corresponding to the acoustic signal; a recording unit that records an acoustic signal database composed of records including a natural language expression corresponding to the acoustic signal and the acoustic signal, the natural language expression being generated from the acoustic signal using a data generation model that is a natural language expression decoder that generates a natural language expression corresponding to the acoustic signal from the latent variables and an indicator of a natural language expression to be generated as a natural language expression corresponding to the acoustic signal; and a search unit that uses the acoustic signal database to determine, as a search result, an acoustic signal corresponding to an input natural language expression (hereinafter referred to as an input natural language expression).

[0010] One aspect of the present invention includes a first domain encoder that generates latent variables corresponding to data of a first domain from data of the first domain; a recording unit that records a first domain database composed of records including natural language expressions corresponding to the data of the first domain and the data of the first domain, the records being generated from the data of the first domain using a data generation model that is a natural language expression decoder that generates a natural language expression corresponding to the data of the first domain from the latent variables and indicators of a natural language expression to be generated as a natural language expression corresponding to the data of the first domain; and a search unit that uses the first domain database to determine data of the first domain corresponding to an input natural language expression (hereinafter referred to as an input natural language expression) as a search result from the input natural language expression. Effect of the Invention

[0011] According to the present invention, it is possible to use a database that can be used in text-based acoustic signal search, which is accurately and efficiently generated, to search for an acoustic signal corresponding to a natural language expression related to a desired acoustic signal as a query. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a diagram for explaining an SCG. [Diagram 2]FIG. 13 is a diagram illustrating the level of detail of a sentence. [Diagram 3] FIG. 13 is a diagram illustrating the level of detail of a sentence. [Figure 4] FIG. 1 is a diagram for explaining CSCG. [Diagram 5] FIG. 13 is a diagram showing experimental results. [Figure 6] FIG. 13 is a diagram showing experimental results. [Figure 7] FIG. 13 is a diagram showing experimental results. [Figure 8] FIG. 13 is a diagram showing experimental results. [Figure 9] FIG. 1 is a diagram illustrating an overview of a data generation model. [Figure 10] FIG. 1 is a block diagram showing a configuration of a data generation model learning device 100. [Figure 11] 4 is a flowchart showing the operation of the data generation model learning device 100. [Figure 12] FIG. 2 is a block diagram showing a configuration of a data generation model learning device 150. [Figure 13] 13 is a flowchart showing the operation of the data generation model learning device 150. [Figure 14] FIG. 2 is a block diagram showing a configuration of a data generating device 200. [Figure 15] 13 is a flowchart showing the operation of the data generating device 200. [Figure 16] FIG. 1 is a diagram showing an overview of an acoustic signal search process. [Figure 17] FIG. 2 is a block diagram showing a configuration of an acoustic signal database generating device 250. [Figure 18] 13 is a flowchart showing the operation of the acoustic signal database generating device 250. [Figure 19] FIG. 3 is a block diagram showing a configuration of an acoustic signal retrieval device 300. [Figure 20] 4 is a flowchart showing the operation of the acoustic signal retrieval device 300. [Figure 21] FIG. 1 is a diagram showing an overview of an acoustic signal search process. [Figure 22] FIG. 4 is a block diagram showing the configuration of an acoustic signal retrieval device 400. [Diagram 23] 4 is a flowchart showing the operation of the acoustic signal retrieval device 400. [Figure 24] FIG. 11 is a block diagram showing a configuration of a data generation model learning device 1100. [Diagram 25] 11 is a flowchart showing the operation of the data generation model learning device 1100. [Figure 26] FIG. 11 is a block diagram showing a configuration of a data generation model learning device 1150. [Figure 27] 11 is a flowchart showing the operation of the data generation model learning device 1150. [Figure 28] FIG. 12 is a block diagram showing a configuration of a data generating device 1200. [Figure 29] 13 is a flowchart showing the operation of the data generating device 1200. [Diagram 30] FIG. 12 is a block diagram showing a configuration of a database generating device 1250. [Diagram 31] 13 is a flowchart showing the operation of the database generating device 1250. [Diagram 32] FIG. 13 is a block diagram showing a configuration of a data search device 1300. [Diagram 33] 13 is a flowchart showing the operation of the data search device 1300. [Diagram 34] FIG. 14 is a block diagram showing a configuration of a data search device 1400. [Diagram 35] 14 is a flowchart showing the operation of the data search device 1400. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described in detail. Note that components having the same functions are given the same reference numbers and duplicated explanations will be omitted.

[0014] Before describing each embodiment, the notation used in this specification will be described.

[0015] ^ (caret) represents a superscript. For example, x y^z Yes z is a superscript to x, and x y^z Yes z is a subscript to x. Also, _ (underscore) represents a subscript. For example, x y_z Yes z is a superscript to x, and x y_z Yes z is a subscript on x.

[0016] The superscripts "^" and "~" such as ^x and ~x for a certain letter x should actually be written directly above the "x", but due to the constraints of the description in the specification, they are written as ^x and ~x.

[0017] <Technical background> In an embodiment of the present invention, a sentence generation model is used when generating a sentence corresponding to an acoustic signal from the acoustic signal. Here, the sentence generation model is a function that receives an acoustic signal as input and outputs a corresponding sentence. Furthermore, the sentence corresponding to the acoustic signal is, for example, a sentence that explains what kind of sound the acoustic signal is (an explanation of the acoustic signal).

[0018] First, we will explain a model called SCG (Sequence-to-sequence Caption Generator) as an example of a sentence generation model.

[0019] 《Saga of the Golden Age》 As shown in FIG. 1, SCG is an encoder-decoder model that employs the Recurrent Language Model (RLM) described in Non-Patent Document 1 in the decoder.

[0020] (Reference non-patent document 1: T. Mikolov, M. Karafiat, L. Burget, J. Cernock`y, and S. Khudanpur, “Recurrent neural network based language model”, In INTERSPEECH 2010, pp.1045-1048, 2010.) The SCG will be described with reference to Fig. 1. The SCG generates a sentence corresponding to an input acoustic signal from the acoustic signal and outputs the generated sentence through the following steps. Note that instead of the acoustic signal, acoustic features extracted from the acoustic signal, for example, a sequence of Mel-Frequency Cepstral Coefficients (MFCC), may be used. A sentence, which is text data, is a string of words.

[0021] (1) SCG uses an encoder to extract a latent variable z, which is a distributed representation of sounds, from the acoustic signal. The latent variable z is expressed as a vector of a certain dimension (for example, 128 dimensions). This latent variable z can be said to be a summary feature of the acoustic signal that contains sufficient information for sentence generation. Therefore, the latent variable z can also be said to be a fixed-length vector that contains the features of both the acoustic signal and the sentence.

[0022] (2) The SCG uses a decoder to extract the word w at time t (t=1, 2, …) from the latent variable z. t The output layer of the decoder generates a sentence by outputting the probability of generating a word at time t, p t From (w), we obtain the word w at time t by the following formula: t Output.

number

[0023] The encoder and decoder that make up the SCG can use any neural network that can process time-series data. For example, a recurrent neural network (RNN) or a long short-term memory (LSTM) can be used. Note that BLSTM and layered LSTM in Figure 1 stand for bi-directional LSTM and multi-layer LSTM, respectively.

[0024] SCG is trained by supervised learning, which uses pairs of an audio signal and a sentence corresponding to the audio signal (the sentence is called training data) as training data. The sum of the cross entropy between the word output by the decoder at time t and the word at time t included in the training data sentence is expressed as an error function L SCG Then, we train the SCG using backpropagation.

[0025] The sentences that are the output of the SCG obtained by the above learning vary in the level of detail of the description. This is due to the following reasons. There is not one correct sentence for one acoustic signal. In other words, there can be many "correct sentences" with various levels of detail for one acoustic signal. For example, there can be multiple correct sentences that describe the state of an acoustic signal, such as "a low sound is produced," "a musical instrument is played for a while," and "a stringed instrument starts to play a low sound, and then the volume slowly decreases," and which of these sentences is preferable varies depending on the situation. For example, there are some situations where a concise description is desired, and other situations where a detailed description is desired. Therefore, if SCG learning is performed without distinguishing between sentences with different levels of detail, the SCG will not be able to control the tendency of the sentences it generates.

[0026] 《Level of detail》 To solve the above problem of variability, we define a specificity, which is an index of the degree of specificity of a sentence. n ] is the specificity I of a sentence s s is defined by the following equation:

number

[0027] The level of detail defined in this way has the following characteristics:

[0028] (1) Sentences that use words that describe concrete objects or actions are more specific (see Figure 2).

[0029] This is because such words occur less frequently and carry more information.

[0030] (2) Sentences that use more words are more specific (see Figure 3).

[0031] The optimal level of detail varies depending on the nature and purpose of the target sound. For example, if you want to describe the sound in more detail, a high level of detail in the sentence is preferable, whereas if you want a concise explanation, a low level of detail in the sentence is preferable. There is also the problem that sentences with high detail tend to be inaccurate. Therefore, it is important to be able to freely control the level of detail according to the granularity of the information required to describe the acoustic signal, and generate sentences that correspond to the acoustic signal. We will explain CSCG (Conditional Sequence-to-sequence Caption Generator) as a model that enables such sentence generation.

[0032] 《CSCG》 Like SCG, CSCG is an encoder-decoder model that uses RLM as the decoder. However, in CSCG, the specificity of the sentence to be generated is controlled by conditioning the decoder (see Figure 4). Conditioning is performed by inputting a specific condition on the specificity of the sentence to the decoder. Here, the specificity condition specifies the specificity of the sentence to be generated.

[0033] The CSCG will be described with reference to Fig. 4. The CSCG generates and outputs a sentence corresponding to an input acoustic signal based on the input acoustic signal and conditions related to the level of detail of the sentence, through the following steps.

[0034] (1) CSCG uses an encoder to extract latent variables z, which are distributed representations of sounds, from the acoustic signal.

[0035] (2) CSCG generates sentences by outputting words at time t (t=1, 2, …) from the latent variable z and the condition C on the sentence detail level using a decoder. The generated sentences have a detail level close to the condition C on the sentence detail level. Figure 4 shows the generated sentence s="Birds are singing" with detail level I s This shows that the condition C on the specificity of a sentence is close to that of

[0036] The CSCG can be trained by supervised learning (hereinafter referred to as first learning) using training data (hereinafter referred to as first training data) which is a set of an acoustic signal and a sentence corresponding to the acoustic signal. The CSCG can also be trained by first learning using the first training data, and supervised learning (hereinafter referred to as second learning) using training data (hereinafter referred to as second training data) which is a set of a level of detail of a sentence and a sentence corresponding to the level of detail. In this case, the CSCG is trained by alternately executing the first learning and the second learning for one epoch at a time, for example.

[0037] (1) First lesson The sentences corresponding to the acoustic signal (i.e., the sentences that are elements of the training data) are manually added. In the first learning, the level of detail of the sentences corresponding to the acoustic signal is calculated and included in the training data. In the first learning, the error L SCG and the error in specificity, L sp The error function L CSCG There are two errors L SCG and L sp For example, the error function L CSCG Then, we can use the linear sum of the two errors as follows:

number

[0038] In addition, the error L sp The specific definition will be described later.

[0039] (2) Second Study When the amount of first training data is small, training the CSCG only by the first training may result in the CSCG over-matching the acoustic signal that is an element of the first training data, making it difficult to appropriately reflect the level of detail. Therefore, in addition to the first training using the first training data, the decoder that constitutes the CSCG is trained by the second training using the second training data.

[0040] In the second learning, a sentence corresponding to the detail level c, which is an element of the second learning data, is generated using the decoder under learning, and the sentence that is an element of the second learning data is used as training data for the generated sentence, and the error L sp The decoder is trained to minimize . The specificity c, which is an element of the second training data, may be generated by a predetermined method, such as random number generation. The sentences, which are elements of the second training data, are sentences having specificity close to the specificity c (i.e., the difference from the specificity c is smaller than or equal to a predetermined threshold).

[0041] Specifically, the error between the generated sentence and a sentence with a specificity close to c is L SCG Regularize using

number

[0042] By performing the second learning in addition to the first learning, the generalization performance of the CSCG can be improved.

[0043] error L sp In the first learning, it can be defined as the difference between the detailedness of the generated sentence and the detailedness of the sentence in the training data, and in the second learning, it can be defined as the difference between the detailedness of the generated sentence and the detailedness given as the training data. In this way, the error L sp When we define , we discretize the output at time t into a single word, so we cannot backpropagate the error. Therefore, to enable learning by the backpropagation method, it is effective to use an estimate of the specificity of the generated sentence instead of the specificity of the generated sentence. For example, the estimated specificity of the generated sentence ^I s As the θ, the one defined by the following formula can be used.

number

[0044] And the error L sp In the first learning, the estimated detail level ^I s and the difference between the detailedness of the sentences in the training data. In the case of the second learning, the estimated detailedness ^I s It is defined as the difference between the level of detail given as training data and the level of detail given as training data.

[0045] "experiment" Here, we explain the results of an experiment to confirm the effectiveness of sentence generation using CSCG. The experiment was conducted for the following two purposes.

[0046] (1) Verification of controllability by level of detail (2) Evaluating the quality of the generated sentences through subjective evaluation of their acceptability. First, the data used in the experiment will be explained. 392 audio sources with explanatory text (supervised learning data) and 579 audio sources without explanatory text (unsupervised learning data) were generated from audio signals (within 6 seconds) that recorded acoustic events such as musical instrument sounds and voices. When generating audio sources with explanatory text, 1 to 4 explanatory texts were added to each audio source. The total number of explanatory texts added here is 1113. These explanatory texts were generated by having subjects listen to each sound source and write a sentence explaining what kind of sound it was. Furthermore, by performing partial deletion and replacement of the above 1113 explanatory texts, the number of explanatory texts was increased to 21726, and an explanatory text database was constructed using the 21726 explanatory texts.

[0047] The experimental results are explained below. The experimental results were evaluated by comparing SCG and CSCG. In the experiment, sentences were generated using a trained SCG and a trained CSCG.

[0048] First, the experimental results for objective (1) will be explained. Figure 5 is a table showing what kind of sentences were generated by SCG and CSCG for the sound source. For example, for the sound source of snapping fingers, the sentence "A light sound is heard for just a moment" (Generated caption) was generated by SCG, and the sentence "Fingers are snapped" was generated by CSCG with a detail level of 20. Figure 6 is a table showing the average and standard deviation of the detail level of each model. These statistics were calculated from the results of generating sentences using 29 sound sources as test data. The table in Figure 6 reveals the following about the detail level.

[0049] (1) SCG has a very large standard deviation in detail.

[0050] (2) CSCG generates sentences with a level of detail that corresponds to the input level of detail c, and the standard deviation is smaller than that of SCG. However, the standard deviation increases as the input level of detail c increases. This is thought to be because there is no description that matches the sound but has a level of detail close to the input level of detail c, resulting in a large variance.

[0051] It can be seen that CSCG is able to suppress variation in the level of detail of the generated sentences and generate sentences appropriate to the level of detail.

[0052] Next, we explain the experimental results for purpose (2). First, we evaluated the subjective acceptability of sentences generated using SCG on a four-point scale. Second, we compared sentences generated using SCG with sentences generated using CSCG.

[0053] In the four-level evaluation, 29 audio sources were used as test data, and 41 subjects responded to all test data. The results are shown in Figure 7. The average score was 1.45, and the variance was 1.28. This shows that sentences generated using SCG received a higher score than "partially applicable" on average.

[0054] In addition, in the comparative evaluation, sentences generated using CSCG and sentences generated using SCG were compared under four conditions (c=20, 50, 80, 100), and the answers that rated CSCG the highest among the four comparative evaluations were selected and tallied. The results are shown in Figure 8. 100 sound sources were used as test data, and 19 subjects were asked to answer the questions. CSCG was rated significantly higher than SCG at a significance level of 1%. The average value was 0.80, and the variance was 1.07.

[0055] Variations in level of detail The specificity is an auxiliary input for controlling the properties of the generated sentence (specifically, the amount of information). As long as it can control the properties of the generated sentence, the specificity can be a single number (scalar value) or a set of numbers (vector). Some examples are given below.

[0056] (Example 1) A method based on the frequency of occurrence of word N-grams, which are sequences of N words This method uses the frequency of occurrence of a sequence of words instead of the frequency of occurrence of a single word. This method can take into account the order of words, which may allow for more appropriate control of the nature of the generated sentences. As with the word occurrence probability, the explanatory sentence database can be used to calculate the occurrence probability of word N-grams. Also, other available corpora may be used instead of the explanatory sentence database.

[0057] (Example 2) Word count method In this method, the degree of specificity is determined by the number of words contained in a sentence. Note that the number of characters may be used instead of the number of words.

[0058] (Example 3) Vector method For example, the level of detail can be a three-dimensional vector consisting of a combination of the word occurrence probability, the word N-gram occurrence probability, and the number of words, as explained above. Also, for example, fields (topics) for classifying words, such as politics, economics, and science, can be established, a dimension can be assigned to each field, and the level of detail can be defined as a vector consisting of a combination of the word occurrence probability in each field. This is believed to make it possible to reflect phrases specific to each field.

[0059] Application Examples The framework for learning SCG / CSCG and generating sentences using SCG / CSCG can be applied to relatively simple sounds such as the sound source shown in Figure 5, as well as more complex sounds such as music and non-sound media. Non-sound media includes images such as paintings, illustrations, and clip art, as well as videos. It may also be industrial design or taste.

[0060] As with SCG / CSCG, it is possible to learn a model that associates these data with sentences corresponding to the data, and generate sentences using the model. For example, in the case of taste, it is possible to generate sentences that describe / comment on wine, agricultural products, etc. by using signals from a taste sensor as input. In this case, signals from an olfactory sensor, tactile sensor, and camera may also be used as input in addition to the taste sensor.

[0061] When non-time series data is handled, the encoder and decoder may be configured using a neural network such as a Convolutional Neural Network (CNN).

[0062] First Embodiment Data generation model learning device 100 The data generation model learning device 100 uses learning data to learn a data generation model to be learned. Here, the learning data includes first learning data, which is a set of an acoustic signal and a natural language expression corresponding to the acoustic signal, and second learning data, which is a set of an index for the natural language expression and a natural language expression corresponding to the index. The data generation model is a function that receives as input a condition related to an acoustic signal and an index for the natural language expression (e.g., the level of detail of a sentence), generates a natural language expression corresponding to the acoustic signal, and outputs the natural language expression. The data generation model is configured as a set of an encoder that generates a latent variable corresponding to the acoustic signal from the acoustic signal, and a decoder that generates a natural language expression corresponding to the acoustic signal from the condition related to the index for the latent variable and the natural language expression (see FIG. 9). The condition related to the index for the natural language expression specifies an index required for the natural language expression to be generated, and the required index may be specified by a single numerical value or may be specified by a range. The encoder and decoder can use any neural network that can process time-series data. In addition to the sentences explained in the <Technical Background>, examples of natural language expressions include phrases consisting of two or more words without a subject and a predicate, and onomatopoeia.

[0063] The data generation model learning device 100 will be described below with reference to Figs. 10 and 11. Fig. 10 is a block diagram showing the configuration of the data generation model learning device 100. Fig. 11 is a flowchart showing the operation of the data generation model learning device 100. As shown in Fig. 10, the data generation model learning device 100 includes a learning mode control unit 110, a learning unit 120, a termination condition determination unit 130, and a recording unit 190. The recording unit 190 is a component that appropriately records information necessary for the processing of the data generation model learning device 100. The recording unit 190 records learning data, for example, before the start of learning.

[0064] The operation of the data generation model learning device 100 will be described with reference to Fig. 11. The data generation model learning device 100 receives first learning data, indices for natural language expressions that are elements of the first learning data, and second learning data, and outputs a data generation model. Note that, instead of receiving the indices for the natural language expressions that are elements of the first learning data, the learning unit 120 may obtain them from the natural language expressions that are elements of the first learning data.

[0065] In S110, the learning mode control unit 110 receives the first learning data, an index for a natural language expression that is an element of the first learning data, and the second learning data, and generates and outputs a control signal for controlling the learning unit 120. Here, the control signal is a signal for controlling the learning mode so as to execute either the first learning or the second learning. The control signal can be, for example, a signal for controlling the learning mode so as to execute the first learning and the second learning alternately.

[0066] In S120, the learning unit 120 receives the first learning data, an index for the natural language expression that is an element of the first learning data, the second learning data, and the control signal output in S110 as inputs. If the learning specified by the control signal is the first learning, the learning unit 120 uses the first learning data and the index for the natural language expression that is an element of the first learning data to learn an encoder that generates a latent variable corresponding to the acoustic signal from the acoustic signal, and a decoder that generates a natural language expression corresponding to the acoustic signal from a condition related to the latent variable and the index for the natural language expression. If the learning specified by the control signal is the second learning, the learning unit 120 uses the second learning data to learn the decoder, and outputs a data generation model that is a combination of the encoder and the decoder, together with information required for the termination condition determination unit 130 to determine the termination condition (for example, the number of times learning has been performed). Whether the learning to be performed is the first learning or the second learning, the learning unit 120 performs the learning in units of one epoch. In addition, the learning unit 120 uses an error function L CSCG The data generation model is trained by backpropagation using the error function L CSCG is defined by the following equation, where λ is a predetermined constant, when the learning to be performed is the first learning:

number

number

[0067] In addition, the error function L CSCG are the two errors L SCG and L sp It is sufficient if the scalar is defined using the above formula.

[0068] Furthermore, when the natural language expression is a sentence, as explained in <Technical Background>, the specificity of the sentence can be used as an index for the natural language expression. In this case, the specificity of the sentence is defined using at least one of the occurrence probability of words included in the sentence defined using at least a predetermined word database, the occurrence probability of word N-grams, the number of words included in the sentence, and the number of characters included in the sentence. For example, the specificity of a sentence can be expressed as follows: s Let us consider a sequence of n words [w1, w2, …, w n The specificity of a sentence s such that ] may be defined as follows:

number

[0069] In addition, the word database may be any database that can define the occurrence probability of a word contained in a sentence, or the occurrence probability of a word N-gram contained in a sentence. For example, the explanatory sentence database described in <Technical Background> can be used as the word database.

[0070] In addition, the decoder output is the estimated specificity of sentence s^I s of,

number

[0071] It should be noted that the level of detail can be defined for phrases as well, in the same way as for sentences.

[0072] In S130, the termination condition determination unit 130 receives as input the data generation model output in S120 and information necessary for determining the termination condition, and determines whether the termination condition, which is a condition for terminating learning, is satisfied (for example, the number of times learning has been performed reaches a predetermined number of repetitions), and if the termination condition is satisfied, outputs the data generation model and terminates the processing, whereas if the termination condition is not satisfied, the processing returns to S110.

[0073] Data generation model learning device 150 The data generation model learning device 150 uses learning data to learn a data generation model to be learned. The data generation model learning device 150 differs from the data generation model learning device 100 in that it executes only the first learning using the first learning data.

[0074] The data generation model learning device 150 will be described below with reference to Figs. 12 and 13. Fig. 12 is a block diagram showing the configuration of the data generation model learning device 150. Fig. 13 is a flowchart showing the operation of the data generation model learning device 150. As shown in Fig. 12, the data generation model learning device 150 includes a learning unit 120, a termination condition determination unit 130, and a recording unit 190. The recording unit 190 is a component that appropriately records information necessary for the processing of the data generation model learning device 150.

[0075] The operation of the data generation model learning device 150 will be described with reference to Fig. 13. The data generation model learning device 150 receives the first learning data and an index for a natural language expression that is an element of the first learning data, and outputs a data generation model. Note that, instead of receiving the index for the natural language expression that is an element of the first learning data, the learning unit 120 may obtain the index from the natural language expression that is an element of the first learning data.

[0076] In S120, the learning unit 120 receives the first learning data and an index for the natural language expression that is an element of the first learning data, learns an encoder and a decoder using the first learning data and the index for the natural language expression that is an element of the first learning data, and outputs a data generation model that is a pair of the encoder and the decoder together with information (e.g., the number of times learning has been performed) that is required for the termination condition determination unit 130 to determine the termination condition. The learning unit 120 executes learning, for example, in units of one epoch. The learning unit 120 also calculates an error function L CSCG The data generation model is trained by backpropagation using the error function L SCG is defined by the following equation, where λ is a predetermined constant.

number

[0077] In S130, the termination condition determination unit 130 receives as input the data generation model output in S120 and information necessary for determining the termination condition, and determines whether the termination condition, which is a condition for terminating learning, is satisfied (for example, whether the number of times learning has been performed has reached a predetermined number of repetitions), and if the termination condition is satisfied, outputs the data generation model and terminates the processing, whereas if the termination condition is not satisfied, the processing returns to S120.

[0078] Data generating device 200 The data generating device 200 generates a natural language expression corresponding to an acoustic signal from conditions related to indicators for the acoustic signal and the natural language expression, using a data generating model trained using the data generating model training device 100 or the data generating model training device 150. Here, the data generating model trained using the data generating model training device 100 or the data generating model training device 150 is also referred to as a trained data generating model. In addition, the encoder and the decoder constituting the trained data generating model are also referred to as a trained encoder and a trained decoder, respectively. It is needless to say that a data generating model trained using a data generating model training device other than the data generating model training device 100 or the data generating model training device 150 may be used.

[0079] The data generating device 200 will be described below with reference to Figs. 14 and 15. Fig. 14 is a block diagram showing the configuration of the data generating device 200. Fig. 15 is a flowchart showing the operation of the data generating device 200. As shown in Fig. 14, the data generating device 200 includes a latent variable generating unit 210, a data generating unit 220, and a recording unit 290. The recording unit 290 is a component that appropriately records information necessary for the processing of the data generating device 200. The recording unit 290 records, for example, a trained data generation model (that is, a trained encoder and a trained decoder) in advance.

[0080] The operation of the data generating device 200 will be described with reference to Fig. 15. The data generating device 200 receives as input an acoustic signal and conditions related to indicators for a natural language expression, and outputs the natural language expression.

[0081] In S210, the latent variable generation unit 210 receives an audio signal as input, generates latent variables corresponding to the audio signal from the audio signal using a trained encoder, and outputs the generated latent variables.

[0082] In S220, the data generation unit 220 receives as input the conditions related to the latent variables and the indices for the natural language expressions output in S210, and generates and outputs a natural language expression corresponding to the acoustic signal using a trained decoder based on the conditions related to the latent variables and the indices for the natural language expressions.

[0083] According to the present embodiment, it is possible to learn a data generation model that uses an index for a natural language expression as an auxiliary input and generates, from an acoustic signal, a natural language expression corresponding to the acoustic signal. Also, according to the present embodiment, it is possible to control the index for the natural language expression from the acoustic signal and generate a natural language expression corresponding to the acoustic signal.

[0084] <Second embodiment> Hereinafter, the encoder and decoder constituting the data generation model trained using the data generation model training device 100 or the data generation model training device 150 are referred to as the acoustic signal encoder and the natural language expression decoder, respectively. The acoustic signal encoder and the natural language expression decoder may also be referred to as the trained acoustic signal encoder and the trained natural language expression decoder, respectively. The data generation model is also referred to as the natural language expression generation model.

[0085] 16 is a diagram showing an outline of sound signal retrieval processing in this embodiment. A sound signal retrieval device 300 uses natural language expressions as queries.

[0086] First, an acoustic signal database generating device 250 that generates an acoustic signal database using a data generation model will be described.

[0087] Acoustic signal database generating device 250 The acoustic signal database generating device 250 uses the trained data generation model to generate an acoustic signal database from a set of conditions related to indicators for (a plurality of) acoustic signals and natural language expressions.

[0088] The acoustic signal database generating device 250 will be described below with reference to Figs. 17 and 18. Fig. 17 is a block diagram showing the configuration of the acoustic signal database generating device 250. Fig. 18 is a flowchart showing the operation of the acoustic signal database generating device 250. As shown in Fig. 17, the acoustic signal database generating device 250 includes a latent variable generating unit 210, a data generating unit 220, an acoustic signal database generating unit 230, and a recording unit 290. The recording unit 290 is a component that appropriately records information required for processing by the acoustic signal database generating device 250. The recording unit 290 records, for example, a trained data generation model (i.e., a trained acoustic signal encoder and a trained natural language expression decoder) in advance.

[0089] The operation of the acoustic signal database generating device 250 will be described with reference to Fig. 18. The acoustic signal database generating device 250 receives a set of conditions relating to an acoustic signal and an index for a natural language expression as input, and outputs an acoustic signal database. The input set of conditions relating to (a plurality of) acoustic signals and an index for a natural language expression may be recorded in a recording device external to the acoustic signal database generating device 250 and read out, or may be recorded in the recording unit 290 and read out.

[0090] In S210, the latent variable generation unit 210 receives an audio signal as input, and generates and outputs latent variables corresponding to the audio signal from the audio signal using a trained audio signal encoder.

[0091] In S220, the data generation unit 220 receives as input the conditions related to the latent variables and the indices for the natural language expressions output in S210, and generates and outputs a natural language expression corresponding to the acoustic signal using a trained natural language expression decoder based on the conditions related to the latent variables and the indices for the natural language expressions.

[0092] In S230, the acoustic signal database generation unit 230 receives as input a natural language expression corresponding to the acoustic signal output in S220 and the input acoustic signal, generates a record including the natural language expression corresponding to the acoustic signal and the acoustic signal from the natural language expression corresponding to the acoustic signal and the acoustic signal, and generates and outputs an acoustic signal database composed of the records.

[0093] The set of conditions relating to the input acoustic signal and the index for the natural language expression may include a condition specifying different indexes for one acoustic signal. By generating an acoustic signal database using such a set, the acoustic signal database is configured to include records including, for one acoustic signal, natural language expressions having different indexes and the acoustic signal itself. In other words, when the acoustic signal database has two or more records including a certain acoustic signal, the records will have two or more types of indexes for the natural language expressions included in the records.

[0094] Acoustic signal retrieval device 300 The acoustic signal retrieval device 300 uses the acoustic signal database generated by the acoustic signal database generation device 250 to retrieve an acoustic signal corresponding to an input natural language expression (hereinafter, referred to as an input natural language expression) from the input natural language expression.

[0095] The acoustic signal retrieval device 300 will be described below with reference to Figs. 19 and 20. Fig. 19 is a block diagram showing the configuration of the acoustic signal retrieval device 300. Fig. 20 is a flowchart showing the operation of the acoustic signal retrieval device 300. As shown in Fig. 19, the acoustic signal retrieval device 300 includes a retrieval unit 330 and a recording unit 390. The recording unit 390 is a component that appropriately records information necessary for processing by the acoustic signal retrieval device 300. The recording unit 390 records, for example, an acoustic signal database in advance.

[0096] The operation of the audio signal retrieval device 300 will be described with reference to Fig. 20. The audio signal retrieval device 300 receives an input natural language expression, and outputs an audio signal corresponding to the input natural language expression.

[0097] In S330, the search unit 330 receives an input natural language expression, and uses the audio signal database to determine and output an audio signal corresponding to the input natural language expression as a search result from the input natural language expression. Here, the search unit 330 determines the search result by calculating the similarity between the input natural language expression and the natural language expression included in the audio signal database. For example, the search unit 330 can determine, as a search result, an audio signal paired with a natural language expression included in the audio signal database that has the highest similarity to the input natural language expression. More generally, where N is an integer equal to or greater than 1, the search unit 330 can determine, as a search result, an audio signal paired with a natural language expression included in N audio signal databases that has a high similarity to the input natural language expression. In addition, the search unit 330 can also determine, as a search result, an audio signal paired with a natural language expression included in an audio signal database that has a similarity to the input natural language expression equal to or greater than a predetermined threshold. Note that when the input natural language expression or the natural language expression included in the audio database is expressed as a string of words, any method for calculating the similarity of the natural language expression can be used by treating it as a word vector. For example, it is possible to use Doc2vec, which treats the meaning of a word as a vector and uses this to calculate vectors of natural language expressions as a collection of words, or cos similarity.

[0098] According to the present embodiment, it is possible to accurately and efficiently generate a database that can be used in text-based acoustic signal retrieval. Specifically, it is possible to generate an acoustic signal database that is composed of records including a natural language expression describing features of an acoustic signal, generated using a data generation model, and the acoustic signal itself. Furthermore, according to the present embodiment, it is possible to use the acoustic signal database to search for an acoustic signal corresponding to a natural language expression related to a desired acoustic signal as a query.

[0099] <Third embodiment> 21 is a diagram showing an outline of the sound signal search process in this embodiment. A sound signal search device 400 uses a sound signal as a query.

[0100] Acoustic signal search device 400 The acoustic signal retrieval device 400 uses the acoustic signal database generated by the acoustic signal database generation device 250 to retrieve an acoustic signal corresponding to an input acoustic signal from an input acoustic signal (hereinafter, referred to as an input acoustic signal).

[0101] The acoustic signal retrieval device 400 will be described below with reference to Figs. 22 and 23. Fig. 22 is a block diagram showing the configuration of the acoustic signal retrieval device 400. Fig. 23 is a flowchart showing the operation of the acoustic signal retrieval device 400. As shown in Fig. 22, the acoustic signal retrieval device 400 includes a latent variable generation unit 210, a data generation unit 220, a retrieval unit 330, and a recording unit 390. The recording unit 390 is a component that appropriately records information necessary for the processing of the acoustic signal retrieval device 400. The recording unit 390 records, for example, an acoustic signal database, a trained acoustic signal encoder, and a trained natural language expression decoder in advance.

[0102] The operation of the sound signal retrieval device 400 will be described with reference to Fig. 23. The sound signal retrieval device 400 receives an input sound signal, and outputs a sound signal corresponding to the input sound signal.

[0103] In S210, the latent variable generation unit 210 receives an input audio signal, and generates and outputs a latent variable corresponding to the input audio signal from the input audio signal using a trained audio signal encoder.

[0104] In S220, the data generation unit 220 receives the conditions related to the indexes for the latent variables and the natural language expressions output in S210 as input, and generates and outputs a natural language expression (hereinafter referred to as a search natural language expression) corresponding to the input acoustic signal from the conditions related to the latent variables and the indexes for the natural language expressions using a trained natural language expression decoder. Here, the number of conditions related to the indexes for the natural language expressions to be input is an integer of 0 or more. When the number of conditions is 0, it means that no conditions related to the indexes for the natural language expressions are input. Also, when the number of conditions is 1 or more, a random number generated by a random number generation unit (not shown) may be used as the index, or a number input by a user using an input unit (not shown) may be used as the index.

[0105] In S330, the search unit 330 receives the search natural language expression output in S220 as an input, and uses the audio signal database to determine an audio signal corresponding to the input audio signal from the search natural language expression as a search result, and outputs it.

[0106] For example, when three or more index-related conditions are input, the data generating unit 220 generates a search natural language expression for each index-related condition (S220), and the searching unit 330 searches for an audio signal that is paired with a natural language expression included in the audio signal database that has the highest similarity for each of the generated search natural language expressions, and determines a set of these audio signals as the search result. Note that, instead of searching for the highest similarity, it is of course possible to search for an audio signal that is equal to or greater than a predetermined threshold.

[0107] According to the present embodiment, it is possible to use the acoustic signal database to search for an acoustic signal corresponding to a given acoustic signal as a query.

[0108] <Fourth embodiment> Hereinafter, a domain is defined as a set of a certain type of data. Examples of domains include the acoustic signal domain, which is a set of acoustic signals used in the first embodiment, and the natural language expression domain, which is a set of natural language expressions. Examples of domain data include various signals obtained using taste sensors, olfactory sensors, tactile sensors, cameras, etc., as described in <Technical Background>. These signals are related to the five human senses, and hereinafter, signals based on sensory information will be used, including acoustic signals.

[0109] Data generation model learning device 1100 The data generation model learning device 1100 uses learning data to learn a data generation model to be learned. Here, the learning data includes first learning data, which is a set of data of a first domain and data of a second domain corresponding to the data of the first domain, and second learning data, which is a set of an index for the data of the second domain and data of the second domain corresponding to the index. The data generation model is a function that receives as input data of the first domain and a condition related to the index for the data of the second domain, generates data of the second domain corresponding to the data of the first domain, and outputs the data. The data generation model is configured as a set of an encoder that generates a latent variable corresponding to the data of the first domain from the data of the first domain, and a decoder that generates data of the second domain corresponding to the data of the first domain from the condition related to the latent variable and the index for the data of the second domain. The condition related to the index for the data of the second domain specifies an index required for the data of the second domain to be generated, and the required index may be specified by a single numerical value or may be specified by a range. The encoder and the decoder can use any neural network that can process data of the first domain and data of the second domain.

[0110] The data generation model learning device 1100 will be described below with reference to Figs. 24 and 25. Fig. 24 is a block diagram showing the configuration of the data generation model learning device 1100. Fig. 25 is a flowchart showing the operation of the data generation model learning device 1100. As shown in Fig. 24, the data generation model learning device 1100 includes a learning mode control unit 1110, a learning unit 1120, a termination condition determination unit 1130, and a recording unit 1190. The recording unit 1190 is a component that appropriately records information required for the processing of the data generation model learning device 1100. The recording unit 1190 records learning data, for example, before the start of learning.

[0111] The operation of the data generation model learning device 1100 will be described with reference to Fig. 25. The data generation model learning device 1100 receives the first learning data, an index for the data of the second domain that is an element of the first learning data, and the second learning data, and outputs a data generation model. Note that, instead of receiving the index for the data of the second domain that is an element of the first learning data, the learning unit 1120 may obtain it from the data of the second domain that is an element of the first learning data.

[0112] In S1110, the learning mode control unit 1110 receives the first learning data, an index for the data of the second domain which is an element of the first learning data, and the second learning data, and generates and outputs a control signal for controlling the learning unit 1120. Here, the control signal is a signal for controlling the learning mode so as to execute either the first learning or the second learning. The control signal can be, for example, a signal for controlling the learning mode so as to execute the first learning and the second learning alternately.

[0113] In S1120, the learning unit 1120 receives the first learning data, an index for the data of the second domain which is an element of the first learning data, the second learning data, and the control signal output in S1110 as input, and when the learning specified by the control signal is the first learning, the learning unit 1120 learns an encoder that generates a latent variable corresponding to the data of the first domain from the data of the first domain using the first learning data and the index for the data of the second domain which is an element of the first learning data, and a decoder that generates data of the second domain corresponding to the data of the first domain from a condition related to the latent variable and the index for the data of the second domain, and when the learning specified by the control signal is the second learning, the learning unit 1120 learns the decoder using the second learning data, and outputs a data generation model that is a combination of the encoder and the decoder together with information (e.g., the number of times learning has been performed) required for the termination condition determination unit 1130 to determine the termination condition. The learning unit 1120 performs the learning in units of one epoch, regardless of whether the learning to be performed is the first learning or the second learning. Furthermore, the learning unit 1120 learns the data generation model by the error backpropagation method using a predetermined error function L. When the learning to be performed is the first learning, the error function L is defined by the following equation, where λ is a predetermined constant:

number

number

[0114] The error function L may be any function defined using two errors L1 and L2.

[0115] In addition, the data of the second domain that is an element of the second learning data is data of the second domain that has an index close to an index that is an element of the second learning data (i.e., the difference between the index and the second domain is smaller than or equal to a predetermined threshold).

[0116] Also, the estimated index ^I of the second domain data s, which is the output of the decoder, s of,

number

[0117] In S1130, the termination condition determination unit 1130 receives as input the data generation model output in S1120 and information necessary for determining the termination condition, and determines whether the termination condition, which is a condition for ending learning, is satisfied (for example, whether the number of times learning has been performed has reached a predetermined number of repetitions), and if the termination condition is satisfied, it outputs the data generation model and terminates the processing, whereas if the termination condition is not satisfied, it returns to the processing of S1110.

[0118] Data generation model learning device 1150 The data generation model learning device 1150 uses learning data to learn a data generation model to be learned. The data generation model learning device 1150 differs from the data generation model learning device 1100 in that it executes only the first learning using the first learning data.

[0119] The data generation model learning device 1150 will be described below with reference to Figs. 26 and 27. Fig. 26 is a block diagram showing the configuration of the data generation model learning device 1150. Fig. 27 is a flowchart showing the operation of the data generation model learning device 1150. As shown in Fig. 26, the data generation model learning device 1150 includes a learning unit 1120, a termination condition determination unit 1130, and a recording unit 1190. The recording unit 1190 is a component that appropriately records information necessary for the processing of the data generation model learning device 1150.

[0120] The operation of the data generation model learning device 1150 will be described with reference to Fig. 27. The data generation model learning device 1150 receives the first learning data and an index for the data of the second domain that is an element of the first learning data, and outputs a data generation model. Note that, instead of receiving the index for the data of the second domain that is an element of the first learning data, the learning unit 1120 may obtain the index from the data of the second domain that is an element of the first learning data.

[0121] In S1120, the learning unit 1120 receives the first learning data and an index for the data of the second domain, which is an element of the first learning data, and learns an encoder and a decoder using the first learning data and the index for the data of the second domain, which is an element of the first learning data, and outputs a data generation model, which is a combination of the encoder and the decoder, together with information (e.g., the number of times learning has been performed) required for the termination condition determination unit 1130 to determine the termination condition. The learning unit 1120 executes learning, for example, in units of one epoch. The learning unit 1120 also learns the data generation model by the error backpropagation method using an error function L. The error function L is defined by the following equation, where λ is a predetermined constant.

number

[0122] In S1130, the termination condition determination unit 1130 receives as input the data generation model output in S1120 and information necessary for determining the termination condition, and determines whether the termination condition, which is a condition for ending learning, is satisfied (for example, whether the number of times learning has been performed has reached a predetermined number of repetitions), and if the termination condition is satisfied, it outputs the data generation model and terminates the processing, whereas if the termination condition is not satisfied, it returns to the processing of S1120.

[0123] Data Generation Device 1200 The data generating device 1200 generates data of a second domain corresponding to data of the first domain from conditions related to indexes for data of the first domain and data of the second domain, using a data generating model trained using the data generating model training device 1100 or the data generating model training device 1150. Here, the data generating model trained using the data generating model training device 1100 or the data generating model training device 1150 is also called a trained data generating model. In addition, the encoder and the decoder constituting the trained data generating model are also called a trained encoder and a trained decoder, respectively. It is of course possible to use a data generating model trained using a data generating model training device other than the data generating model training device 1100 or the data generating model training device 1150.

[0124] The data generating device 1200 will be described below with reference to Figs. 28 and 29. Fig. 28 is a block diagram showing the configuration of the data generating device 1200. Fig. 29 is a flowchart showing the operation of the data generating device 1200. As shown in Fig. 28, the data generating device 1200 includes a latent variable generating unit 1210, a data generating unit 1220, and a recording unit 1290. The recording unit 1290 is a component that appropriately records information required for the processing of the data generating device 1200. The recording unit 1290 records, for example, a trained data generation model (that is, a trained encoder and a trained decoder) in advance.

[0125] The operation of the data generating device 1200 will be described with reference to Fig. 29. The data generating device 1200 receives as input data of the first domain and conditions related to indices for data of the second domain, and outputs data of the second domain.

[0126] In S1210, the latent variable generation unit 1210 receives data of the first domain as input, and generates and outputs latent variables corresponding to the data of the first domain from the data of the first domain using a trained encoder.

[0127] In S1220, the data generation unit 1220 receives as input the conditions related to the latent variables and the indices for the data of the second domain output in S1210, and generates and outputs data of the second domain corresponding to the data of the first domain using a trained decoder based on the conditions related to the latent variables and the indices for the data of the second domain.

[0128] (Specific examples) A specific example will be described below, assuming that the data in the first domain is a signal based on sensory information and the data in the second domain is a sentence or phrase.

[0129] (1) Taste In this case, for example, a description of the place of origin related to the taste can be obtained from the signal from the taste sensor. The description of the place of origin related to the taste can be, for example, a description such as "2015 Koshu wine."

[0130] (2) Sense of smell In this case, a description of the odor is obtained from the signal from the olfactory sensor.

[0131] (3) Touch In this case, for example, a description of the hardness or texture can be obtained from the signal from the tactile sensor or hardness sensor.

[0132] (4) Vision In this case, for example, captions for videos or descriptions of the subjects of images can be obtained from signals from an image sensor such as a camera.

[0133] According to the present embodiment, it is possible to learn a data generation model that uses an index for data of a second domain as an auxiliary input and generates data of a second domain corresponding to the data of a first domain from data of a first domain. Also, according to the present embodiment, it is possible to generate data of a second domain corresponding to the data of the first domain from the data of the first domain by controlling a predetermined index.

[0134] <Fifth embodiment> Hereinafter, the encoder and decoder constituting the data generation model trained using the data generation model training device 1100 or the data generation model training device 1150 are referred to as the first domain encoder and the natural language expression decoder, respectively. The first domain encoder and the natural language expression decoder may also be referred to as the trained first domain encoder and the trained natural language expression decoder, respectively. The data generation model is also referred to as the natural language expression generation model.

[0135] First, a database generating device 1250 that generates a first domain database using a data generation model will be described.

[0136] <<Database generating device 1250>> The database generation device 1250 uses the trained data generation model to generate a first domain database from a set of conditions related to data of (a plurality of) first domains and indicators for natural language expressions.

[0137] Hereinafter, the database generating device 1250 will be described with reference to Figs. 30 and 31. Fig. 30 is a block diagram showing the configuration of the database generating device 1250. Fig. 31 is a flowchart showing the operation of the database generating device 1250. As shown in Fig. 30, the database generating device 1250 includes a latent variable generating unit 1210, a data generating unit 1220, a database generating unit 1230, and a recording unit 1290. The recording unit 1290 is a component that appropriately records information required for the processing of the database generating device 1250. The recording unit 1290 records, for example, a trained data generation model (that is, a trained first domain encoder and a trained natural language expression decoder) in advance.

[0138] The operation of the database generating device 1250 will be described with reference to Fig. 31. The database generating device 1250 receives a set of conditions related to data of the first domain and indexes for natural language expressions, and outputs a first domain database. The input set of data of (a plurality of) first domains and conditions related to indexes for natural language expressions may be recorded in a recording device external to the database generating device 1250 and read, or may be recorded in the recording unit 1290 and read.

[0139] In S1210, the latent variable generation unit 1210 receives data of the first domain as input, and generates and outputs latent variables corresponding to the data of the first domain from the data of the first domain using the trained first domain encoder.

[0140] In S1220, the data generation unit 1220 receives as input the conditions related to the latent variables and the indices for the natural language expressions output in S1210, and generates and outputs natural language expressions corresponding to the data of the first domain using a trained natural language expression decoder from the conditions related to the latent variables and the indices for the natural language expressions.

[0141] In S1230, the database generation unit 1230 receives as input the natural language expression corresponding to the data of the first domain output in S1220 and the input data of the first domain, generates records including the natural language expression corresponding to the data of the first domain and the data of the first domain from the natural language expression corresponding to the data of the first domain and the data of the first domain, generates and outputs a first domain database consisting of the records.

[0142] In addition, the set of conditions related to the input data of the first domain and the index for the natural language expression may include a condition that specifies different indexes for one data of the first domain. By generating the first domain database using such a set, the first domain database is configured to include records including the data and natural language expressions having different indexes for one data of the first domain. In other words, when there are two or more records including data of a certain first domain in the first domain database, there are two or more types of indexes for the natural language expressions included in the records.

[0143] Data Search Device 1300 The data search device 1300 uses the first domain database generated by the database generation device 1250 to search for data of the first domain corresponding to an input natural language expression from an input natural language expression (hereinafter, referred to as an input natural language expression).

[0144] The data search device 1300 will be described below with reference to Figs. 32 and 33. Fig. 32 is a block diagram showing the configuration of the data search device 1300. Fig. 33 is a flowchart showing the operation of the data search device 1300. As shown in Fig. 32, the data search device 1300 includes a search unit 1330 and a recording unit 1390. The recording unit 1390 is a component that appropriately records information necessary for the processing of the data search device 1300. The recording unit 1390 records, for example, a first domain database in advance.

[0145] The operation of the data search device 1300 will be described with reference to Fig. 33. The data search device 1300 receives an input natural language expression, and outputs data of a first domain corresponding to the input natural language expression.

[0146] In S1330, the search unit 1330 receives an input natural language expression, and uses the first domain database to determine and output data of the first domain corresponding to the input natural language expression as a search result from the input natural language expression. Here, the search unit 330 determines the search result by calculating the similarity between the input natural language expression and the natural language expressions included in the first domain database.

[0147] According to the present embodiment, it is possible to accurately and efficiently generate a database that can be used in a text-based data search. Specifically, it is possible to generate a first domain database that is composed of a natural language expression describing the characteristics of data of a first domain, generated using a data generation model, and a record including the data. Furthermore, according to the present embodiment, it is possible to use the first domain database to search for data of the first domain that corresponds to a natural language expression as a query for the natural language expression related to the desired data of the first domain.

[0148] Sixth Embodiment Data Search Device 1400 The data search device 1400 uses the first domain database generated by the database generation device 1250 to search for first domain data corresponding to the input first domain data from the first domain data that is input (hereinafter referred to as input first domain data).

[0149] Hereinafter, the data search device 1400 will be described with reference to Figs. 34 and 35. Fig. 34 is a block diagram showing the configuration of the data search device 1400. Fig. 35 is a flowchart showing the operation of the data search device 1400. As shown in Fig. 34, the data search device 1400 includes a latent variable generation unit 1210, a data generation unit 1220, a search unit 1330, and a recording unit 1390. The recording unit 1390 is a component that appropriately records information required for the processing of the data search device 1400. The recording unit 1390 records, for example, a first domain database, a trained first domain encoder, and a trained natural language expression decoder in advance.

[0150] The operation of the data search device 1400 will be described with reference to Fig. 35. The data search device 1400 receives input first domain data, and outputs first domain data corresponding to the input first domain data.

[0151] In S1210, the latent variable generation unit 1210 receives the input first domain data, and generates and outputs latent variables corresponding to the input first domain data from the input first domain data using the trained first domain encoder.

[0152] In S1220, the data generation unit 1220 receives the conditions related to the indexes for the latent variables and the natural language expressions output in S1210 as input, and generates and outputs a natural language expression (hereinafter referred to as a search natural language expression) corresponding to the input first domain data from the conditions related to the indexes for the latent variables and the natural language expressions using a trained natural language expression decoder. Here, the number of conditions related to the indexes for the natural language expressions to be input is an integer of 0 or more. When the number of conditions is 0, it means that no conditions related to the indexes for the natural language expressions are input. Also, when the number of conditions is 1 or more, a random number generated by a random number generation unit (not shown) may be used as the index, or a number input by a user using an input unit (not shown) may be used as the index.

[0153] In S1330, the search unit 1330 receives the search natural language expression output in S1220 as input, and uses the first domain database to determine and output, as a search result, data of the first domain corresponding to the input first domain data from the search natural language expression.

[0154] According to the present embodiment, it is possible to use the first domain database to search for an acoustic signal corresponding to data of the first domain as a query.

[0155] <Additional Notes> The device of the present invention has, as a single hardware entity, an input section to which a keyboard or the like can be connected, an output section to which a liquid crystal display or the like can be connected, a communication section to which a communication device (e.g., a communication cable) capable of communicating with the outside of the hardware entity can be connected, a CPU (which may also have a central processing unit, cache memory, registers, etc.), memories such as RAM and ROM, an external storage device such as a hard disk, and buses connecting the input section, output section, communication section, CPU, RAM, ROM, and external storage device so that data can be exchanged between them. If necessary, the hardware entity may also be provided with a device (drive) capable of reading and writing a recording medium such as a CD-ROM. A physical entity equipped with such hardware resources is, for example, a general-purpose computer.

[0156] The external storage device of the hardware entity stores the programs required to realize the above-mentioned functions and the data required in the processing of these programs (not limited to the external storage device, but for example the programs may be stored in a ROM, which is a read-only storage device). Data obtained by the processing of these programs is stored appropriately in the RAM, the external storage device, etc.

[0157] In the hardware entity, each program stored in an external storage device (or ROM, etc.) and the data required to process each program are loaded into memory as necessary, and interpreted, executed, and processed by the CPU as appropriate, so that the CPU realizes a predetermined function (each of the components represented as the above, "units," "means," etc.).

[0158] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention. In addition, the processes described in the above-described embodiment are not limited to being executed in chronological order according to the order described, but may be executed in parallel or individually depending on the processing capacity of the device that executes the processes or as necessary.

[0159] As described above, when the processing functions of the hardware entities (the devices of the present invention) described in the above embodiments are realized by a computer, the processing contents of the functions that the hardware entities should have are described by a program. Then, by executing this program on a computer, the processing functions of the hardware entities are realized on the computer.

[0160] The program describing the processing contents can be recorded on a computer-readable recording medium. The computer-readable recording medium may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium. Specifically, for example, a hard disk drive, a flexible disk, a magnetic tape, or the like can be used as the magnetic recording device, a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read Only Memory), a CD-R (Recordable) / RW (ReWritable), or the like can be used as the optical disk, an MO (Magneto-Optical disc), or the like can be used as the magneto-optical recording medium, and an EEP-ROM (Electronically Erasable and Programmable-Read Only Memory) or the like can be used as the semiconductor memory.

[0161] The program may be distributed, for example, by selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of a server computer and transferring the program from the server computer to other computers via a network.

[0162] A computer that executes such a program, for example, first stores the program recorded on a portable recording medium or the program transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored in its own storage device and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from the portable recording medium and execute a process according to the program, or may execute a process according to the received program each time a program is transferred from the server computer to the computer. The server computer may not transfer the program to the computer, but may execute the above-mentioned process by a so-called ASP (Application Service Provider) type service that realizes the processing function only by issuing an execution instruction and obtaining the result. Note that the program in this embodiment includes information used for processing by an electronic computer that is equivalent to a program (data that is not a direct instruction to the computer but has a nature that specifies the processing of the computer, etc.).

[0163] In addition, in this embodiment, a hardware entity is configured by executing a specific program on a computer, but at least a part of the processing contents may be realized by hardware.

[0164] The foregoing description of the embodiments of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Modifications and variations are possible in light of the above teachings. The embodiments have been chosen and depicted to provide a best illustration of the principles of the invention and to enable those skilled in the art to utilize the invention in various embodiments and with various modifications as may be suitable for the practical use contemplated. All such modifications and variations are within the scope of the invention as defined by the appended claims interpreted in accordance with the breadth to which they are fairly, legally, and equitably entitled.< / eos> < / bos>

Claims

1. an audio signal encoder that generates, from an audio signal, a latent variable corresponding to the audio signal; and a recording unit that records an audio signal database including records including the audio signal and a natural language expression corresponding to the audio signal, the natural language expression being generated from the audio signal using a data generation model that is a natural language expression decoder that generates a natural language expression corresponding to the audio signal from the latent variables and an index of a natural language expression to be generated as a natural language expression corresponding to the audio signal; a search unit that uses the audio signal database to determine, from an input natural language expression (hereinafter referred to as an input natural language expression), an audio signal corresponding to the input natural language expression as a search result; An acoustic signal retrieval device comprising:

2. 2. The acoustic signal retrieval device according to claim 1, The acoustic signal database includes: For a given acoustic signal, if there are two or more records that include the acoustic signal, there are two or more natural language expressions that have different index values ​​among the natural language expressions included in the two or more records.

1. An acoustic signal retrieval device comprising:

3. a search step in which an audio signal retrieval device includes an audio signal encoder that generates from an audio signal a latent variable corresponding to the audio signal, and a recording unit that records an audio signal database including records including a natural language expression corresponding to the audio signal and the audio signal, the natural language expression being generated from the audio signal using a data generation model that is a natural language expression decoder that generates a natural language expression corresponding to the audio signal from the latent variables and an index of a natural language expression to be generated as a natural language expression corresponding to the audio signal, and determines an audio signal corresponding to the input natural language expression as a search result from an input natural language expression (hereinafter referred to as an input natural language expression) using the audio signal database; 2. A method for retrieving an acoustic signal comprising:

4. a first domain encoder that generates latent variables corresponding to the data of the first domain from the data of the first domain; and a recording unit that records a first domain database composed of records including the data of the first domain and natural language expressions corresponding to the data of the first domain, the records including the data of the first domain and the data of the first domain, the natural language expressions being generated from the data of the first domain using a data generation model that is a natural language expression decoder that generates natural language expressions corresponding to the data of the first domain from the latent variables and indicators of the natural language expressions to be generated as natural language expressions corresponding to the data of the first domain; a search unit that uses the first domain database to determine, from an input natural language expression (hereinafter referred to as an input natural language expression), data of the first domain corresponding to the input natural language expression as a search result; A data retrieval device comprising:

5. a data search device including a first domain encoder for generating latent variables corresponding to data of a first domain from the data of a first domain, and a recording unit for recording a first domain database including records including natural language expressions corresponding to the data of the first domain and the data of the first domain, the records including the data of the first domain and natural language expressions corresponding to the data of the first domain, the natural language expressions being generated from the data of the first domain using a data generation model that is a first domain encoder for generating latent variables corresponding to the data of the first domain from the data of the first domain, and an index of a natural language expression to be generated as a natural language expression corresponding to the data of the first domain from the latent variables and an index of a natural language expression to be generated as a natural language expression corresponding to the data of the first domain; and a search step for determining data of the first domain corresponding to the input natural language expression as a search result from a natural language expression to be input (hereinafter referred to as an input natural language expression) using the first domain database; A data retrieval method including:

6. 5. A program for causing a computer to function as either the acoustic signal retrieval device according to claim 1 or 2 or the data retrieval device according to claim 4.

Images