Information conversion device and information conversion method

Abstract

An information conversion device according to an embodiment comprises a model training unit. The model training unit trains a machine learning model by using, as input data, an acoustic signal indicating acoustic characteristics of a hand with respect to sound waves, and a shape image indicating the shape of the hand as correct answer data.

Description

Information conversion device and information conversion method

[0001] The present invention relates to an information conversion device and an information conversion method.

[0002] There is a conventional method for estimating hand shape using an acoustic sensor attached to the hand. Specifically, this conventional method uses two piezoelectric elements attached to the back of the hand, one of which emits ultrasonic waves and the other piezoelectric element captures the ultrasonic waves that propagate through the surface and interior of the hand. This conventional method estimates the hand shape from the acoustic signals associated with the captured ultrasonic waves using a machine learning model (see Non-Patent Document 1).

[0003] Yuki Kubo, Yuto Koguchi, Buntarou Shizuki, Shin Takahashi, and Otmar Hilliges. AudioTouch: Minimally Invasive Sensing of Micro-Gestures via Active Bio-Acoustic Sensing. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '19), Article No. 36, Oct. 2019.

[0004] However, in conventional methods, in order to classify hand shapes (or motion states) into multiple types using a machine learning model, it is necessary to assign as many labels as the number of types to be classified to the acoustic signal. In this case, it is very costly to distinguish between mutually similar hand shapes and assign labels for each hand shape to the acoustic signal.

[0005] An object of the present invention is to omit labeling of acoustic signals.

[0006] An information conversion device according to an embodiment includes a model training unit that trains a machine learning model using an acoustic signal representing the acoustic characteristics of a hand in response to sound waves as input data and a shape image representing the shape of the hand as ground truth data.

[0007] According to the present invention, it is possible to omit labeling of acoustic signals.

[0008] FIG. 1 is a configuration diagram showing an information conversion system according to an embodiment. FIG. 2 is a configuration diagram showing a model training unit of an information conversion device according to an embodiment. FIG. 3 is a flow diagram showing the training phase of a machine learning model. FIG. 4 is a diagram showing two types of shape images. FIG. 5 is a diagram showing two types of shape images and ground truth data. FIG. 6 is a flow diagram showing an acoustic signal acquisition process. FIG. 7 is a flow diagram showing a machine learning model training process. FIG. 8 is a configuration diagram showing an encoder of a machine learning model. FIG. 9 is a configuration diagram showing a shape image decoder of a machine learning model. FIG. 10 is a flow diagram showing the application phase of a machine learning model. FIG. 11 is a diagram showing an application example of a machine learning model. FIG. 12 is a hardware configuration diagram of an information conversion device according to an embodiment.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Parts with the same reference numerals are considered to operate in the same manner, and redundant description will be omitted as appropriate.

[0010] 1 is a configuration diagram showing an information conversion system 100 according to an embodiment. The information conversion system 100 is a system for converting information. The information conversion system 100 includes an information conversion device 1, a sonic transmitter 2A, a sonic receiver 2B, and a display device 3.

[0011] The information conversion device 1 is a device that converts information. The information conversion device 1 includes a shape image acquisition unit 11A, an acoustic signal acquisition unit 11C, a training data creation unit 12, a storage unit 13, a model training unit 14, and a model application unit 15. The information conversion device 1 may further include at least one of an acoustic wave transmitter 2A, an acoustic wave receiver 2B, and a display device 3.

[0012] The shape image acquisition unit 11A is a means for acquiring a shape image SG. The shape image SG is an image showing the shape of a hand. The shape image SG may be an image related to the hand (e.g., a depth image or a color image) or model data (e.g., 3D model data). The shape image SG may be captured by a camera (e.g., a depth camera or a color camera). The shape image acquisition unit 11A inputs the acquired shape image SG to the training data creation unit 12.

[0013] The acoustic signal acquisition unit 11C is a means for acquiring an acoustic signal AS (or acoustic feature quantity). The acoustic signal AS is a signal that indicates the acoustic characteristics of the hand in response to sound waves. The acoustic signal AS may be a spectrum (frequency spectrum) that indicates a frequency response in a predetermined frequency band (e.g., 20 kHz to 40 kHz). The acoustic signal acquisition unit 11C is communicably connected to the acoustic transmitter 2A and the acoustic receiver 2B. The acoustic signal acquisition unit 11C transmits a sound wave transmission signal to the acoustic transmitter 2A. The acoustic signal acquisition unit 11C receives a sound wave reception signal from the acoustic receiver 2B. The acoustic signal acquisition unit 11C extracts a reception signal at regular intervals from time-series data of the reception signal and acquires (or generates) the acoustic signal AS by performing a fast Fourier transform (FFT) on the extracted reception signal. The acoustic signal acquisition unit 11C inputs the acquired acoustic signal AS to the training data creation unit 12 during model training, and to the model application unit 15 during model application (inference).

[0014] The training data creation unit 12 is a means for creating training data TD. The training data TD is data in which the shape image SG and the acoustic signal AS are associated with each other. The training data creation unit 12 creates the training data TD by associating the acoustic signal AS as input data and the shape image SG as correct answer data. The training data creation unit 12 stores the created training data TD in the storage unit 13.

[0015] The storage unit 13 is a means for storing various types of data. The storage unit 13 stores the training data TD and the machine learning model ML. The storage unit 13 is connected to the training data creation unit 12, the model training unit 14, and the model application unit 15 so that they can communicate with each other. The storage unit 13 may be a database.

[0016] The model training unit 14 is a means for training the machine learning model ML. The model training unit 14 trains the machine learning model ML using training data TD. The trained machine learning model ML is also called a trained model (see FIG. 2).

[0017] The model application unit 15 is a means for applying the machine learning model ML. The model application unit 15 inputs the acoustic signal AS to the trained machine learning model ML, thereby obtaining a shape image SG from the trained machine learning model ML. That is, the model application unit 15 converts the acoustic signal AS into a shape image SG through the machine learning model ML. The model application unit 15 transmits the obtained shape image SG to the display device 3.

[0018] The ultrasonic wave transmitter 2A is a device that transmits ultrasonic waves. Based on a transmission signal from the acoustic signal acquisition unit 11C, the ultrasonic wave transmitter 2A generates ultrasonic waves having the same frequency characteristics as the transmission signal. The ultrasonic wave transmitter 2A transmits the generated ultrasonic waves to the hand. The ultrasonic wave transmitter 2A may be a device that converts between an electrical signal and vibration (e.g., a piezoelectric element).

[0019] The ultrasonic receiver 2B is a device that receives ultrasonic waves. Based on the ultrasonic waves that have propagated through the surface and inside of the hand, the ultrasonic receiver 2B generates a reception signal corresponding to the vibration of the ultrasonic waves. The ultrasonic receiver 2B transmits the generated reception signal to the acoustic signal acquisition unit 11C. The ultrasonic receiver 2B may be a device that converts between an electrical signal and vibration (e.g., a piezoelectric element).

[0020] The ultrasonic transmitter 2A and the ultrasonic receiver 2B may be two (or a pair) of piezoelectric elements. The ultrasonic transmitter 2A and the ultrasonic receiver 2B may be devices that transmit vibrations while in contact with the hand. The ultrasonic transmitter 2A and the ultrasonic receiver 2B may be attached to the back of the hand with double-sided adhesive tape for use on a living body.

[0021] The display device 3 is a device that displays various types of data. The display device 3 displays the shape image SG from the model application unit 15. The display device 3 may be a display (e.g., a liquid crystal display or an organic EL display).

[0022] 2 is a configuration diagram showing the model training unit 14 of the information conversion device 1 according to the embodiment. The model training unit 14 includes a parameter update unit 141 and a loss calculation unit 142.

[0023] The parameter update unit 141 is a means for updating the parameters of the machine learning model ML. The parameter update unit 141 acquires training data TD from the storage unit 13. The parameter update unit 141 inputs the acoustic signal AS from the acquired training data TD as input data to the machine learning model ML, and inputs the shape image SG as ground truth data to the loss calculation unit 142. The parameter update unit 141 trains the machine learning model ML by updating the parameters of the machine learning model ML (e.g., weights of the encoder and decoder) in accordance with the loss from the loss calculation unit 142.

[0024] The machine learning model ML is a vector quantized latent variable z q (x). The structure of the machine learning model ML is based on a Vector Quantized-Variational AutoEncoder (VQ-VAE). The machine learning model ML includes an encoder EN, a vector quantization unit VQ, and a shape image decoder SD.

[0025] The encoder EN converts the acoustic signal AS from the parameter update unit 141 into a latent variable z e The structure of the encoder EN is based on a Convolutional Neural Network (CNN) (see Figure 8). The encoder EN encodes the latent variable z e (x) is input to the vector quantization unit VQ.

[0026] The vector quantization unit VQ calculates the latent variables z e The vector quantization unit VQ is a means for vector quantizing the vector-quantized latent variable z q (x) is input to the shape image decoder SD. A general method can be applied to vector quantization.

[0027] The shape image decoder SD uses the latent variables z q The shape image decoder SD is a means for decoding (x) into a shape image SG. The structure of the shape image decoder SD is based on a deconvolutional neural network (DNN) (see FIG. 9). The shape image decoder SD inputs the shape image SG to the loss calculation unit 142.

[0028] The loss calculation unit 142 is a means for calculating a loss related to the difference between the shape image SG from the machine learning model ML and the shape image SG from the parameter update unit 141. The loss calculation unit 142 calculates a loss (first loss) related to the difference between the shape image SG output from the machine learning model ML for the acoustic signal AS that is input data and the shape image SG that is the correct answer data.

[0029] The loss calculation unit 142 calculates the loss using a loss function for training the machine learning model ML. The loss function includes the first loss. The loss calculation unit 142 may weight the first loss. The loss calculation unit 142 inputs the calculated loss to the parameter update unit 141.

[0030] FIG. 3 is a flow diagram showing the training phase of the machine learning model ML. In the training phase, the information conversion device 1 uses the sonic transmitter 2A and the sonic receiver 2B to acquire an acoustic signal AS indicating the acoustic characteristics of the hand in response to sound waves. The acoustic characteristics change depending on the motion state of the hand (e.g., the degree of flexion of the fingers, the pressure applied to the fingers). The information conversion device 1 acquires a shape image SG related to the acoustic characteristics. The information conversion device 1 creates training data TD using the acquired acoustic signal AS and shape image SG, and trains the machine learning model ML using the created training data TD. The information conversion device 1 executes a series of processes related to steps SA1 to SA5.

[0031] (Step SA1) First, the acoustic signal acquisition unit 11C acquires the acoustic signal AS. Specifically, the acoustic signal acquisition unit 11C acquires the acoustic signal AS using the acoustic wave transmitter 2A and the acoustic wave receiver 2B (see FIG. 6).

[0032] (Step SA2) Next, the shape image acquisition unit 11A acquires a shape image SG. For example, the shape image acquisition unit 11A acquires a depth image of 512 x 512 size from a depth camera. The shape image acquisition unit 11A may acquire the shape image SG from an external database. Step SA2 may be performed before step SA1.

[0033] The shape image SG includes an image region of the hand (hereinafter also referred to as the "hand region") and a background region other than the image region. The shape image acquisition unit 11A may extract the image region of the hand from the shape image SG by removing the background region from the shape image SG. The shape image acquisition unit 11A may remove the background region by specifying a range of pixel values ​​for the shape image SG.

[0034] (Step SA31) Here, the shape image acquisition unit 11A determines whether the proportion of the hand image area is equal to or greater than one-fourth. Specifically, for the shape image SG acquired in step SA2, the shape image acquisition unit 11A determines whether the proportion of the hand image area to the entire area of ​​the shape image SG (hereinafter also referred to as the "hand area occupancy proportion") is equal to or greater than one-fourth. If the proportion is equal to or greater than one-fourth (step SA31—YES), the process proceeds to step SA32. If the proportion is less than one-fourth (step SA31—NO), the process proceeds to step SA4. Alternatively, the shape image acquisition unit 11A may determine whether the proportion of the hand image area is equal to or greater than an arbitrary threshold (e.g., an arbitrary fraction less than 1; one-half, one-third, two-thirds).

[0035] (Step SA32) Next, the shape image acquisition unit 11A reduces the image area of ​​the hand. Specifically, for the shape image SG determined in step SA31 that the ratio is equal to or greater than one-quarter, the shape image acquisition unit 11A reduces the image area of ​​the hand relative to the entire area of ​​the shape image SG. After step SA32, the process returns to step SA31. By repeating the processes of steps SA31 and SA32, the image area of ​​the hand is reduced until the ratio becomes less than one-quarter (see FIG. 4).

[0036] (Step SA4) Next, the training data creation unit 12 creates training data TD. Specifically, the training data creation unit 12 uses the acoustic signal AS acquired in step SA1 and the shape image SG determined in step SA31 to have the above ratio of less than one-fourth. The training data creation unit 12 creates training data TD by associating the acoustic signal AS with the shape image SG as input data and the correct answer data. The training data creation unit 12 stores the created training data TD in the storage unit 13.

[0037] (Step SA5) Finally, the model training unit 14 trains the machine learning model ML. Specifically, the model training unit 14 trains the machine learning model ML using the training data TD created in step SA4. The model training unit 14 may determine whether the training of the machine learning model ML is complete. For example, the model training unit 14 determines that the training of the machine learning model ML is complete when the loss calculated from the loss function reaches or exceeds a threshold. Completion of the training of the machine learning model ML generates a trained machine learning model ML (see FIG. 7 ).

[0038] 4A and 4B show two types of shape images SG. Fig. 4A shows a shape image SG1 in which the hand region occupancy ratio is equal to or greater than one-quarter. Fig. 4B shows a shape image SG2 in which the hand region occupancy ratio is less than one-quarter. Shape image SG1 has a hand region H1, and shape image SG2 has a hand region H2. Shape images SG1 and SG2 may have the same image size (e.g., 128 x 128).

[0039] The shape image acquisition unit 11A may generate the shape image SG2 by reducing the hand region H1 relative to the entire area of ​​the shape image SG1. For example, the shape image acquisition unit 11A reduces the vertical and horizontal sizes of the shape image SG1 by half, thereby reducing the shape image SG1 to a quarter of its original image size (e.g., 64 x 64). Next, the shape image acquisition unit 11A generates a shape image SG2 that includes the reduced shape image SG1 and has the same image size (e.g., 128 x 128) as the original shape image SG1. At this time, the shape image acquisition unit 11A generates the shape image SG2 by filling the image area other than the reduced shape image SG1 with zero values. The shape image acquisition unit 11A may reduce the hand region occupancy ratio of the shape image SG1 to less than a quarter by any method.

[0040] 5A and 5B are diagrams showing two types of shape images SG and correct answer data. FIG. 5A shows two types of shape images SG output from the machine learning model ML when the same acoustic signal AS is input, and a shape image SG that is correct answer data corresponding to the input acoustic signal AS. FIG. 5A shows two types of shape images SG11 and SG12 output for an acoustic signal AS corresponding to "input 1," and a shape image SG13 that is correct answer data corresponding to this acoustic signal AS. FIG. 5B shows two types of shape images SG21 and SG22 output for an acoustic signal AS corresponding to "input 2," and a shape image SG23 that is correct answer data corresponding to this acoustic signal AS.

[0041] The shape images SG11 and SG21 corresponding to "output 1" are output images from the machine learning model ML trained with a shape image SG having a normal image size. The shape images SG12 and SG22 corresponding to "output 2" are output images from the machine learning model ML trained with a shape image SG having an image size reduced to one-fourth the normal size. The shape images SG13 and SG23 corresponding to "correct data" are correct images corresponding to "input 1" and "input 2", respectively.

[0042] As shown in FIG. 5A, shape images SG11, SG12, and SG13 have hand regions H11, H12, and H13, respectively. Hand regions H11 to H13 show hands with their palms open. Hand region H11 contains more noise than hand region H12. The outline of hand region H11 is less clear than the outline of hand region H12, and there is a lot of noise around the fingertips of hand region H11. The outline of hand region H12 is as clear as the outline of hand region H13.

[0043] As shown in FIG. 5B, shape images SG21, SG22, and SG23 have hand regions H21, H22, and H23, respectively. Hand regions H21 to H23 show the thumb bent into the palm. Hand region H21 contains more noise than hand region H22. The outline of hand region H21 is less clear than the outline of hand region H22, and there is a lot of noise around the fingertips of hand region H21. The outline of hand region H22 is as clear as the outline of hand region H23.

[0044] In this way, when trained with a shape image SG in which the hand region occupancy ratio is less than one-quarter, the machine learning model ML can generate a clearer shape image SG with reduced noise.

[0045] 6 is a flow diagram showing the process of acquiring the acoustic signal AS. The acoustic signal acquisition unit 11C, the acoustic wave transmitter 2A, and the acoustic wave receiver 2B execute a series of processes relating to steps SA11 to SA16.

[0046] (Step SA11) First, the acoustic signal acquisition unit 11C generates a transmission signal. Specifically, the acoustic signal acquisition unit 11C generates the transmission signal based on arbitrarily set parameters. For example, the acoustic signal acquisition unit 11C generates a transmission signal for transmitting ultrasound that sweeps a frequency band from 20 kHz to 40 kHz. The acoustic signal acquisition unit 11C may set parameters regarding whether or not to sweep, the use of other frequency bands, etc., and generate the transmission signal based on the set parameters.

[0047] (Step SA12) Next, the ultrasonic wave transmitter 2A transmits ultrasonic waves. Specifically, the ultrasonic wave transmitter 2A generates ultrasonic waves corresponding to the transmission signal generated in step SA11. The ultrasonic wave transmitter 2A transmits the generated ultrasonic waves to the hand, thereby applying vibrations to the hand. The frequency of the applied vibrations may include the frequency of the vibrations used to generate the hand shape feature amounts registered in the database. In this case, the frequency of the applied vibrations may include other frequencies.

[0048] (Step SA13) Next, the sound wave receiver 2B receives the sound waves. Specifically, the sound wave receiver 2B receives the sound waves that were transmitted to the hand in step SA12 and that have propagated through the surface and interior of the hand. At this time, the acoustic characteristics of the sound waves change depending on the movement state of the hand, which is the propagation path of the sound waves.

[0049] (Step SA14) Next, the sonic wave receiver 2B generates a received signal. Specifically, the sonic wave receiver 2B generates a received signal corresponding to the sonic waves received in step SA13. The generated received signal is attenuated as the sonic waves propagate through the object (particularly the hand).

[0050] (Step SA15) Next, the acoustic signal acquirer 11C amplifies the received signal. Specifically, the acoustic signal acquirer 11C amplifies the received signal generated in step SA14 to a level at which the acoustic signal AS can be extracted. The acoustic signal acquirer 11C may amplify the received signal using an audio interface.

[0051] (Step SA16) Next, the acoustic signal acquisition unit 11C extracts an acoustic signal AS. Specifically, the acoustic signal acquisition unit 11C extracts a received signal at regular intervals from the received signal amplified in step SA15. For example, the acoustic signal acquisition unit 11C extracts the received signal at regular intervals (e.g., 8,192 samples). The acoustic signal acquisition unit 11C performs a fast Fourier transform (FFT) on the extracted received signal to extract an acoustic signal AS that indicates the acoustic characteristics of the received signal. After step SA16, the process proceeds to step SA2 (see FIG. 3 ).

[0052] 7 is a flow diagram showing the training process of the machine learning model ML. The machine learning model ML, the parameter update unit 141, and the loss calculation unit 142 execute a series of processes relating to steps SA51 to SA55. By repeating this series of processes, the machine learning model ML is trained to output a shape image SG from the acoustic signal AS.

[0053] (Step SA51) First, the machine learning model ML converts the acoustic signal AS into a latent variable z e Specifically, the encoder EN compresses the audio signal AS using a convolutional neural network (CNN).

[0054] For example, assume that the acoustic signal AS has a feature quantity of (192 × 1). Furthermore, assume that the number of types of embedded representations K is 16 (K = 16) and the dimension D of the embedded representation is 256 (D = 256). When the feature quantity is input, the encoder EN generates a latent variable z e (x) is output (see Figure 8).

[0055] (Step SA52) Next, the machine learning model ML calculates the latent variable z e (x) is vector quantized and the latent variable z q Specifically, the vector quantization unit VQ generates the latent variable z e For (x), the embedding representation space e∈R （Ｋ×Ｄ） K types of embedding vectors e j In this case, the probability q(z=k|x) that z=k is expressed by the following formula (1):

[0056] (Step SA53) Next, the machine learning model ML converts the vector-quantized latent variable z q (x) into a shape image SG. The shape image decoder SD uses the selected k-th embedded representation. The selection of the k-th embedded representation is expressed by the following equation (2). The shape image decoder SD uses the latent variable z q(x) is converted into a shape image SG by a structure based on a deconvolution network (DNN) (see FIG. 9).

[0057] The shape image SG obtained in step SA53 has the same number of channels as the shape image SG used to train the machine learning model ML.

[0058] (Step SA54) Subsequently, the loss calculation unit 142 calculates the loss using a loss function. Specifically, the loss calculation unit 142 sets a loss function for optimizing the encoder EN and the shape image decoder SD, and calculates the loss. The loss function includes three elements (reconstruction loss, vector quantization error, and encoder loss).

[0059] The loss calculation unit 142 calculates the reconstruction loss, which is a loss related to the difference between the output image and the ground truth image, using the mean squared error (MSE). The loss calculation unit 142 calculates the loss (first loss) related to the difference between the shape image SG output from the machine learning model ML and the shape image SG that is the ground truth data.

[0060] (Step SA55) Finally, the parameter updater 141 updates the parameters of the machine learning model ML. Specifically, the parameter updater 141 updates the parameters of the machine learning model ML using the loss calculated in step SA54. The parameter updater 141 may use ADAM (Adaptive Moment Estimation) as an optimizer. Any type of library may be used as the optimizer.

[0061] 8 is a block diagram showing the encoder EN of the machine learning model ML. The encoder EN converts a (192×1) acoustic signal AS into a (4×4×64) latent variable z e (x). The encoder EN includes three one-dimensional convolution layers, three batch normalization layers, three ReLU (Rectified Linear Unit) layers, and one reshape layer.

[0062] The parameter update unit 141 inputs a one-channel acoustic signal AS (192 × 1) to a first one-dimensional convolutional layer "Conv1d, 3 × 1." The first one-dimensional convolutional layer "Conv1d, 3 × 1" performs convolution processing on the input acoustic signal AS using a 3 × 1 kernel to generate a (64 × 1 × 256) intermediate signal. The batch normalization layer batch normalizes the generated intermediate signal. The ReLU layer transforms the normalized intermediate signal using the ReLU function, and inputs the (64 × 1 × 256) intermediate signal to a second one-dimensional convolutional layer "Conv1d, 2 × 1."

[0063] The second one-dimensional convolutional layer "Conv1d,2×1" performs convolution processing on the input intermediate signal using a 2×1 kernel to generate a (32×1×128) intermediate signal. The batch normalization layer batch normalizes the generated intermediate signal. The ReLU layer transforms the normalized intermediate signal using the ReLU function, and inputs the (32×1×128) intermediate signal to the third one-dimensional convolutional layer "Conv1d,2×1".

[0064] The third one-dimensional convolutional layer "Conv1d,2×1" performs convolution processing on the input intermediate signal using a 2×1 kernel to generate a (16×1×64) intermediate signal. The batch normalization layer batch normalizes the generated intermediate signal. The ReLU layer transforms the normalized intermediate signal using the ReLU function and inputs the (16×1×64) intermediate signal to the reshape layer.

[0065] The reshape layer performs shape transformation on the input intermediate signal using the reshape function, and generates a (4 × 4 × 64) latent variable z e Outputs (x).

[0066] 9 is a diagram showing the configuration of the shape image decoder SD of the machine learning model ML. The shape image decoder SD uses a (4×4×1) vector-quantized latent variable z q (x) is converted into a (128×128×1) monochrome shape image SG. The shape image decoder SD may output the shape image SG as an RGB image.

[0067] The shape image decoder SD includes two recurrence units, a first set of three two-dimensional transposed convolution layers, one batch normalization layer, one ReLU layer, a second set of three other two-dimensional transposed convolution layers, and one two-dimensional transposed convolution layer. Each of the two recurrence units includes three two-dimensional transposed convolution layers, one batch normalization layer, and one ReLU layer. The three two-dimensional transposed convolution layers include two two-dimensional transposed convolution layers "Conv2dTranspose,1×1" with a kernel size of 1×1 and one two-dimensional transposed convolution layer "Conv2dTranspose,2×2" with a kernel size of 2×2.

[0068] The vector quantization unit VQ uses one channel of latent variable z q (x) (4 × 4 × 1) is input to the first repetition unit. The first repetition unit calculates the input latent variable z q A transposed convolution process is performed on (x) using two 1x1 kernels and one 2x2 kernel to generate an (8x8x128) intermediate signal. The batch normalization layer batch normalizes the generated intermediate signal. The ReLU layer transforms the normalized intermediate signal using the ReLU function, and the (8x8x128) intermediate signal is input to the second recursion unit.

[0069] The second recurrence unit performs transposed convolution processing on the input intermediate signals using two 1x1 kernels and one 2x2 kernel to generate a (16x16x64) intermediate signal. The batch normalization layer batch normalizes the generated intermediate signals. The ReLU layer transforms the normalized intermediate signals using the ReLU function and inputs the (16x16x64) intermediate signal to a first set of three two-dimensional transposed convolution layers.

[0070] The first set includes one two-dimensional transposed convolution layer "Conv2dTranspose,1×1" with a kernel size of 1×1 and two two-dimensional transposed convolution layers "Conv2dTranspose,2×2" with kernel sizes of 2×2. The first set performs transposed convolution processing on the input intermediate signals using one 1×1 kernel and two 2×2 kernels to generate (64×64×32) intermediate signals. The batch normalization layer batch normalizes the generated intermediate signals. The ReLU layer transforms the normalized intermediate signals using the ReLU function, and inputs the (64×64×32) intermediate signals to a second set of three other two-dimensional transposed convolution layers.

[0071] The second set includes two two-dimensional transposed convolution layers "Conv2dTranspose,1×1" with a kernel size of 1×1 and one two-dimensional transposed convolution layer "Conv2dTranspose,2×2" with a kernel size of 2×2. The second set performs transposed convolution processing on the input intermediate signals using two 1×1 kernels and one 2×2 kernel to generate (128×128×16) intermediate signals. The second set inputs the generated intermediate signals to one two-dimensional transposed convolution layer.

[0072] One two-dimensional transposed convolution layer performs transposed convolution processing on the input intermediate signal using a 1×1 kernel, and outputs a one-channel shape image SG (128×128×1).

[0073] 10 is a flow diagram showing the application phase of the machine learning model ML. In the application phase, the information conversion device 1 estimates and restores a shape image SG from the acoustic signal AS using the trained machine learning model ML. The information conversion device 1 and the display device 3 execute a series of processes related to steps SB1 to SB3.

[0074] (Step SB1) First, the acoustic signal acquisition unit 11C acquires the acoustic signal AS. Specifically, the acoustic signal acquisition unit 11C acquires the acoustic signal AS using the acoustic wave transmitter 2A and the acoustic wave receiver 2B. Step SB1 is similar to step SA1 (see FIG. 6).

[0075] (Step SB2) Next, the model application unit 15 applies the trained machine learning model ML. Specifically, the model application unit 15 inputs the acoustic signal AS acquired in step SB1 to the trained machine learning model ML, thereby obtaining a shape image SG from the trained machine learning model ML (see FIG. 11 ).

[0076] (Step SB3) Finally, the display device 3 displays the shape image SG. Specifically, the display device 3 displays the shape image SG obtained in step SB2.

[0077] FIG. 11 is a diagram showing an application example of the machine learning model ML. When an acoustic signal AS is input, the trained machine learning model ML outputs a shape image SG. The acoustic signal AS represents a frequency spectrum in a predetermined frequency band. For the acoustic signal AS, the horizontal axis represents frequency and the vertical axis represents signal intensity. The shape image SG represents the shape of a hand in grayscale. The shape image SG includes an image region corresponding to the hand (hand region H).

[0078] As shown in Figure 11 (A), when an acoustic signal AS5 is input, the machine learning model ML outputs a shape image SG5. The shape image SG5 has a hand region H5. The hand region H5 shows a clenched hand with the index finger and middle finger extended. The outline of the hand is clear in the hand region H5.

[0079] As shown in Figure 11 (B), when an acoustic signal AS6 is input, the machine learning model ML outputs a shape image SG6. The shape image SG6 has a hand region H6. The hand region H6 shows a clenched hand. The outline of the hand is clear in the hand region H6.

[0080] 12 is a hardware configuration diagram of an information conversion device 1 according to an embodiment. The information conversion device 1 includes, as its components, a CPU 111, a RAM 112, a ROM 113, a storage 114, an input device 115, and a communication device 116. The components are connected to each other via an internal bus so that they can communicate with each other. The information conversion device 1 may include at least some of the components.

[0081] The CPU 111 is a processor that executes various processes according to programs. The CPU 111 uses a predetermined area of ​​the RAM 112 as a work area. The CPU 111 realizes each unit (e.g., the shape image acquisition unit 11A, the acoustic signal acquisition unit 11C, the training data creation unit 12, the model training unit 14, and the model application unit 15) by reading and executing each program stored in the ROM 113 or the storage 114. Each unit may be realized by a dedicated hardware circuit. The CPU 111 is an example of a processing unit.

[0082] The RAM 112 is a memory that stores various data in a rewritable manner. The RAM 112 is an SDRAM (Synchronous Dynamic Random Access Memory) or the like. The ROM 113 is a memory that stores various data in a non-rewritable manner. The storage 114 is various storage media. The storage 114 may be a drive device that writes or reads various data to or from the storage media. The storage 114 may write or read various data to or from the storage media in accordance with control by the CPU 111. The RAM 112, the ROM 113, or the storage 114 is an example of a storage unit (e.g., the storage unit 13).

[0083] The input device 115 is a device that receives various input operations from a user. The input device 115 is a mouse, a keyboard, etc. The input device 115 receives operations input by the user as instruction signals and inputs the received instruction signals to the CPU 111. The input device 115 is an example of an input unit.

[0084] The communication device 116 communicates with external devices via a network under the control of the CPU 111. The communication device 116 is an example of a communication unit.

[0085] According to the embodiment described above, the information conversion device 1 trains the machine learning model ML using an acoustic signal AS indicating the acoustic characteristics of a hand in response to sound waves as input data and a shape image SG indicating the shape of the hand as ground truth data. The information conversion device 1 inputs the target (or any) acoustic signal AS to the trained machine learning model ML, thereby obtaining a shape image SG from the trained machine learning model ML.

[0086] That is, the information conversion device 1 uses a pair of the acoustic signal AS and the shape image SG as training data TD, and thereby uses the shape image SG as a label for the acoustic signal AS. Therefore, the information conversion device 1 can omit labeling the acoustic signal AS.

[0087] Each embodiment of the present invention is presented as an example and does not limit the scope of the present invention. Each embodiment can be implemented in various forms without departing from the spirit of the present invention. Each embodiment may be combined with others, and in such cases, combined effects can be obtained. Each embodiment includes multiple components, and various combinations of these multiple components can result in various inventions. Each embodiment or combination of each component is included within the scope of the present invention.

[0088] 1...information conversion device, 2A...sound wave transmitter, 2B...sound wave receiver, 3...display device, 11A...shape image acquisition unit, 11C...acoustic signal acquisition unit, 12...training data creation unit, 13...memory unit, 14...model training unit, 15...model application unit, 100...information conversion system, 111...CPU, 112...RAM, 113...ROM, 114...storage, 115...input device, 116...communication device, 141...parameter update unit, 142...loss calculation unit, AS...acoustic signal, EN...encoder, H...hand region, ML...machine learning model, SD...shape image decoder, SG...shape image, TD...training data, VQ...vector quantization unit

Claims

1. An information conversion device comprising: a model training unit that uses an acoustic signal indicating the acoustic characteristics of a hand in response to sound waves as input data and a shape image indicating the shape of the hand as ground truth data to train a machine learning model.

2. The information conversion device according to claim 1, further comprising a model application unit that obtains the shape image from the trained machine learning model by inputting the acoustic signal to be processed to the trained machine learning model.

3. The information conversion device of claim 1, further comprising a shape image acquisition unit that acquires the shape image, wherein, when the proportion of the image area of ​​the hand to the entire area of ​​the shape image is equal to or greater than a threshold, the shape image acquisition unit reduces the image area until the proportion becomes less than the threshold, and the model training unit uses the shape image having the reduced image area as the correct answer data to train the machine learning model.

4. An information conversion method in which a computer trains a machine learning model using an acoustic signal indicating the acoustic characteristics of a hand in response to sound waves as input data and a shape image indicating the shape of the hand as correct answer data.

Images