Sign language symbol string generation device and program

Abstract

To provide a sign language symbol string generation device capable of receiving input of a natural language sentence corresponding to a sign language video to generate a sign language symbol string with high recognition accuracy by solving a problem that the natural language sentence (such as a Japanese text) cannot be used for recognition of the sign language video in a conventional technique.SOLUTION: A sign language symbol string generation device includes a decoder function unit. The decoder function unit performs decoding processing on the basis of encoding result information corresponding to a natural language sentence and encoding result information corresponding to a sign language image series to output a sign language symbol string.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a sign language symbol string generating device and a program. [Background technology]

[0002] Technology that automatically recognizes the content of sign language speech based on sign language video (sign language recognition technology) is being researched. Sign language recognition technology is of course used to support communication for the hearing impaired, but it is also used to create bilingual data between natural language texts (Japanese texts, etc.) and sign language texts.

[0003] Non-Patent Document 1 reports a technology for creating bilingual data from a sign language program consisting of Japanese speech and sign language video. The technology described in Non-Patent Document 1 creates natural language sentences (Japanese sentences) from natural language speech (Japanese speech) by speech recognition, creates sign language sentences from sign language video by sign language recognition, and creates these as bilingual data. The created bilingual data is used in a technology for machine translating Japanese into sign language sentences. Non-Patent Document 1 uses this machine translation in the process of automatically generating CG animation of sign language from Japanese sentences.

[0004] When sign language recognition is realized using conventional technology, for example, an encoder and decoder using deep learning technology are used as a sign language recognition network. In this conventional technology, a sign language image is input to an image encoder, and a decoder outputs a sign language sentence based on the output from the encoder. A sign language sentence is expressed as a string of symbols (sign language word labels) that represent sign language words. A sign language word label is written by enclosing a Japanese word that is close in meaning to a sign language word in curly brackets {}.

[0005] More specifically, the operation of a sign language recognition network according to the prior art is as follows. That is, an input image (sign language image) is input to an image encoder to obtain an image encoder output. The image encoder output is a sequence of vectors that encode information necessary to understand the meaning conveyed by the input image. Next, a start symbol (BOS symbol; "BOS" stands for "beginning of sentence") is input to the decoder, which determines and outputs the first word of the sign language sentence by referring to the image encoder output. Thereafter, the word determined by the decoder is input recursively to the decoder, which further determines words sequentially by referring to the image encoder output. In this way, the sign language sentence is completed as a string of sign language words output by the decoder. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Takeshi Kajiyama, Hiroyuki Kaneko, Masanori Sano, "Development of a database creation support tool for sign language translation", IEICE Technical Report WIT2021-47(2022-03), pages 24-28, Institute of Electronics, Information and Communication Engineers, 2022 Summary of the Invention [Problem to be solved by the invention]

[0007] A sign language recognition network according to the conventional technology is a network intended to be used to support communication for the hearing impaired, and is intended to accept only sign language video as input. However, when sign language recognition technology is used to create bilingual data between natural language sentences (Japanese sentences, etc.) and sign language sentences, it is possible to use the natural language sentences (Japanese sentences, etc.) to recognize the sign language video. However, the sign language recognition network according to the conventional technology is not configured to use the natural language sentences (Japanese sentences, etc.) to recognize the sign language video. In other words, the conventional technology has a problem in that it is not possible to use the natural language sentences (Japanese sentences, etc.) to recognize the sign language video.

[0008] The present invention has been made based on the recognition of the above problem, and aims to provide a sign language symbol string generation device and program capable of generating a sign language symbol string with high recognition accuracy by inputting a natural language sentence corresponding to a sign language video. [Means for solving the problem]

[0009] [1] In order to solve the above problems, a sign language symbol string generation device according to one embodiment of the present invention includes a decoder functional unit that outputs a string of sign language symbols by performing a decoding process based on encoding result information corresponding to a natural language sentence and encoding result information corresponding to a sign language image sequence.

[0010] [2] Also, one aspect of the present invention is that in the sign language symbol string generation device of [1] above, the decoder function unit is provided with a single decoder that outputs the string of sign language symbols based on both encoding result information corresponding to a natural language sentence and encoding result information corresponding to a sign language image sequence.

[0011] [3] Also, one aspect of the present invention is that in the sign language symbol sequence generation device of [1] above, the decoder function unit includes a first decoder that outputs a first decoding result based on encoding result information corresponding to the natural language sentence, and a second decoder that outputs a string of sign language symbols that is a second decoding result based on encoding result information corresponding to the sign language image sequence and the first decoding result, and the sign language symbol output from the second decoder is input to the first decoder to determine the succeeding sign language symbol in the string.

[0012] [4] Also, one aspect of the present invention is that in the above-mentioned sign language symbol sequence generating device, the decoder function unit includes a first decoder that outputs a first decoding result based on encoding result information corresponding to the sign language image sequence, and a second decoder that outputs a string of sign language symbols that is a second decoding result based on encoding result information corresponding to the natural language sentence and the first decoding result, and the sign language symbol output from the second decoder is input to the first decoder to determine the succeeding sign language symbol in the string.

[0013] [5] Another aspect of the present invention is a program for causing a computer to function as a sign language symbol sequence generation device having a decoder functional unit that outputs a sequence of sign language symbols based on encoding result information corresponding to a natural language sentence and encoding result information corresponding to a sign language image sequence. Effect of the Invention

[0014] According to the present invention, the decoder function unit can perform a decoding process based not only on encoding result information corresponding to a sign language image sequence, but also on encoding result information corresponding to a natural language sentence. Note that this natural language sentence is a sentence corresponding to the sign language image sequence. This enables the sign language symbol string generation device to generate a sign language symbol string with high accuracy as a recognition result of the sign language image sequence. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a sign language recognition device (sign language symbol string generation device) according to a first embodiment of the present invention. [Diagram 2] 3 is a flowchart showing a processing procedure of the sign language recognition device (sign language symbol string generation device) according to the first embodiment. [Diagram 3] FIG. 11 is a block diagram showing a schematic functional configuration of a sign language recognition device (sign language symbol string generation device) according to a second embodiment. [Figure 4] 10 is a flowchart showing a processing procedure of a sign language recognition device (sign language symbol string generation device) according to a second embodiment. [Diagram 5] 1 is a block diagram showing an example of the internal configuration of a sign language recognition device (sign language symbol string generation device) according to a first or second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Next, a number of embodiments of the present invention will be described with reference to the drawings.

[0017] [First embodiment] FIG. 1 is a block diagram showing a schematic functional configuration of a sign language recognition device according to a first embodiment. As shown in the figure, the sign language recognition device 1 includes a sign language image encoder 10, a decoder 20, a natural language sentence encoder 30, and a concatenation processing unit 40. The sign language recognition device 1 is characterized in that it includes the natural language sentence encoder 30. Information output from the natural language sentence encoder 30 is passed to the decoder 20 by a method described later. Each of these functional units can be realized, for example, by a computer and a program. Each functional unit has a storage means as necessary. The storage means is, for example, a variable in the program or a memory allocated by the execution of the program. Non-volatile storage means such as a magnetic hard disk drive or a solid state drive (SSD) may be used as necessary. At least a part of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program. The functions of each unit constituting the sign language recognition device 1 are as follows.

[0018] The sign language image encoder 10 inputs a sign language image and outputs latent variables. The sign language image may be a person expressing themselves using sign language (for example, computer graphics representing a character resembling a person). Specifically, the sign language image encoder 10 inputs the sign language image as a sequence of frame images and outputs latent variables [v1 v2 v3 . . . v T ] is output. Note that v1, v2, v3, . . . , v TEach of is a vector. The output from the sign language image encoder 10 may be called a "sign language image encoder output." The sign language image encoder 10 is realized using an RNN (recurrent neural network) such as a bidirectional LSTM (Long Short-Term Memory).

[0019] The natural language sentence encoder 30 inputs a natural language sentence (for example, a Japanese sentence) and outputs latent variables. Specifically, the natural language sentence encoder 30 inputs a symbol string corresponding to the natural language sentence. When the input natural language sentence is a Japanese sentence, the natural language sentence encoder 30 inputs an input word string (morpheme string) obtained by morphologically analyzing the Japanese sentence, and outputs latent variables [u1u2u3...u w ] is output. Note that u1, u2, u3, , u w Each of is a vector. Also, w is an integer equal to or greater than 1. This output may be called the "natural language sentence encoder output." This natural language sentence encoder output is a sequence of vectors that encodes information necessary for semantic understanding of the natural language sentence. The natural language sentence encoder 30 is realized, for example, by using a Transformer encoder.

[0020] When the natural language sentence is in a language other than Japanese, the natural language sentence encoder 30 inputs an input word string (input symbol string) in a format appropriate for that language, and outputs a natural language sentence encoder output.

[0021] The concatenation processing unit 40 concatenates the natural language sentence encoder output [u1 u2 u3 . . . u w ] and the sign language image encoder output [v1v2v3··· v T In the illustrated example, the concatenation processing unit 40 concatenates the two outputs, resulting in [u1u2u3...u w v1v2v3... v T]. Here, T is an integer equal to or greater than 1. This output from concatenation processing unit 40 may be called a "concatenated encoder output."

[0022] The decoder 20 is also called a "decoder function unit." The decoder 20 performs a decoding process while referring to the concatenated encoder output output from the concatenation processing unit 40. As a result, the decoder 20 sequentially determines sign language words. That is, the decoder 20 outputs a string of sign language words. As in the conventional technology described above, a start symbol (BOS symbol; "BOS" is an abbreviation for "beginning of sentence") is first input to the decoder 20. The sign language words output by the decoder are recursively input to the decoder 20. Referring to FIG. 1, in response to the input of the start symbol, the decoder 20 outputs a sign language word s1. Next, the sign language word s1 is input, and in response to the input of s1, the decoder 20 outputs a sign language word s2. Thereafter, the decoder 20 outputs the sign language words s3, s4, ... in order in the same manner. The decoder 20 is realized, for example, by using a Transformer decoder.

[0023] The operation of the decoder 20 itself is the same as that of the decoder described in the prior art. However, a feature of this embodiment is that the decoder 20 performs processing while referring to the concatenated encoder output. In other words, the decoder 20 does not perform decoding processing based only on the image encoder output information, but performs decoding processing based on both the image encoder output and the natural language sentence encoder output information. In other words, the configuration shown in Fig. 1 enables the decoder 20 to determine the sign language words to be output while simultaneously referring to the sign language video information and the natural language sentence (Japanese sentence, etc.) information.

[0024] That is, the decoder 20 (decoder function unit) outputs a string of sign language symbols by performing a decoding process based on the encoding result information corresponding to the natural language sentence and the encoding result information corresponding to the sign language image sequence. The decoder 20 of this embodiment is a single decoder that outputs the string of sign language symbols based on both the encoding result information corresponding to the natural language sentence and the encoding result information corresponding to the sign language image sequence.

[0025] Next, the operation procedure of the sign language recognition device 1 will be described. The sign language recognition device 1 can operate in a learning mode or an estimation mode. When operating in the learning mode, the sign language recognition device 1 uses training data to train an internal model. That is, the sign language recognition device 1 can adjust (optimize) the values ​​of internal parameters of the neural network. When operating in the estimation mode, the sign language recognition device 1 estimates a sign language word string based on the trained neural network, and outputs it as a recognition result.

[0026] 2 is a flowchart showing the operation procedure of the sign language recognition device 1. This flowchart includes both the processing procedure in the learning mode and the processing procedure in the estimation mode. The processing procedure will be described below with reference to this flowchart.

[0027] In step S101, the sign language image encoder 10 acquires an input image (sign language image) and performs forward propagation processing of a neural network, resulting in a sign language image encoder output.

[0028] Next, in step S102, the natural language sentence encoder 30 obtains an input word string (input symbol string) and performs forward propagation processing of the neural network, resulting in a natural language sentence encoder output.

[0029] The process of step S101 and the process of step S102 may be executed in the reverse order, or at least a part of the process of step S101 and the process of step S102 may be executed simultaneously in parallel.

[0030] Next, in step S103, the concatenation processing unit 40 performs processing to concatenate the sign language image encoder output and the natural language sentence encoder output, and outputs a concatenated encoder output. The concatenation processing unit 40 passes the obtained concatenated encoder output to the decoder 20.

[0031] Next, in step S104, the decoder 20 performs forward decoder propagation. That is, the decoder 20 refers to the entire concatenated encoder output passed from the concatenation processing unit 40, inputs them in order starting from the start symbol (BOS symbol), and determines the output one by one. The symbol string obtained by the decoder 20 making sequential determinations is a string of sign language words (sign language symbols). These sign language words are words that make up a sign language sentence. That is, the decoder 20 outputs a sign language sentence, and this sign language sentence is the sign language recognition result (estimated result).

[0032] In step S105, the sign language recognition device 1 determines whether the output from the decoder 20 is an end symbol. If an end symbol is output (step S105: YES), the process proceeds to the next step S106. If the output symbol is not an end symbol (step S105: NO), the process returns to step S104 to output the next symbol.

[0033] That is, the sign language recognition device 1 outputs sign language words one by one by repeating the forward propagation process in the decoder 20 in step S104 until the end symbol is output.

[0034] In step S106, the sign language recognition device 1 determines whether to perform learning. In other words, the sign language recognition device 1 determines whether to adjust the internal parameters depending on whether it is operating in a learning mode or an estimation mode. If learning is to be performed (step S106: YES), the process proceeds to the next step S107. If learning is not to be performed (step S106: NO), the sign language recognition device 1 outputs the obtained sign language sentence (sign language word string) and ends the entire process of this flowchart.

[0035] When the process proceeds to step S107, the sign language recognition device 1 adjusts the values ​​of parameters held inside. That is, the sign language recognition device 1 calculates the loss (error) between the sign language word string (estimated result by the current model) output by the decoder corresponding to the input data (pair of natural language sentence and sign language video) and the correct sign language word string corresponding to the input data. The loss calculated here is, for example, cross-entropy loss. Then, the sign language recognition device 1 uses an error backpropagation method to change (adjust, optimize) the values ​​of the internal parameters held by each model in the direction that reduces the loss. The parameter optimization by error backpropagation itself can be performed using existing technology.

[0036] Next, in step S108, the sign language recognition device 1 determines whether or not to end the learning. This determination may be made, for example, based on whether or not the value of the set of internal parameters has sufficiently converged to a predetermined value. Alternatively, this determination may be made based on whether or not learning has already been performed using a predetermined amount of learning data. Regardless of which method is used to determine whether or not the learning is to be ended, if the learning is to be ended (step S108: YES), the entire processing of this flowchart is ended. If the learning is not to be ended (step S108: NO), the process returns to step S101 and performs processing using the next learning data.

[0037] As described above, according to this embodiment, it is possible to generate high-quality sign language sentences corresponding to sign language videos based on natural language sentences (for example, Japanese sentences) and sign language videos. The sign language recognition device 1 of this embodiment inputs and processes not only sign language videos but also natural language sentences, thereby improving the quality of the sign language sentences corresponding to the sign language videos.

[0038] According to this embodiment, the process performed by the sign language recognition device 1 can secondarily provide translation data between a natural language sentence (for example, a Japanese sentence) and a sign language sentence.

[0039] [Evaluation Experiment for the First Embodiment] An evaluation experiment was conducted on the sign language recognition device 1 of the first embodiment. A sign language recognition device of the type described in the prior art was used as a comparison target. The sign language recognition device was trained using a data set consisting of 30,000 sign language videos, sign language sentences, and Japanese sentences. For evaluation, 6,000 sign language videos, Japanese sentences, and sign language sentences were used, and an evaluation was conducted on the word error rate in the estimated sign language sentences. The word error rate was 34% for the sign language recognition device of the prior art that does not use Japanese sentences for estimation. In contrast, the word error rate was 31% for the sign language recognition device of the first embodiment that uses Japanese for estimation, and an improvement of 3 points was confirmed.

[0040] [Second embodiment] Next, a second embodiment of the present invention will be described. Note that the matters already described in the previous embodiment may not be described below. Here, the matters unique to this embodiment will be mainly described.

[0041] 3 is a block diagram showing the functional configuration of a sign language recognition device according to the second embodiment. As shown in the figure, the sign language recognition device 2 includes a sign language image encoder 210, a second decoder 221, a first decoder 222, and a natural language sentence encoder 230. Similarly to the sign language recognition device 1 of the previous embodiment, the sign language recognition device 2 can also be realized by, for example, a computer and a program. Furthermore, at least some of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.

[0042] A feature of the network in the sign language recognition device 2 shown in Fig. 3 is that the first decoder 222 refers to the output from the natural language sentence encoder 230. Also, the second decoder 221 refers to the output from the sign language image encoder 210. The first decoder 222 and the second decoder 221 are connected. In other words, the first decoder 222 is connected to the second decoder 221 so that the first decoder 222 is connected to the second decoder 221, which is connected to the second decoder 221, and the second decoder 221 is connected to the first decoder 222. The functions of each part constituting the sign language recognition device 2 are as follows.

[0043] The natural language sentence encoder 230, like the natural language sentence encoder 30 in the first embodiment, receives a natural language sentence (for example, a Japanese sentence) as an input and outputs the natural language sentence encoder output [u1 u2 u3 . . . u w Specifically, the natural language sentence encoder 230 receives an input word string (morpheme string) corresponding to the natural language sentence. The natural language sentence encoder 230 outputs the natural language sentence encoder output [u1 u2 u3 . . . u w ] to the first decoder 222. Note that the natural language sentence encoder 230 may be realized using, for example, a Transformer encoder.

[0044] The sign language image encoder 210 receives a sign language video (a sequence of frame images) and outputs [v1 v2 v3 . . . v T The sign language image encoder 210 outputs the sign language image encoder output [v1 v2 v3 . . . v T ] to the second decoder 221. Note that the sign language image encoder 210 can be realized using, for example, a bidirectional LSTM.

[0045] The first decoder 222 receives the natural language sentence encoder output [u1 u2 u3 . . . u w ], the first decoder 222 sequentially determines the symbols to be output. Note that the first decoder 222 can be realized by using, for example, a Transformer decoder.

[0046] The second decoder 221 receives the sign language image encoder output [v1 v2 v3 . . . v T ], the second decoder 221 sequentially determines the symbols to be output. Note that the second decoder 221 can be realized by using, for example, a Transformer decoder.

[0047] The relationship between the first decoder 222 and the second decoder 221 is as follows: The first decoder 222 first receives a start symbol (BOS symbol), outputs information estimated with reference to the output of the natural language sentence encoder, and passes it to the second decoder 221. The second decoder 221 receives the information output by the first decoder 222 as an estimation result, and determines a word with reference to the output of the sign language image encoder. The word determined by the second decoder 221 is output as a sign language word string constituting the sign language recognition result, and also recursively becomes an input to the first decoder 222.

[0048] That is, the first decoder 222 and the second decoder 221 work together to determine the sign language word s1 based on the start symbol (BOS symbol). The sign language word s1 becomes the input to the first decoder 222. The first decoder 222 and the second decoder 221 then work together to determine the sign language word s2. Thereafter, the first decoder 222 and the second decoder 221 similarly output the sign language words s3, s4, s5, ... in sequence. The string of sign language words output in this manner is a sign language sentence that is the recognition result of the input sign language.

[0049] The functions of the first decoder 222 and the second decoder 221 may be collectively referred to as a "decoder function unit." In other words, the decoder function unit outputs a string of sign language symbols by performing a decoding process based on encoding result information corresponding to a natural language sentence and encoding result information corresponding to a sign language image sequence.

[0050] That is, the first decoder 222 outputs a first decoded result based on encoding result information corresponding to a natural language sentence. The second decoder 221 outputs a string of sign language symbols, which is a second decoded result, based on encoding result information corresponding to a sign language image sequence and the first decoded result. The sign language symbols output from the second decoder 221 are input to the first decoder 222 to obtain the next sign language symbol in the string of sign language symbols.

[0051] 4 is a flowchart showing the operation procedure of the sign language recognition device 2. This flowchart includes both the processing procedure in the learning mode and the processing procedure in the estimation mode. The processing procedure will be described below with reference to this flowchart.

[0052] In step S201, the natural language sentence encoder 230 obtains an input word string (input symbol string) and performs a forward propagation process of a neural network. As a result, a natural language sentence encoder output is obtained. The natural language sentence encoder 230 passes the natural language sentence encoder output to the first decoder 222.

[0053] Next, in step S202, the sign language image encoder 210 obtains an input image (sign language image) and performs forward propagation processing of the neural network. As a result, a sign language image encoder output is obtained. The sign language image encoder 210 passes the sign language image encoder output to the second decoder 221.

[0054] The process of step S201 and the process of step S202 may be executed in the reverse order, or at least a part of the process of step S201 and the process of step S202 may be executed simultaneously in parallel.

[0055] Next, in step S203, the first decoder 222 performs forward propagation processing while referring to the natural language sentence encoder output passed from the natural language sentence encoder 230. Note that the initial value of the symbol input to the first decoder 222 is the start symbol (BOS symbol). Thereafter, the output from the second decoder 221 is input to the first decoder 222. The first decoder 222 passes the information output as a result of the processing of this step to the second decoder 221.

[0056] Next, in step S204, the second decoder 221 performs forward propagation processing while referring to the sign language image encoder output passed from the sign language image encoder 210. At this time, the input to the second decoder 221 is the information output from the first decoder 222 in step S203 above. The symbol strings sequentially output by the second decoder 221 are strings of sign language words (sign language symbols). These sign language words are words that make up a sign language sentence. In other words, the second decoder 221 outputs a sign language sentence, and this sign language sentence is the sign language recognition result (estimated result).

[0057] In step S205, the sign language recognition device 1 determines whether the output from the second decoder 221 is an end symbol. If an end symbol is output (step S205: YES), the process proceeds to the next step S206. If the output symbol is not an end symbol (step S205: NO), the process returns to step S203 to output the next symbol.

[0058] In other words, the sign language recognition device 2 outputs sign language words one by one by repeating the forward propagation process in the first decoder 222 in step S203 and the forward propagation process in the second decoder 221 in step S204 until an end symbol is detected.

[0059] In step S206, the sign language recognition device 2 determines whether or not to perform learning, as in the first embodiment. If learning is to be performed (step S206: YES), the process proceeds to the next step S207. If learning is not to be performed (step S206: NO), the sign language recognition device 1 outputs the obtained sign language sentence (sign language word string) and ends the entire process of this flowchart.

[0060] When the process proceeds to step S207, the sign language recognition device 2 adjusts the values ​​of its internal parameters. That is, the sign language recognition device 2 calculates the loss (error) between the sign language word string (the estimation result by the current model) output by the decoder corresponding to the input data (a pair of a natural language sentence and a sign language video) and the correct sign language word string corresponding to the input data. Then, the sign language recognition device 2 uses the error backpropagation method to change (adjust, optimize) the values ​​of the internal parameters of each model in the direction that reduces the loss.

[0061] Next, in step S208, the sign language recognition device 2 determines whether or not to end the learning. If the learning is to be ended (step S208: YES), the entire process of this flowchart is ended. If the learning is not to be ended (step S208: NO), the process returns to step S201 and performs the process using the next learning data.

[0062] In this way, in the second embodiment, the natural language sentence encoder output and the sign language image encoder output are not concatenated to form a single vector sequence, but similarly to the first embodiment, the decoder performs decoding processing based on information from both the natural language sentence encoder output and the sign language image encoder output. In other words, the sign language sentence is estimated by performing sign language recognition processing based not only on the sign language image but also on the natural language sentence. In other words, the second embodiment can generate high-quality sign language sentences corresponding to sign language videos. Also, in the second embodiment, it is possible to secondarily obtain bilingual data between natural language sentences (e.g., Japanese sentences) and sign language sentences.

[0063] [Modification of the second embodiment] Next, a modified example of the second embodiment will be described. In the sign language recognition device 2 of the second embodiment, as shown in FIG. 3, the natural language sentence encoder output output by the natural language sentence encoder 230 is passed to the first decoder 222 (a decoder on the front stage), and the sign language image encoder output output by the sign language image encoder 210 is passed to the second decoder 221 (a decoder on the back stage) for decoding. As a modified example, this may be reversed. That is, as a modified example of the second embodiment, the sign language image encoder output output by the sign language image encoder 210 is passed to the first decoder 222 (a decoder on the front stage), and the natural language sentence encoder output output by the natural language sentence encoder 230 is passed to the second decoder 221 (a decoder on the back stage) for decoding.

[0064] That is, in this modified example, the first decoder 222 outputs a first decoded result based on encoding result information corresponding to a sign language image sequence. The second decoder 221 outputs a string of sign language symbols, which is a second decoded result, based on encoding result information corresponding to a natural language sentence and the first decoded result. The sign language symbols output from the second decoder 221 are input to the first decoder 222 to obtain the next sign language symbol in the string of sign language symbols.

[0065] [Example of the internal configuration of the device] FIG. 5 is a block diagram showing an example of the internal configuration of a sign language recognition device in the first embodiment and the second embodiment (including modified examples). The sign language recognition devices 1 and 2 (including modified examples) can be realized using a computer and a program. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, etc., and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902 or the like. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with an external input / output device or the like. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from and to the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port 903 via the bus 906.

[0066] At least some of the functions of the sign language recognition device in the above-mentioned embodiment can be realized by a computer and a program. In that case, the program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, a DVD-ROM, a USB memory, and a storage device such as a hard disk built into a computer system. In other words, the term "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may include a medium that temporarily and dynamically holds a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. Furthermore, the above-mentioned program may be a program for realizing some of the above-mentioned functions, and may further be a program that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0067] Although a number of embodiments have been described above, the present invention can also be embodied in the following modified examples.

[0068] [Variations] In the above-described embodiment, a Japanese sentence is processed as a natural language sentence, but a sentence in any natural language may be processed, not limited to Japanese. Natural languages ​​include, but are not limited to, English, German, French, Russian, Spanish, Italian, Chinese, Korean, etc. Since sign languages ​​have different vocabulary and grammar depending on the country or region, a sign language sentence corresponding to a sign language video with different vocabulary and grammar may be processed as a sentence in a natural language.

[0069] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Industrial Applicability]

[0070] The present invention can be used, for example, for sign language recognition or for generating bilingual data pairs. The generated bilingual data pairs can be used for machine learning. The machine learning model may be, for example, a model for converting between sign language video and sign language sentences, or a model for converting between natural language sentences and sign language sentences. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0071] 1,2 Sign language recognition device (sign language symbol string generator) 10 Sign Language Image Encoder 20 Decoder (decoder function part) 30 Natural Language Sentence Encoder 40 Connection processing section 210 Sign Language Image Encoder 221 Second decoder (decoder function part) 222 First decoder (decoder function part) 230 Natural Language Sentence Encoder 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. a decoder functional unit that performs a decoding process based on encoding result information corresponding to the natural language sentence and encoding result information corresponding to the sign language image sequence, thereby outputting a string of sign language symbols; A sign language symbol string generating device comprising:

2. the decoder function unit includes a single decoder that outputs the sequence of sign language symbols based on both encoding result information corresponding to a natural language sentence and encoding result information corresponding to a sign language image sequence; The sign language symbol string generating device according to claim 1 .

3. The decoder function unit includes: a first decoder for outputting a first decoding result based on encoding result information corresponding to the natural language sentence; a second decoder that outputs a string of sign language symbols as a second decoding result based on encoding result information corresponding to the sign language image sequence and the first decoding result; Equipped with the sign language symbol output from the second decoder is input to the first decoder to determine the next sign language symbol in the sequence; The sign language symbol string generating device according to claim 1 .

4. The decoder function unit includes: a first decoder for outputting a first decoded result based on encoding result information corresponding to the sign language image sequence; a second decoder that outputs a string of sign language symbols as a second decoding result based on encoding result information corresponding to the natural language sentence and the first decoding result; Equipped with the sign language symbol output from the second decoder is input to the first decoder to determine the next sign language symbol in the sequence; The sign language symbol string generating device according to claim 1 .

5. a decoder functional unit that outputs a string of sign language symbols based on encoding result information corresponding to the natural language sentence and encoding result information corresponding to the sign language image sequence; A program for causing a computer to function as a sign language symbol string generation device comprising the above.

Images