Learning device, sign language recognition device and program
The learning device enhances sign language recognition by using a CNN and decoder to process simultaneous hand expressions, updating parameters to improve accuracy in recognizing both right-hand and left-hand signs.
Patent Information
- Application Number
- JP2024073530
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-04-30
- Publication Date
- 2025-11-12
AI Technical Summary
Conventional sign language recognition techniques struggle to accurately recognize simultaneous expressions using both hands, as they fail to associate two words with an image at each time and lack effective methods for improving recognition performance for such representations.
A learning device and recognition system that employs a convolutional neural network (CNN) to extract features, an encoder to generate word probability distributions for both hands, and a decoder to estimate a two-hand sign language sentence, with error calculation units updating internal parameters to enhance recognition accuracy.
The system significantly improves the recognition performance of simultaneous hand expressions in sign language by updating internal parameters based on error calculations, enabling accurate decoding of right-hand and left-hand signs.
Smart Images
Figure 2025168782000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a learning device, a sign language recognition device, and a program. [Background technology]
[0002] Technology for automatically recognizing sign language is being researched as a means of supporting communication for the hearing impaired, and its practical application is anticipated. In automatic sign language recognition, the content of spoken sign language is treated as a time series of sign language words (sign sentence). Sign language words include words that are expressed using both hands and words that are expressed using only one hand. In addition, two words can sometimes be expressed simultaneously using each hand. Such simultaneous expression using both hands will be referred to below as "simultaneous expression."
[0003] Symbols representing sign language words are written by enclosing a Japanese word that is close in meaning to the sign language word in curly brackets {}.
[0004] In the field of research into automatic recognition of sign language sentences based on video of sign language, the use of deep learning has become mainstream. Non-Patent Document 1 describes an example of a sign language recognition network that uses deep learning. The sign language recognition network shown in Non-Patent Document 1 performs processing in three stages: image analysis, time series analysis, and matching images with words (classifier).
[0005] Another example of a sign language recognition network is one that is being studied, which estimates sign language sentences by taking a bird's-eye view of information across all time periods. In this method, the time series of the results of image analysis at each time period (A: image analysis) is analyzed (B: time series analysis), and this information from all time periods is passed to a network for estimating sign language sentences to estimate the sign language sentences. This method makes it possible to estimate two sign language words contained in simultaneous expressions as words that co-occur in a chain. For convenience, a network using this method is called an "all-time information utilization network."
[0006] Furthermore, by promoting the learning of A: image analysis and B: time series analysis in conjunction with the all-time information utilization network method, it is expected that the performance of sign language sentence estimation will be indirectly improved. A network using this method will be referred to as a "combinatorial network" for convenience.
[0007] Furthermore, Non-Patent Document 2 describes a multimodal sign language recognition technology that uses both sign language video and Japanese translation. [Prior art documents] [Non-patent literature]
[0008] [Non-Patent Document 1] Takeshi Kajiyama and Yoshiaki Shikakui, Evaluation of Sigh Language Recognition with Higher Resolution Face Images, Journal of the Institute of Image Information and Television Engineers, Vol. 77, No. 6, pp. 788-794, 2023. [Non-patent document 2] Takeshi Kajiyama, Taro Miyazaki, Hiroyuki Kaneko, Multimodal Sign Language Video Recognition with Japanese Bilingual Text, 11B-8, 2023 ITE Winter Conference. Summary of the Invention [Problem to be solved by the invention]
[0009] However, in order to accurately recognize sign language that includes simultaneous expressions using conventional techniques, the following problems arise.
[0010] The method described in Non-Patent Document 1 has a problem in that it cannot recognize simultaneous expressions because it cannot associate two words with an image at each time.
[0011] Furthermore, the supplementary learning in the above-mentioned "combinatorial network" method can only handle representations of either the right or left hand, which is a problem in that it is not possible to expect improvement in recognition performance for simultaneous representations.
[0012] Furthermore, Non-Patent Document 2 does not particularly describe any technology for improving the recognition accuracy of simultaneous expressions.
[0013] The present invention has been made based on the above-mentioned problem recognition, and aims to provide a learning device, a sign language recognition device, and a program that can improve the automatic recognition performance of simultaneous expressions in sign language. [Means for solving the problem]
[0014] [1] In order to solve the above problem, a learning device according to one aspect of the present invention includes a convolution processing unit that extracts features by convolution processing for each image included in an input image sequence; an encoder that encodes information output from the convolution processing unit to generate word probability distribution information that represents a probability distribution for both right-hand words and left-hand words represented by the image sequence; a decoder that decodes the word probability distribution information output from the encoder to estimate and output a two-hand sign language sentence as a string including both right-hand words and left-hand words; a first algorithm processing unit that obtains a right-hand sign language sentence as a string of right-hand words based on the word probability distribution information output from the encoder; a second algorithm processing unit that obtains a left-hand sign language sentence as a string of left-hand words based on the word probability distribution information output from the encoder; a two-hand error calculation unit that calculates an error between the two-hand sign language sentence output from the decoder and a two-hand sign language sentence of teacher data corresponding to the input image sequence; and a right-hand sign language sentence obtained by the first algorithm processing unit and a left-hand sign language sentence of teacher data corresponding to the input image sequence. and a left-hand error calculation unit that calculates the error between the left-hand sign language sentence determined by the second algorithm processing unit and a left-hand sign language sentence of teacher data corresponding to the input image sequence, wherein the convolution processing unit has an updatable first internal parameter and performs a process of convolving an image based on the first internal parameter, the encoder has an updatable second internal parameter and generates the word probability distribution information based on the second internal parameter, and the decoder has an updatable third internal parameter and estimates the two-hand sign language sentence based on the third internal parameter, and is configured to be able to update the third internal parameter, the second internal parameter, and the first internal parameter based on the error calculated by the two-hand error calculation unit, and is configured to be able to update the second internal parameter and the first internal parameter based on the error calculated by the right-hand error calculation unit and the error calculated by the left-hand error calculation unit.
[0015] [2] Furthermore, one aspect of the present invention is that in the learning device of [1] above, for the image sequence contained in one given learning data, the second internal parameter and the first internal parameter are updated based on the error calculated by the right-hand error calculation unit, and then the second internal parameter and the first internal parameter are updated based on the error calculated by the left-hand error calculation unit, or the second internal parameter and the first internal parameter are updated based on the error calculated by the left-hand error calculation unit, and then the second internal parameter and the first internal parameter are updated based on the error calculated by the right-hand error calculation unit.
[0016] [3] Furthermore, one aspect of the present invention is that in the learning device of [1] above, for the image sequence contained in one given learning data, the second internal parameter and the first internal parameter are updated based on a total error calculated based on the error calculated by the right-hand error calculation unit and the error calculated by the left-hand error calculation unit.
[0017] [4] Another aspect of the present invention is a sign language recognition device comprising: a convolution processing unit that extracts features by convolution processing for each image included in an input image sequence; an encoder that encodes information output from the convolution processing unit to generate word probability distribution information that represents the probability distribution of both right-hand words and left-hand words represented by the image sequence; and a decoder that decodes the word probability distribution information output from the encoder to estimate and output a two-handed sign language sentence as a string including both right-hand words and left-hand words, wherein the convolution processing unit convolves the image based on a first internal parameter obtained as a result of learning by the learning device described in any of [1] to [3] above; the encoder generates the word probability distribution information based on a second parameter obtained as a result of learning by the learning device described in any of [1] to [3] above; and the decoder estimates the two-handed sign language sentence based on a third parameter obtained as a result of learning by the learning device described in any of [1] to [3] above.
[0018] [5] Another aspect of the present invention is a system including: a convolution processing unit that extracts features by convolution processing for each image included in an input image sequence; an encoder that encodes information output from the convolution processing unit to generate word probability distribution information that represents a probability distribution for both right-hand words and left-hand words represented by the image sequence; a decoder that decodes the word probability distribution information output from the encoder to estimate and output a two-hand sign language sentence as a sequence including both right-hand words and left-hand words; a first algorithm processing unit that obtains a right-hand sign language sentence as a sequence of right-hand words based on the word probability distribution information output from the encoder; a second algorithm processing unit that obtains a left-hand sign language sentence as a sequence of left-hand words based on the word probability distribution information output from the encoder; a two-hand error calculation unit that calculates an error between the two-hand sign language sentence output from the decoder and a two-hand sign language sentence of training data corresponding to the input image sequence; and a two-hand error calculation unit that calculates an error between the right-hand sign language sentence obtained by the first algorithm processing unit and a right-hand sign language sentence of training data corresponding to the input image sequence. a right-hand error calculation unit that calculates an error between the left-hand sign language sentence obtained by the second algorithm processing unit and a left-hand sign language sentence of training data corresponding to the input image sequence, wherein the convolution processing unit has an updatable first internal parameter and performs a process of convolving an image based on the first internal parameter, the encoder has an updatable second internal parameter and generates the word probability distribution information based on the second internal parameter, and the decoder has an updatable third internal parameter and estimates the two-hand sign language sentence based on the third internal parameter, wherein the third internal parameter, the second internal parameter, and the first internal parameter can be updated based on the error calculated by the two-hand error calculation unit, and the second internal parameter and the first internal parameter can be updated based on the error calculated by the right-hand error calculation unit and the error calculated by the left-hand error calculation unit.It is a program that makes a computer function as a [Effects of the Invention]
[0019] According to the present invention, the learning device not only updates the third internal parameter, the second internal parameter, and the first internal parameter based on the errors calculated by the two-hand error calculation unit, but also updates the second internal parameter and the first internal parameter based on the errors calculated by the right-hand error calculation unit and the left-hand error calculation unit, thereby improving the recognition performance of the sign language recognition device. [Brief explanation of the drawings]
[0020] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a sign language recognition device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing the input / output relationship between a CNN and an encoder in the sign language recognition device of the embodiment. [Figure 3] FIG. 2 is a block diagram for explaining the processing of a decoder and an error calculation unit (28) in the sign language recognition device of the embodiment. [Figure 4] 2 is a block diagram for explaining the processing of a CTC algorithm processing unit (12-1, 12-2) and an error calculation unit (18-1, 18-2) in the sign language recognition device of the embodiment. FIG. [Figure 5] 10 is a flowchart showing an example of a processing procedure for the sign language recognition device according to the embodiment to perform machine learning and optimize internal parameters. [Figure 6] FIG. 2 is a block diagram showing an example of an internal configuration for realizing the sign language recognition device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION
[0021] Next, an embodiment of the present invention will be described with reference to the drawings. The basic configuration of this embodiment is a sign language recognition device having a network that learns two words in parallel in the "auxiliary learning" of the "combinatorial network" described in the background art. This configuration of this embodiment aims to improve the recognition performance of simultaneous expressions in sign language.
[0022] FIG. 1 is a block diagram showing a schematic functional configuration of a sign language recognition device according to this embodiment. As shown in the figure, the sign language recognition device 1 includes a CNN 10, an encoder 11, CTC algorithm processing units 12-1 and 12-2, error calculation units 18-1 and 18-2, a decoder 21, an error calculation unit 28, and a control unit 90. Each of these functional units can be implemented, for example, by a computer and a program. Each functional unit also includes a storage unit as needed. The storage unit is, for example, a memory allocated by program variables or program execution. Nonvolatile storage units such as a magnetic hard disk drive or solid-state drive (SSD) may also be used as needed. At least some of the functions of each functional unit may be implemented as a dedicated electronic circuit rather than a program.
[0023] Among the functions of the sign language recognition device 1, the CNN 10, the encoder 11, and the decoder 21 are each configured to be capable of machine learning. More specifically, the CNN 10, the encoder 11, and the decoder 21 can each be configured using neural network technology. That is, the CNN 10, the encoder 11, and the decoder 21 each have internal parameters. The CNN 10, the encoder 11, and the decoder 21 each perform calculations based on the input data and the internal parameter values they currently have, and calculate the data to be output. The internal parameter values of the CNN 10, the encoder 11, and the decoder 21 can be adjusted (optimized) through machine learning using training data. Note that the error backpropagation method is used to update the internal parameter values. The error (loss) used here refers to the error between the training data included in the training data and the output result corresponding to that training data.
[0024] The control unit 90 controls the operation of the sign language recognition device 1. The sign language recognition device 1 operates in either a learning mode for learning internal parameters, or an inference mode for making inferences about unknown input video based on the learned internal parameters. When the sign language recognition device 1 operates in either of these modes, the control unit 90 controls each unit within the sign language recognition device 1 to perform processing according to the operating mode at that time. Note that the sign language recognition device 1 operating in the learning mode may also be called a "learning device."
[0025] The functions of the other units included in the sign language recognition device 1 will be described next with reference to other drawings.
[0026] FIG. 2 is a block diagram showing the input / output relationship between the CNN 10 and the encoder 11 in the sign language recognition device 1. As shown in the figure, images at each time (time 1, 2, . . . , t, . . . , T) are input to the CNN 10. A series of these images constitutes the input video 101. The encoder 11 generates the encoded information v1, v2, . . . , vt ,···,v T This v1, v2, , v t ,···,v T is the word probability distribution information 102.
[0027] CNN·10 inputs an image and analyzes it. CNN stands for "convolutional neural network." As shown in Figure 1, CNN·10 individually analyzes the image at time 1, the image at time 2, the image at time t, the image at time T, and the image at time T, and passes the results to encoder 11. Note that 1≦t≦T. This sequence of T images represents a video. For example, the video may be a video of a person speaking using sign language. Note that, in implementing sign language recognition device 1, T CNN·10s may simultaneously analyze images corresponding to each time, or one or more (fewer than T) CNN·10s may sequentially analyze images at multiple times. In other words, CNN·10 extracts features from each image in the input image sequence (video) through convolution processing. Note that convolutional neural networks themselves are an existing technology. Note that CNN·10 may also be referred to as a "convolution processing unit."
[0028] CNN·10 has an updatable first internal parameter and performs image convolution processing based on the value of that first internal parameter.
[0029] The encoder 11 performs a process of encoding information based on the analysis results of each piece of data passed from the CNN·10. In other words, the encoder 11 performs a time series analysis based on the series of data passed from the CNN·10. Specifically, the encoder 11 calculates v1 based on the information passed from the CNN·10 based on the image at time 1. The encoder 11 also calculates v2 based on the information passed from the CNN·10 based on the image at time 2 and the information of the image analyzed at a past time. The same applies below, and the encoder 11 calculates v2 based on the information passed from the CNN·10 based on the image at time t (1≦t≦T). t In addition, the encoder 11 calculates v based on the information passed from the CNN 10 based on the image at time T. T As mentioned above, v1, v2, , v t ,···,v T is the word probability distribution information 102. In other words, the word probability distribution information 102 is a word probability distribution (a distribution of the probability that the input image corresponds to which sign language word) at each time point obtained by analyzing the image at each time point. t ,···,v T are vectors. In other words, the word probability distribution information 102 is a sequence of vectors corresponding to each time. The word probability distribution information 102 is expressed as [v1, v2, . . . , v t ,···,v T That is, the encoder 11 encodes the information output from the CNN 10 to generate word probability distribution information that represents the probability distribution of both the right-hand and left-hand words represented by the image sequence (input video).
[0030] The encoder 11 has an updatable second internal parameter, and generates word probability distribution information based on the value of the second internal parameter.
[0031] As shown in FIG. 1, the word probability distribution information 102 (v1, v2, . . . , v t ,···,vT ) is passed to the CTC algorithm processing unit 12-1, the CTC algorithm processing unit 12-2, and the decoder 21.
[0032] 3 is a block diagram for explaining the processing of the decoder 21 and the error calculation unit 28. The above-mentioned word probability distribution information 102 is passed to the decoder 21. The word probability distribution information 102 is information that represents the analysis results of images over all time points (from time 1 to time T).
[0033] The decoder 21 decodes the word probability distribution information 102 passed from the encoder 11 and infers the sign language recognition result. The sign language recognition result is expressed as a sign language sentence. A sign language sentence is a sequence of sign language words. First, a start symbol 201 is input to the decoder 21. The start symbol is represented as {BOS}. BOS stands for "Beginning of Sentence." In response to the input of this start symbol 201, the decoder 21 outputs the first word O1 of the sign language sentence (inference result). Next, the decoder 21 inputs this word O1 to itself. In response to the input of this word O2, the decoder 21 outputs the next word O2 of the sign language sentence (inference result). In the same manner, the decoder 21 recursively inputs the sign language words of the inference result to itself, thereby sequentially outputting subsequent sign language words (inference results) O3, O4, ... The string of sign language words output by the decoder 21 in this way is the sign language sentence of the recognition result (inference result). In other words, the decoder 21 decodes the word probability distribution information output by the encoder 11 to estimate and output a two-handed sign language sentence as a string including both right-hand words and left-hand words. Note that this output from the decoder 21 reflects the sign language expression using both hands and is called a sign language sentence (two-handed inference) 105.
[0034] As a specific example, the sign language sentence (two-hand inference) 105 may be a string of sign language words such as [{instruction}, {right hand = long}, {left hand = pt3}, {name}, ...]. Here, each of {instruction}, {right hand = long}, {left hand = pt3}, {name}, ... is a sign language word. Note that {right hand = XXX} is a sign language word that indicates an expression using the right hand. Also, {left hand = XXX} is a sign language word that indicates an expression using the left hand. The subsequence [{right hand = long}, {left hand = pt3}] included in the above example of the sign language word string is an example of simultaneous expression using the right and left hands. Note that "pt3" represents a special sign language word that indicates pointing.
[0035] The decoder 21 has an updatable third internal parameter, and estimates a two-handed sign language sentence based on the value of the third internal parameter.
[0036] When the sign language recognition device 1 operates in the learning mode, the decoder 21 passes the obtained sign language sentence (both hands inference) 105 to the error calculation unit 28. When the sign language recognition device 1 operates in the learning mode, the error calculation unit 28 is provided with teacher data corresponding to the input video. The sign language sentence (both hands teacher) 125 in FIG. 3 is an example of such teacher data. The illustrated sign language sentence (both hands teacher) 125 is, for example, a string of sign language words, [{instruction}, {right hand = long}, {left hand = pt3}, {name}, ...].
[0037] When the sign language recognition device 1 operates in the learning mode, the error calculation unit 28 calculates the error between the inference result output from the decoder 21, i.e., the sign language sentence (two-hand inference) 105, and the corresponding sign language sentence (two-hand teacher) 125. The error may be, for example, the cross entropy between the sign language word strings. The error calculated by the error calculation unit 28 is an error related to the decoding result, and may be, for example, L decode That is, the error calculation unit 28 calculates the error between the two-handed sign language sentence output from the decoder 21 and the two-handed sign language sentence of the training data corresponding to the input image sequence (video). The error calculation unit 28 is also called a "two-handed error calculation unit."
[0038] When the sign language recognition device 1 operates in the learning mode, the encoder 11, the decoder 21, and the CNN 10 update their internal parameters using the backpropagation algorithm based on the errors calculated by the error calculation unit 28. The learning data here is a set of pairs of images input to the CNN 10 and the sign language sentences (two-hand teacher) 125 corresponding to the images. Typically, a large number of pairs are included in the learning data, and the encoder 11, the decoder 21, and the CNN 10 can repeatedly update their internal parameters using the backpropagation algorithm.
[0039] The sign language recognition device 1 of this embodiment can perform the following supplementary learning in parallel with the machine learning related to the above-mentioned processing flow from CNN 10 → encoder 11 → decoder 21. This supplementary learning adjusts internal parameters for the ability to associate images with sign language words for both right-hand and left-hand expressions.
[0040] 4 is a block diagram for explaining the processing of the CTC algorithm processing units 12-1 and 12-2 and the error calculation units 18-1 and 18-2. As shown in the figure, the CTC algorithm processing units 12-1 and 12-2 and the error calculation units 18-1 and 18-2 perform calculations based on word probability distribution information 102 generated by the encoder 11. The CTC algorithm processing unit 12-1 and the error calculation unit 18-1 perform processing for the right hand expression in the sign language expression included in the input video. The CTC algorithm processing unit 12-2 and the error calculation unit 18-2 perform processing for the left hand expression in the sign language expression included in the input video.
[0041] When the sign language recognition device 1 operates in the learning mode, each of the CTC algorithm processors 12-1 and 12-2 finds a word sequence with the highest likelihood by using the CTC algorithm based on the word probability distribution information 102. CTC is an abbreviation for Connectionist Temporal Classification.
[0042] The CTC algorithm processing unit 12-1 calculates a sign language sentence (right hand inference) 103 by using the CTC algorithm based on the word probability distribution information 102. The sign language sentence (right hand inference) 103 is a string of sign language words inferred corresponding to the right hand expression in sign language. The CTC algorithm processing unit 12-1 passes the obtained sign language sentence (right hand inference) 103 to the error calculation unit 18-1. Meanwhile, the CTC algorithm processing unit 12-2 similarly performs processing related to the left hand expression in sign language. That is, the CTC algorithm processing unit 12-2 calculates a sign language sentence (left hand inference) 104 by using the CTC algorithm based on the word probability distribution information 102. The sign language sentence (left hand inference) 104 is a string of sign language words inferred corresponding to the left hand expression in sign language. The CTC algorithm processing unit 12-2 passes the obtained sign language sentence (left hand inference) 104 to the error calculation unit 18-2. That is, the CTC algorithm processing unit 12-1 obtains a right-hand sign language sentence as a sequence of right-hand words based on the word probability distribution information output from the encoder 11. The CTC algorithm processing unit 12-2 obtains a left-hand sign language sentence as a sequence of left-hand words based on the word probability distribution information output from the encoder 11. The CTC algorithm processing unit 12-1 is also referred to as the "first algorithm processing unit." The CTC algorithm processing unit 12-2 is also referred to as the "second algorithm processing unit."
[0043] The error calculation units 18-1 and 18-2 respectively calculate the error between the sign language sentence (right hand inference) 103 passed from the CTC algorithm processing unit 12-1 and the sign language sentence (left hand inference) 104 passed from the CTC algorithm processing unit 12-2 and the training data.
[0044] That is, error calculation unit 18-1 calculates the error between sign language sentence (right hand inference) 103 obtained by CTC algorithm processing unit 12-1 and sign language sentence (right hand teacher) 113, which is teacher data provided corresponding to the input video. The error calculated by error calculation unit 18-1 may be, for example, cross entropy. Note that sign language sentence (right hand teacher) 113 is a correct sign language word string for the right hand expression included in the input video. That is, sign language sentence (right hand teacher) 113 is a correct sign language word string. The same applies to the left hand expression. That is, error calculation unit 18-2 calculates the error between sign language sentence (left hand inference) 104 obtained by CTC algorithm processing unit 12-2 and sign language sentence (left hand teacher) 114, which is teacher data provided corresponding to the input video. The error calculated by error calculation unit 18-2 may be, for example, cross entropy. The sign language sentence (left hand teacher) 114 is a correct string of sign language words for the expression of the left hand included in the input video.
[0045] That is, error calculation unit 18-1 calculates the error between the right-hand sign language sentence obtained by CTC algorithm processing unit 12-1 and the right-hand sign language sentence of the teacher data corresponding to the input image sequence (video). Error calculation unit 18-2 calculates the error between the left-hand sign language sentence obtained by CTC algorithm processing unit 12-2 and the left-hand sign language sentence of the teacher data corresponding to the input image sequence (video). Error calculation unit 18-1 is also called a "right-hand error calculation unit." Error calculation unit 18-2 is also called a "left-hand error calculation unit."
[0046] The sign language recognition device 1 adjusts the internal parameters of the encoder 11 and the CNN 10 based on the errors calculated by the error calculation unit 18-1 and the error calculation unit 18-2. The error calculated by the error calculation unit 18-1 is an error related to the expression of the right hand, for example, L right The error calculated by the error calculation unit 18-2 is an error related to the expression of the left hand, for example, L left When one set of learning data (a set of an input video, a sign language sentence (right hand teacher) 113, and a sign language sentence (left hand teacher) 114) is given, the sign language recognition device 1 calculates an error Lright After updating the internal parameters of the encoder 11 and the CNN 10 based on left Alternatively, the sign language recognition device 1 may update the internal parameters of the encoder 11 and the CNN 10 based on the error L right and error L left The internal parameters of the encoder 11 and the CNN 10 may be updated based on the combined error. This combined error is expressed, for example, by the following equation (1). In equation (1), ω is a parameter that is determined appropriately and is in the range of 0≦ω≦1. As an example, ω may be set to 0.5.
[0047] ω·L right +(1-ω)·L left ···(1)
[0048] Next, an example of the machine learning procedure in the sign language recognition device 1 of this embodiment will be described.
[0049] 5 is a flowchart showing an example of a processing procedure for performing machine learning to optimize internal parameters in the sign language recognition device 1. The learning procedure will be described below with reference to this flowchart.
[0050] In step S1, the control unit 90 of the sign language recognition device 1 determines whether a learning termination condition is met. If the learning termination condition is met (step S1: YES), the entire process of this flowchart ends. If the learning termination condition is not met (step S1: NO), the process proceeds to step S2. Specific learning termination conditions may be determined as appropriate. For example, the learning termination condition may be determined to be met when learning using a predetermined number of pieces of training data is completed. Alternatively, the learning termination condition may be determined to be met when learning using given training data a predetermined number of times (number of epochs) is completed. Alternatively, the learning termination condition may be determined to be met when the set of values of all internal parameters to be optimized by training has sufficiently converged. Alternatively, the termination condition may be determined based on other conditions. Furthermore, the termination condition may be determined by combining two or more of the conditions listed here.
[0051] In step S2, the sign language recognition device 1 acquires one item (one set) of learning data. The learning data is provided as appropriate. One item of learning data is a set of an input video, a sign language sentence (both hands teacher) 125 corresponding to the input video, a sign language sentence (right hand teacher) 113 corresponding to the input video, and a sign language sentence (left hand teacher) 114 corresponding to the input video. The input image is a sequence of still images.
[0052] In step S3, the CNN 10 and the encoder 11 perform processing based on the input video to obtain word probability distribution information 102. The word probability distribution information 102 obtained in this step may be temporarily stored in a storage means (e.g., a semiconductor memory or a magnetic hard disk drive). This word probability distribution information 102 is read out and used later in step S6. Also in step S3, the decoder 21 obtains a sign language sentence (two-hand inference) 105 based on the word probability distribution information 102.
[0053] Next, in step S4, the error calculation unit 28 calculates the error between the sign language sentence (two hands inference) 105 calculated by the decoder 21 and the sign language sentence (two hands teacher) 125 included in the given training data.
[0054] Then, in step S5, the sign language recognition device 1 uses the backpropagation method based on the error calculated in step S4 to update the internal parameter values of the decoder 21, the encoder 11, and the CNN 10. The control unit 90 controls the decoder 21, the encoder 11, and the CNN 10 to update the parameter values.
[0055] Next, in step S6, the CTC algorithm processing unit 12-1 obtains a sign language sentence (right-hand inference) 103 based on the already obtained word probability distribution information 102. Similarly, the CTC algorithm processing unit 12-2 obtains a sign language sentence (left-hand inference) 104 based on the word probability distribution information 102. Note that the CTC algorithm processing units 12-1 and 12-2 do not have internal updatable parameters. In other words, the CTC algorithm processing units 12-1 and 12-2 are not targets for learning. That is, the CTC algorithm processing units 12-1 and 12-2 calculate the sign language sentences (right-hand inference and left-hand inference, respectively) without relying on internal parameters, etc.
[0056] In step S7, the error calculation unit 18-1 calculates the error L between the sign language sentence (right hand inference) 103 obtained by the CTC algorithm processing unit 12-1 and the sign language sentence (right hand teacher) 113 included in the given learning data. right Calculate.
[0057] In step S8, the error calculation unit 18-2 calculates the error L between the sign language sentence (left-hand inference) 104 obtained by the CTC algorithm processing unit 12-2 and the sign language sentence (left-hand teacher) 114 included in the given training data. left and calculate.
[0058] In step S9, the sign language recognition device 1 uses the backpropagation method based on the errors calculated in steps S7 and S8 to update the internal parameter values of the encoder 11 and the CNN 10. The control unit 90 controls the encoder 11 and the CNN 10 to update the parameter values.
[0059] In the process of step S9, the sign language recognition device 1 first calculates the error L right Then, the internal parameters of the encoder 11 and the CNN 10 are updated based on the error L left Alternatively, the sign language recognition device 1 may update the internal parameters of the encoder 11 and the CNN 10 based on the error L left Then, the internal parameters of the encoder 11 and the CNN 10 are updated based on the error L right Alternatively, the sign language recognition device 1 may calculate the overall error (Equation (1)) and update the internal parameters of the encoder 11 and the CNN 10 based on the overall error.
[0060] After the process of step S9 is completed, the process returns to step S1 to determine whether or not to repeat the process.
[0061] 5 may be partially modified. right and the error L calculated by the error calculation unit 18-2. left The internal parameters of the encoder 11 and the CNN 10 may be updated by backpropagation based on the error calculated by the error calculation unit 28 (processing in step S9), and then the internal parameters of the decoder 21, the encoder 11, and the CNN 10 may be updated by backpropagation based on the error calculated by the error calculation unit 28 (processing in step S5).
[0062] In either case, the sign language recognition device 1 simultaneously performs learning based on the output from the decoder 21 and auxiliary learning based on the output from the CTC algorithm processing units 12-1 and 12-2, based on a large number of sets of training data.
[0063] That is, when the sign language recognition device 1 operates in the learning mode, the third internal parameters (internal parameters of the decoder 21), the second internal parameters (internal parameters of the encoder 11), and the first internal parameters (internal parameters of the CNN 10) can be updated based on the errors calculated by the error calculation unit 28 (both hand error calculation unit).The second internal parameters and the first internal parameters can be updated based on the errors calculated by the error calculation unit 18-1 (right hand error calculation unit) and the errors calculated by the error calculation unit 18-2 (left hand error calculation unit).
[0064] When the sign language recognition device 1 operates in the learning mode, the second internal parameters and the first internal parameters may be updated based on the error calculated by the error calculation unit 18-1 (right-hand error calculation unit) for an image sequence (input video) included in one piece of learning data provided, and then the second internal parameters and the first internal parameters may be updated based on the error calculated by the error calculation unit 18-2 (left-hand error calculation unit). The second internal parameters and the first internal parameters may be updated based on the error calculated by the error calculation unit 18-2 (left-hand error calculation unit), and then the second internal parameters and the first internal parameters may be updated based on the error calculated by the error calculation unit 18-1 (right-hand error calculation unit). Alternatively, the second internal parameters and the first internal parameters may be updated based on the overall error (see Equation (1) above) calculated based on the error calculated by the error calculation unit 18-1 (right-hand error calculation unit) and the error calculated by the error calculation unit 18-2 (left-hand error calculation unit) for an image sequence (input video) included in one piece of learning data provided.
[0065] Furthermore, in steps S5 and S9, parameter updates may not be performed for each piece (set) of learning data acquired in step S2, but may be performed only under conditions where processing has been completed for a predetermined number of pieces of learning data.
[0066] As described above, the sign language recognition device 1 having the configuration specific to this embodiment also performs the above-mentioned supplementary training to improve the recognition accuracy of each of the right-hand words and left-hand words in simultaneous expressions. In other words, by performing the above-mentioned supplementary training, the CNN 10 and the encoder 11 are trained to output better word probability distribution information 102 for estimating each of the right-hand words and left-hand words in simultaneous expressions.
[0067] The sign language recognition device 1 performs sign language recognition (estimation) processing for an unknown image sequence based on the values of internal parameters obtained as a result of learning. Furthermore, the sign language recognition device 1 does not necessarily have the function of parameter learning when recognizing an unknown image sequence. That is, the sign language recognition device 1 may be configured as follows: The CNN 10 performs a process of extracting features from each image included in an input image sequence through a convolution process. The encoder 11 encodes the information output from the CNN 10 to generate word probability distribution information representing the probability distribution of both right-hand and left-hand words represented by the image sequence (input video). The decoder 21 decodes the word probability distribution information output from the encoder 11 to estimate and output a two-handed sign language sentence as a string including both right-hand and left-hand words. Here, the CNN 10 performs a process of convolving images based on a first internal parameter obtained as a result of learning by the above-mentioned learning device. The encoder 11 also generates word probability distribution information based on a second parameter obtained as a result of learning by the above-mentioned learning device. The decoder 21 also estimates the two-handed sign language sentence based on the third parameter obtained as a result of learning by the learning device described above.
[0068] FIG. 6 is a block diagram showing an example of the internal configuration for realizing the sign language recognition device 1. The sign language recognition device 1 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904, 905, etc., and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902, etc. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices, etc. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port 903 via the bus 906.
[0069] At least some of the functions of the sign language recognition device 1 in the above-described embodiment can be realized by a computer and a program. In this case, the functions can be realized by recording a program for realizing the functions on a computer-readable recording medium and loading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into a computer system. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within a computer system that serves as a server or client in such cases. The program may be designed to realize some of the functions described above, or may be capable of realizing the functions described above in combination with a program already stored in the computer system.
[0070] The method for realizing a predetermined function using machine learning techniques is as follows. Here, for convenience, a unit of function that can be machine-learned is referred to as a "function unit." The function unit calculates data to be output based on input data. The function unit uses internal parameter values when calculating the data to be output. The internal parameters can be updated and optimized through a learning process. When training the function unit, training data is used. The training data includes input data to the function unit and a correct answer for output data calculated based on the input data. During the learning process, the function unit reads the input data included in the training data and references output data based on the input data and using the internal parameter values at that time. This output data is an estimated value calculated based on the internal parameters at that time and does not necessarily coincide with the correct answer. To update the internal parameters, the difference between the estimated value calculated and output by the function unit based on the input data at that time and the correct answer corresponding to the input data is calculated. This difference is called an error, loss, etc. The difference calculated here may be, for example, the absolute value of the difference between scalars, a squared error, a cross-entropy error, or a difference calculated by other methods. Based on the calculated difference, the internal parameter values can be updated using backpropagation. This operation adjusts the internal parameter values so as to reduce the error. By performing the above operation multiple times (multiple times) using a predetermined amount (large amount) of training data, the internal parameter values are optimized. In other words, the functional unit is adjusted to perform the processing exemplified by the training data provided. After a sufficient amount of training has been performed, the training process may be terminated. The set of trained internal parameter values at this time may be stored in a storage device or the like, enabling estimation based on the training results. A function that enables data input and output, including the internal parameters, may be called a "model." In other words, a model includes internal parameters. Training the functional unit may be called "model training."One way to implement the model is to use a neural network, and depending on the type of processing to be performed, an appropriate form of neural network can be used.
[0071] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Industrial Applicability]
[0072] The present invention can be used, for example, to recognize human movements (such as sign language) that include simultaneous expressions of the right and left hands. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]
[0073] 1. Sign language recognition device (learning device) 10 CNN (Convolutional Neural Network, convolution processing unit) 11 Encoder 12-1 CTC algorithm processing section (first algorithm processing section) 12-2 CTC algorithm processing section (second algorithm processing section) 18-1 Error calculation section (right-hand error calculation section) 18-2 Error calculation section (left hand error calculation section) 21 Decoder 28 Error calculation unit (both hands error calculation unit) 90 Control Unit 101 Input video 102 Word Probability Distribution Information 103 Sign Language Sentences (Right-Hand Inference) 104 Sign Language Sentences (Left-Hand Inference) 105 Sign Language Sentences (Two-Handed Inference) 113 Sign Language Text (Right Hand Teacher) 114 Sign Language Text (Left Hand Teacher) 125 Sign Language Text (Teacher with Both Hands) 201 Start symbol 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus
Claims
1. a convolution processing unit that extracts features by convolution processing for each image included in an input image sequence; an encoder that encodes the information output from the convolution processor to generate word probability distribution information representing probability distributions for both right-hand and left-hand words represented by the sequence of images; a decoder that decodes the word probability distribution information output from the encoder to estimate and output a two-handed sign language sentence as a string including both right-hand words and left-hand words; a first algorithm processing unit that obtains a right-hand sign language sentence as a sequence of right-hand words based on word probability distribution information output from the encoder; a second algorithm processing unit that obtains a left-hand sign language sentence as a sequence of left-hand words based on word probability distribution information output from the encoder; a two-handed error calculation unit that calculates an error between a two-handed sign language sentence output from the decoder and a two-handed sign language sentence of training data corresponding to the input image sequence; a right-hand error calculation unit that calculates an error between the right-hand sign language sentence obtained by the first algorithm processing unit and a right-hand sign language sentence of training data corresponding to the input image sequence; a left-hand error calculation unit that calculates an error between the left-hand sign language sentence obtained by the second algorithm processing unit and a left-hand sign language sentence of training data corresponding to the input image sequence; Equipped with the convolution processing unit has an updatable first internal parameter, and performs a process of convolving an image based on the first internal parameter; the encoder has an updatable second internal parameter, and generates the word probability distribution information based on the second internal parameter; the decoder has an updatable third internal parameter, and estimates the two-handed sign language sentence based on the third internal parameter; The third internal parameter, the second internal parameter, and the first internal parameter are configured to be updated based on the errors calculated by the two-hand error calculation unit, and the second internal parameter and the first internal parameter are configured to be updated based on the errors calculated by the right-hand error calculation unit and the errors calculated by the left-hand error calculation unit. Learning device.
2. With respect to the image sequence contained in one given learning data, updating the second internal parameters and the first internal parameters based on the error calculated by the right-hand error calculation unit, and then updating the second internal parameters and the first internal parameters based on the error calculated by the left-hand error calculation unit; or updating the second internal parameters and the first internal parameters based on the error calculated by the left-hand error calculation unit, and then updating the second internal parameters and the first internal parameters based on the error calculated by the right-hand error calculation unit; The learning device according to claim 1 .
3. With respect to the image sequence contained in one given learning data, updating the second internal parameters and the first internal parameters based on a total error calculated based on the error calculated by the right-hand error calculation unit and the error calculated by the left-hand error calculation unit; The learning device according to claim 1 .
4. a convolution processing unit that extracts features by convolution processing for each image included in an input image sequence; an encoder that encodes the information output from the convolution processor to generate word probability distribution information representing probability distributions for both right-hand and left-hand words represented by the sequence of images; a decoder that decodes the word probability distribution information output from the encoder to estimate and output a two-handed sign language sentence as a string including both right-hand words and left-hand words; Equipped with the convolution processing unit performs a process of convolving an image based on a first internal parameter obtained as a result of learning by the learning device according to claim 1; The encoder generates the word probability distribution information based on a second parameter obtained as a result of learning by the learning device according to claim 1; The decoder estimates the two-handed sign language sentence based on a third parameter obtained as a result of learning by the learning device according to claim 1. Sign language recognition device.
5. a convolution processing unit that extracts features by convolution processing for each image included in an input image sequence; an encoder that encodes the information output from the convolution processor to generate word probability distribution information representing probability distributions for both right-hand and left-hand words represented by the sequence of images; a decoder that decodes the word probability distribution information output from the encoder to estimate and output a two-handed sign language sentence as a string including both right-hand words and left-hand words; a first algorithm processing unit that obtains a right-hand sign language sentence as a sequence of right-hand words based on word probability distribution information output from the encoder; a second algorithm processing unit that obtains a left-hand sign language sentence as a sequence of left-hand words based on word probability distribution information output from the encoder; a two-handed error calculation unit that calculates an error between a two-handed sign language sentence output from the decoder and a two-handed sign language sentence of training data corresponding to the input image sequence; a right-hand error calculation unit that calculates an error between the right-hand sign language sentence obtained by the first algorithm processing unit and a right-hand sign language sentence of training data corresponding to the input image sequence; a left-hand error calculation unit that calculates an error between the left-hand sign language sentence obtained by the second algorithm processing unit and a left-hand sign language sentence of training data corresponding to the input image sequence; Equipped with the convolution processing unit has an updatable first internal parameter, and performs a process of convolving an image based on the first internal parameter; the encoder has an updatable second internal parameter, and generates the word probability distribution information based on the second internal parameter; the decoder has an updatable third internal parameter, and estimates the two-handed sign language sentence based on the third internal parameter; The third internal parameter, the second internal parameter, and the first internal parameter are configured to be updated based on the errors calculated by the two-hand error calculation unit, and the second internal parameter and the first internal parameter are configured to be updated based on the errors calculated by the right-hand error calculation unit and the errors calculated by the left-hand error calculation unit. A program that enables a computer to function as a learning device.