Sign language moving image translation apparatus, method, and computer program product

Abstract

This invention provides a sign language motion image translation device, method, and computer program product. It provides a sign language motion image translation device, method, and program that can recognize sign language content in real time without waiting for the output of a translated article. It comprises: a sign language motion image segmentation unit (11) that segments information related to sign language motion images obtained by capturing sign language based on preset conditions, generating time-series segmented sign language motion image information; a translation unit (12) that translates the time-series segmented sign language motion image information to generate words; an article generation unit (13) that uses a string of words obtained by periodically summarizing the time-series words generated by the translation unit (12) to generate an article; and a display control unit (14) that sequentially updates and displays the articles periodically generated by the article generation unit (13).

Description

Technical Field

[0001] This invention relates to a sign language motion image translation apparatus, method, program, and system for translating and displaying motion images of sign language obtained by photographing sign language. Background Technology

[0002] In the past, when audiences (hearing individuals) and hearing-impaired individuals communicated, the common method was to convert the audience's speech into text for the hearing-impaired to read, and then the hearing-impaired would respond via keyboard input or handwriting. However, in scenarios such as meetings where rapid and accurate communication is required, using keyboard input and handwriting is sometimes insufficient.

[0003] In recent years, various techniques related to using cameras to capture and translate motion images of sign language have been proposed as a solution to this problem.

[0004] For example, in Non-Patent Literature 1, a technique is proposed that uses a fixed time width (sliding window) to segment sign language motion images and translates them into words using a learned isolated sign language recognition model.

[0005] In addition, Non-Patent Document 2 proposes a technique that uses CNN (Convolutional Neural Network) and LSTM (Long Short-Term Memory) to recognize sign language motion images and generate articles.

[0006] In addition, Patent Document 1 proposes a technique that combines speech recognition and sign language recognition to improve translation accuracy.

[0007] Existing technical documents

[0008] Patent documents

[0009] Patent Document 1: Japanese Patent Application Publication No. 2015-76774

[0010] Non-patent literature

[0011] Non-patent literature 1: Ronglai Zuo, Fangyun Wei, Brian Mak, "Towards Online SignLanguage Recognition and Translation", arXiv, 2401.05336v1 [cs.CV], 10 Jan 2024

[0012] Non-Patent Document 2: Doi Yurika, Yagi Takuma, Mizuguchi Tomohito, "Development of a Sign Language Recognition System Using CNN-LSTM", Technical Report of the Japanese Society for Artificial Intelligence, SIG-AGI-001-06, December 15, 2015 Summary of the Invention

[0013] Problem to be Solved by the Invention

[0014] However, in Non-Patent Document 1, although a word string can be obtained in real time, there is no proposal on how to generate a sentence using the word string.

[0015] In addition, in Non-Patent Document 2, sign language moving images are translated in sentence units, so if the sign language is long, there is no clue about the content before the translation result is obtained, which causes the problem that understanding is difficult.

[0016] In addition, in Patent Document 1, there is a problem that the recognition accuracy decreases when there is no audio information and only sign language is recognized.

[0017] In view of the above circumstances, an object of the present invention is to provide a sign language moving image translation apparatus, method and program capable of recognizing the content of sign language in real time without waiting for the output of a translation result of one complete sentence.

[0018] Solution to Problem

[0019] The sign language moving image translation apparatus of the present invention comprises: a sign language moving image segmentation unit that segments information related to a sign language moving image obtained by capturing sign language based on preset conditions, and generates time-series segmented sign language moving image information; a translation unit that translates the time-series segmented sign language moving image information to generate words; a sentence generation unit that generates a sentence using a word string obtained by periodically aggregating time-series words generated by the translation unit; and a display control unit that sequentially updates and displays sentences periodically generated by the sentence generation unit.

[0020] Effect of the Invention

[0021] According to the sign language moving image translation apparatus of the present invention, information related to a sign language moving image obtained by capturing sign language is segmented to generate time-series segmented sign language moving image information, the time-series segmented sign language moving image information is translated to generate words, a sentence is generated using a word string obtained by periodically aggregating the time-series words, and the generated sentence is sequentially updated and displayed. Therefore, the content of sign language can be recognized in real time without waiting for the output of a translation result of one complete sentence. Brief Description of Drawings

[0022] Figure 1This is a block diagram illustrating the outline structure of an embodiment of a sign language motion image translation system using one embodiment of the sign language motion image translation device of the present invention.

[0023] Figure 2 This is a graph showing the velocity changes of three feature points before and after standardization, based on shoulder width.

[0024] Figure 3 It is used for explanation Figure 1 The flowchart shown is a process flow of the sign language motion image translation system.

[0025] Figure 4 It is used for explanation Figure 1 The diagram illustrates the processing flow of the sign language motion image translation system. Detailed Implementation

[0026] Hereinafter, a sign language motion image translation system 1 using an embodiment of the sign language motion image translation device of the present invention will be described in detail with reference to the accompanying drawings. Figure 1 This is a block diagram showing the outline structure of the sign language motion image translation system 1 of this embodiment.

[0027] The sign language motion image translation system 1 of this embodiment is a system for translating and displaying sign language motion images obtained by photographing sign language. Specifically, it is a system that translates sign language motion images word by word and updates and displays the text whenever a new word is added.

[0028] like Figure 1 As shown, the sign language motion image translation system 1 of this embodiment includes a sign language motion image translation device 10 and a terminal device 20 having a camera for capturing sign language.

[0029] The sign language motion image translation device 10 and the terminal device 20 are connected via communication lines such as Internet lines and LAN (Local Area Network) lines, enabling them to exchange various types of information.

[0030] like Figure 1 As shown, the sign language motion image translation device 10 includes a sign language motion image segmentation unit 11, a translation unit 12, a text generation unit 13, and a display control unit 14.

[0031] The sign language motion image segmentation unit 11 segments information related to the sign language motion images captured by the terminal device 20 based on preset conditions, and generates time-series segmented sign language motion image information.

[0032] Specifically, the sign language motion image segmentation unit 11 first extracts feature points (also called landmarks) from each frame of the image constituting the input sign language motion image. For example, the positions of joints, fingertips, etc., associated with sign language are extracted as feature points. Furthermore, it is not limited to hand movements; feature points can also be extracted to include facial movements, expressions, arm movements, and body movements. As for the feature point extraction process, existing image processing methods can be used, but a machine learning model obtained by pre-processing feature points can also be used to extract feature points.

[0033] Then, the sign language motion image segmentation unit 11 uses feature points extracted from each frame image to generate the aforementioned time-series segmented sign language motion image information. In this embodiment, the sign language motion image segmentation unit 11 uses the position, angle, etc. of each part to segment (distinguish) feature points of a series of frame images into action units, thereby generating time-series segmented sign language motion image information.

[0034] More specifically, the sign language motion image segmentation unit 11, for example, calculates the coordinates of the center position of the palm based on the extracted hand feature points, and segments the action by using the time point when the movement speed of the palm's center position reaches or exceeds a predetermined threshold. Additionally, the sign language motion image segmentation unit 11 calculates the coordinates of the center position of the face based on the extracted facial feature points, and calculates the coordinates of the center position of the palm based on the hand feature points, calculates the angular velocity of the palm's center position centered on the face's center position, and segments the action by using the time point when this angular velocity reaches or exceeds a predetermined threshold.

[0035] The reason for using speed information derived from feature points to identify the boundaries of actions is that there is a tendency for hand movements to speed up at the boundaries of sign language actions.

[0036] Furthermore, when segmentation is performed based on velocity information derived from coordinate values ​​of feature points such as hands and faces, as described above, the appropriate threshold may change depending on the distance from the terminal device 20 (camera) and the size of the sign language interpreter. As a solution, initial calibration is considered, but this is not only time-consuming but also cannot cope with situations where the distance to the terminal device 20 (camera) changes constantly.

[0037] Therefore, the coordinate values ​​of feature points in each frame can also be standardized using the shoulder width of the sign language interpreter in each frame. In this case, the threshold is also preset based on the standardized values. For example, shoulder width-based standardization can be performed by performing the following calculation.

[0038] Let the coordinates of the right shoulder at a certain time t be xrs(t) and yrs(t), and the coordinates of the left shoulder be xls(t) and yls(t). The shoulder width d(t) at that time is obtained by the following formula (1).

[0039]

[0040] Using the coordinates xrh(t) and yrh(t) of the right hand before standardization, the velocity vrh_n(t) of the right hand after standardization with the shoulder width at that moment can be obtained by the following formula (2).

[0041]

[0042] By standardizing based on shoulder width, the distance to the terminal device 20 (camera) and the differences in the sign language interpreter's physique can be ignored.

[0043] Figure 2 A is a graph showing the velocity changes (in pixels per second) of three feature points (P1, P2, P3) of the right hand before standardization. Figure 2 Figure B is a graph showing the velocity changes (in [shoulder width / s]) of three feature points (P1, P2, P3) of the right hand after standardization to shoulder width. Figure 2 A and Figure 2 As shown in B, it is known that the velocity changes of the three feature points before and after standardization based on shoulder width did not change.

[0044] Furthermore, in the above description, the standardization is set to shoulder width, but it is not limited to this. As long as there are two feature points with a certain width that are reflected in the terminal device 20 (camera) regardless of the sign language activity or body posture, other feature points (physical information) can be used. For example, it is also possible to use two feature points that represent the size of the head.

[0045] Furthermore, when the coordinate values ​​of feature points extracted from each frame contain noise, they may be frequently demarcated in unintended areas. Therefore, it is also possible to remove noise by taking a moving average of the coordinate values ​​of each feature point over a specified time width.

[0046] Alternatively, instead of using feature points as described above, segmentation of sign language motion image information can be generated by dividing the image at predetermined intervals, such as 0.5-second intervals.

[0047] The translation unit 12 translates the segmented sign language motion image information generated in the time series by the sign language motion image segmentation unit 11 to generate words. Specifically, in this embodiment, the translation unit 12 obtains words by inputting the segmented sign language motion image information into an isolated sign language recognition model obtained through machine learning. In the isolated sign language recognition model, segmented sign language motion image information that is input in a time series is used to obtain each word corresponding to each segmented sign language motion image information.

[0048] The article generation unit 13 uses word strings obtained by periodically summarizing the words in the time series generated by the translation unit 12 to generate articles.

[0049] In this embodiment, the article generation unit 13 performs preprocessing before generating the article. As preprocessing, the article generation unit 13 first deletes any duplicate words in the chronologically ordered list of words if any word exists before or after a specified word. In sign language, actions are sometimes repeated, and the number of repetitions varies depending on the sign language interpreter. In such cases, words generated by the translation unit 12 may be repeated, therefore they are deleted.

[0050] Next, after deleting words that are repeated before or after the given words, the article generation unit 13 calculates the combination of a given word and the words preceding or following it. If the combination is a pre-defined combination, it converts it into other pre-defined words. Specifically, for example, if the combination is "first time" and "meeting", it converts it into "first meeting". If the combination is "today" and "evening", it converts it into "tonight". If the combination is "today" and "morning", it converts it into "this morning". In other words, the article generation unit 13 converts the words into other words that are synonymous with the combination of words.

[0051] The article generation unit 13 has a pre-set dictionary table that includes combinations of words as described above and words corresponding to those combinations (other words that are synonyms of the word combinations). The article generation unit 13 refers to this dictionary table to perform word conversion.

[0052] Next, after performing the word transformation as described above, the article generation unit 13 generates a word string obtained by periodically summarizing the words in the time series, and inputs the word string into a large-scale language model (LLM) to generate an article.

[0053] Specifically, the article generation unit 13 sequentially accumulates the words of the time series in a cache. Whenever a new word is input into the cache, the word string accumulated in the cache up to that point in time is input into the large-scale language model to generate an article. That is, in this embodiment, whenever a new word is added to n (n is a natural number greater than 1) words or word strings, a new article is generated and the article is updated. In this embodiment, articles are not generated on a sentence-by-sentence basis, but rather the word string accumulated in the cache is used to generate an article before all the words constituting a sentence are complete.

[0054] Then, the article generation unit 13, upon detecting the article's delimitation based on the words in the time series, clears the word strings accumulated in the cache.

[0055] Furthermore, in this embodiment, as a method for periodically summarizing words in a time series as described above, a word string is generated whenever a new word is input. However, the method for periodically summarizing words is not limited to this. For example, it can also be set to sum up words stored in the cache at preset time intervals to generate a word string.

[0056] Furthermore, when the text generation unit 13 inputs the word string into the large-scale language model as described above, it also inputs the grammatical rules of the sign language. Examples of grammatical rules for sign language include, for instance, the sign language word for "end" sets the preceding verb to the past tense, and the sign language word for "place" functions as a preposition such as "to" or "at." These are input as prompts into the large-scale language model. Moreover, the grammatical rules for sign language are not limited to these; other well-known grammatical rules can be input.

[0057] As a large-scale language model, it can be used for natural language processing, such as ChatGPT (registered trademark), Microsoft 365Copilot (registered trademark), and Gemini (registered trademark). However, it is not limited to these; other known techniques can also be used to generate articles.

[0058] The display control unit 14 sequentially updates and displays articles periodically generated by the article generation unit 13. The destination for displaying the articles can be a display device such as a monitor connected to the sign language motion image translation device 10 (not shown), or other terminal devices (not shown).

[0059] The sign language motion image translation device 10 includes a CPU (Central Processing Unit), ROM (Read Only Memory) and RAM (Random Access Memory) and other semiconductor memory, storage devices such as a hard disk, and communication I / F (Interface).

[0060] One embodiment of the sign language motion image translation program of the present invention is installed in the storage device of the sign language motion image translation device 10. The functions of the aforementioned parts of the sign language motion image translation device 10 are executed by starting the sign language motion image translation program by the CPU.

[0061] In addition, in this embodiment, the functions of each part are performed by the CPU executing the sign language motion image translation program. However, it is also possible that some or all of the functions performed by the sign language motion image translation program are performed by hardware such as GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and other circuits.

[0062] Next, the terminal device 20 will be described.

[0063] As described above, the terminal device 20 is a device for use by sign language interpreters, and may be composed of a mobile terminal such as a tablet terminal or a smartphone. However, it is not limited to this and may also be composed of a personal computer.

[0064] The terminal device 20 has a camera for capturing the sign language of the sign language interpreter. The motion images of the sign language captured by the terminal device 20 are sent to the sign language motion image translation device 10.

[0065] Furthermore, in this embodiment, the sign language is recorded using the terminal device 20, but it is not limited to this; it can also be recorded using only a camera.

[0066] Next, refer to Figure 3 The flowchart and Figure 4 The illustrated diagrams will be used to explain the processing flow of the sign language motion image translation system 1 of this embodiment.

[0067] First, the sign language terminal device 20 captures the sign language (S10), and sends the captured sign language motion image to the sign language motion image translation device 10.

[0068] The sign language motion image translation device 10 receives sign language motion images sent from the terminal device 20 and extracts feature points from each frame of the sign language motion image in the sign language motion image segmentation unit 11 (S12).

[0069] Next, in the sign language motion image segmentation unit 11, feature points of a series of frame images are segmented into action units to generate time-series segmented sign language motion image information (S14). Furthermore, Figure 4 F shown represents the feature points of each frame of the image. Figure 3 D1, D2, D3, and D4 shown represent segmented sign language motion image information.

[0070] Next, the segmented sign language motion image information of the time series is input into the translation unit 12, which translates the input segmented sign language motion image information into words (S16). Figure 3 In the example shown, the segmented sign language motion image information D1~D4 is translated as “yesterday”, “evening”, “drink” and “end”, respectively.

[0071] Then, the time-series words translated in translation unit 12 are input to article generation unit 13, where preprocessing (S18) is performed as described above. Specifically, as mentioned above, repeated words are removed, and pre-defined word combinations are converted into other words. Figure 4 In the example shown, the combination of "yesterday" and "evening" is changed to "last night".

[0072] Next, the article generation unit 13 generates a word string obtained by periodically summarizing the words in the time series, and inputs this word string and the grammatical rules of sign language into a large-scale language model, thereby generating an article (S20). In this embodiment, as described above, an article is generated whenever a new word is added; therefore, in Figure 4 In the example shown, first, after generating an article consisting of the single word "last night," an article consisting of the two words "last night" and "drink" is generated: "Last night, I drank." Then, using the word "end" and the sign language rule of "setting the preceding word to the past tense when 'end' appears," an article consisting of the three words "last night," "drink," and "end" is generated: "Last night, I drank."

[0073] Then, the display control unit 14 sequentially updates and displays the articles periodically generated by the article generation unit 13 (S22). Figure 4 In the example shown, first, after displaying only words like "tonight," an article like "last night, drank." is displayed, and finally, an article like "last night, drank." is displayed.

[0074] In addition to displaying the periodically generated articles in S22, it is also possible to set it to display the words generated in S16 side-by-side (in... Figure 4 The examples shown are "yesterday", "evening", "drink", and "end", and the words generated in S18 (in Figure 4 The examples shown are "last night", "drink", and "end". Alternatively, it can be configured to generate multiple translated texts in different modes and display them side-by-side, or allow the user to select from these modes. These translated texts, in addition to the word strings generated in S16 and S18, can also be translations using different sign language grammar rules, or translations using synonyms or near-synonyms for the same sign language.

[0075] According to the above-described sign language motion image translation system 1, information related to the sign language motion image obtained by capturing sign language is segmented to generate time-series segmented sign language motion image information. The time-series segmented sign language motion image information is translated to generate words. A word string obtained by periodically summarizing the words in the time series is used to generate an article. The generated article is updated and displayed sequentially. Therefore, the content of sign language can be recognized in real time without waiting for the translation result of an article.

[0076] Furthermore, in the sign language motion image translation system 1 described above, articles are generated by inputting word strings and sign language grammar rules into a large-scale language model, thereby improving translation accuracy and generating easily understandable articles.

[0077] Furthermore, in the sign language motion image translation system 1 of the above embodiment, when a given word and the words before or after it are a pre-set combination, the given word and the words before or after it are converted into other pre-set words, thus enabling the generation of articles that are easier to understand.

[0078] Furthermore, in the sign language motion image translation system 1 of the above embodiment, feature points extracted from the sign language motion image are used to generate segmented sign language motion image information, so more appropriate segmentation can be achieved through simple computational processing.

[0079] Furthermore, in the sign language motion image translation system 1 of the above embodiment, the velocity information of feature points is used to generate segmented sign language motion image information, thus enabling high-precision and more appropriate segmentation.

[0080] Furthermore, in the sign language motion image translation system 1 of the above embodiment, when the coordinate values ​​of feature points are standardized using the shoulder width of the person performing the sign language, more appropriate segmented sign language motion image information can be generated without depending on the physique of the person performing the sign language or the distance from the terminal device 20 (camera).

[0081] Furthermore, the present invention is not limited to the embodiments described above, and can be further embodied by modifying the constituent elements during the implementation phase without departing from its spirit. Additionally, various inventions can be formed by appropriately combining the multiple constituent elements disclosed in the above embodiments. For example, all the constituent elements shown in the embodiments can be appropriately combined. Of course, various modifications and applications can be made within this scope without departing from the spirit of the invention.

[0082] The following notes further disclose the present invention.

[0083] (Postscript 1)

[0084] The sign language motion image translation device of the present invention comprises: a sign language motion image segmentation unit that segments information related to sign language motion images obtained by capturing sign language based on preset conditions, and generates time-series segmented sign language motion image information; a translation unit that translates the time-series segmented sign language motion image information to generate words; an article generation unit that generates an article using a word string obtained by periodically summarizing the words in the time-series generated by the translation unit; and a display control unit that sequentially updates and displays the articles periodically generated by the article generation unit.

[0085] (Postscript 2)

[0086] In the sign language motion image translation device described in Appendix 1, the article generation unit is able to generate articles by inputting word strings and sign language grammar rules into a large-scale language model.

[0087] (Note 3)

[0088] In the sign language motion image translation device described in Appendix 1 or 2, when the specified word and the words before or after it are a pre-set combination, the text generation unit is able to convert the specified word and the words before or after it into other pre-set words.

[0089] (Postscript 4)

[0090] In any of the sign language motion image translation devices described in Appendix 1 to 3, the sign language motion image segmentation unit is able to acquire feature points extracted from motion images obtained by photographing sign language as information related to the sign language motion image, and use the feature points to generate segmented sign language motion image information.

[0091] (Note 5)

[0092] In the sign language motion image translation device described in Appendix 4, the sign language motion image segmentation unit is able to generate segmented sign language motion image information using the velocity information of feature points.

[0093] (Note 6)

[0094] In the sign language motion image translation device described in Appendix 4 or 5, the sign language motion image segmentation unit is able to standardize the coordinate values ​​of feature points using the physical information of the person performing the sign language.

[0095] (Note 7)

[0096] In the sign language motion image translation method of the present invention, information related to the sign language motion image obtained by shooting sign language is segmented based on pre-set conditions to generate time series segmented sign language motion image information, words are generated by translating the generated time series segmented sign language motion image information, and articles are generated by periodically summarizing the words in the generated time series. The periodically generated articles are then updated and displayed sequentially.

[0097] (Note 8)

[0098] The sign language motion image translation program of the present invention enables a computer to perform the following steps: segmenting information related to sign language motion images obtained by shooting sign language based on preset conditions, generating time-series segmented sign language motion image information; translating the generated time-series segmented sign language motion image information to generate words; using word strings obtained by periodically summarizing the words in the generated time series to generate articles; and sequentially updating and displaying the periodically generated articles.

[0099] Explanation of reference numerals in the attached figures

[0100] 1. Sign Language Motion Image Translation System

[0101] 10 Sign Language Motion Image Translation Device

[0102] 11 Sign Language Motion Image Segmentation Unit

[0103] 12 Translation Department

[0104] 13. Article Generation Department

[0105] 14 Display Control Unit

[0106] 20 Terminal devices

[0107] D1~D3 segment sign language motion image information.

[0108] F-frame image.

Claims

1. A sign language motion image translation device, comprising: The sign language motion image segmentation unit segments information related to the sign language motion images obtained by shooting sign language based on pre-set conditions, and generates time-series segmented sign language motion image information. The translation department translates the segmented sign language motion image information of the time series to generate words; The article generation department uses word strings obtained by periodically summarizing the words in the time series generated by the translation department to generate articles; as well as The display control unit updates and displays articles periodically generated by the article generation unit.

2. The sign language motion image translation device according to claim 1, wherein, The article generation unit generates articles by inputting the word strings and sign language grammar rules into a large-scale language model.

3. The sign language motion image translation device according to claim 1, wherein, When the specified word and the words before or after it are a pre-defined combination, the article generation unit converts the specified word and the words before or after it into other pre-defined words.

4. The sign language motion image translation device according to claim 1, wherein, The sign language motion image segmentation unit acquires feature points extracted from the motion image obtained by capturing the sign language as information related to the sign language motion image, and uses the feature points to generate the segmented sign language motion image information.

5. The sign language motion image translation device according to claim 4, wherein, The sign language motion image segmentation unit uses the velocity information of the feature points to generate the segmented sign language motion image information.

6. The sign language motion image translation device according to claim 4 or 5, wherein, The sign language motion image segmentation unit uses the physical information of the person performing the sign language to standardize the coordinate values ​​of the feature points.

7. A method for translating motion images of sign language, wherein, Based on pre-defined conditions, information related to sign language motion images obtained from filming sign language is segmented to generate time-series segmented sign language motion image information. Words are generated by translating the segmented sign language motion image information generated from the time series. Articles are generated by periodically summarizing the words in the generated time series. The article, which is generated periodically, is updated and displayed sequentially.

8. A computer program product comprising a sign language motion image translation program, the sign language motion image translation program causing a computer to perform the following steps: Based on pre-defined conditions, information related to sign language motion images obtained from filming sign language is segmented to generate time-series segmented sign language motion image information. The segmented sign language motion image information generated from this time series is translated to generate words; The article is generated by periodically summarizing the words in the generated time series. The article, which is generated periodically, is updated and displayed sequentially.

Images