A barrier-free sign language communication system

Abstract

The present invention discloses a barrier-free sign language communication system, including a sign language image acquisition module, a sign language recognition module, a display module and a database, wherein the database stores the standard meaning data of the sign language, the sign language image acquisition module is used to acquire the sign language image, and transmit the acquired data to the sign language recognition module, the sign language recognition module includes a feature extraction module, a semantic analysis module, and a text processing module, the feature extraction module extracts the finger joint part features in the sign language image, and sends the extracted data to the semantic analysis module, the semantic analysis module classifies the extracted finger joint feature parts according to the standard meaning data of the sign language in the database, and outputs the sign language meaning. The present invention recognizes gestures through the sign language recognition module, and adopts reinforcement learning to understand different semantics in the same gesture, can accurately express the body language of the deaf-mute, well improve the communication of the deaf-mute, and is relatively simple in operation.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a barrier-free sign language communication system. Background Art

[0002] Human-computer interaction is one of the major scientific technologies today, which includes virtual reality technology and intelligent perception technology. Intelligent perception technology is of great significance as a means of interaction between machines and humans. However, the current intelligent perception technology is relatively rigid, especially in terms of gesture language, which cannot achieve accurate recognition, causing great inconvenience for the deaf and mute in human-computer interaction. Summary of the invention

[0003] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a barrier-free sign language communication system. The sign language recognition module is used to recognize gestures, and reinforcement learning is used to understand the different semantics in the same gesture. The body language of the deaf-mute can be accurately expressed, which greatly improves the communication of the deaf-mute and is relatively simple to operate.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0005] The present invention discloses a barrier-free sign language communication system, comprising a sign language image acquisition module, a sign language recognition module, a display module and a database, wherein the database stores standard sign language meaning data, the sign language image acquisition module is used to acquire sign language images, and transmit the acquired data to the sign language recognition module, the sign language recognition module comprises a feature extraction module, a semantic analysis module and a text processing module, the feature extraction module extracts finger joint part features in the sign language image, and sends the extracted data to the semantic analysis module, the semantic analysis module classifies the extracted finger joint feature parts according to the standard sign language meaning data in the database, outputs the sign language meaning, and sends the sign language meaning to the text processing module, the text processing module uses a machine learning model to output the meaning of the sign language meaning in the text, and sends the data to the display module, and the display module displays the meaning of the sign language meaning in the text.

[0006] As a preferred technical solution of the present invention, the sign language image acquisition module uses a high-definition camera to acquire images, and the high-definition camera collects the image of each frame. The semantic analysis module performs semantic analysis on the image of each frame, and images that cannot be semantically recognized are automatically deleted.

[0007] As a preferred technical solution of the present invention, the text processing module uses a reinforcement learning model to process data into strategies, states, and strategy evaluation values, takes the meaning of the sign language in the text as the strategy, and takes the text as the state. If the meaning of the output sign language in the text is consistent with the meaning of the strategy itself, the strategy evaluation value of the strategy is increased; if the meaning of the output sign language in the text is inconsistent with the meaning of the strategy itself, the strategy evaluation value of the strategy is reduced.

[0008] As a preferred technical solution of the present invention, the feature extraction module adopts a residual convolutional neural network to perform feature extraction, and the residual convolutional neural network adopts a network structure in the form of a convolution unit that is convolved to batch normalized to activate the Relu function.

[0009] Compared with the prior art, the present invention has the following beneficial effects:

[0010] The present invention recognizes gestures through a sign language recognition module and uses reinforcement learning to understand different semantics in the same gesture. It can accurately express the body language of the deaf-mute, greatly improve the communication of the deaf-mute, and the operation is relatively simple. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0012] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0013] In the figure: 1. Sign language image acquisition module; 2. Sign language recognition module; 3. Display module; 4. Database; 5. Feature extraction module; 6. Semantic analysis module; 7. Text processing module. DETAILED DESCRIPTION

[0014] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0015] Furthermore, if a detailed description of known techniques is not necessary for illustrating the characteristics of the present invention, it will be omitted.

[0016] Example 1

[0017] like Figure 1As shown, the present invention provides a barrier-free sign language communication system, including a sign language image acquisition module 1, a sign language recognition module 2, a display module 3 and a database 4, wherein the database 4 stores standard sign language meaning data, the sign language image acquisition module 1 is used to acquire sign language images, and transmit the acquired data to the sign language recognition module 2, the sign language recognition module 2 includes a feature extraction module 5, a semantic analysis module 6, and a text processing module 7, the feature extraction module 5 extracts features of finger joint parts in the sign language image, and sends the extracted data to the semantic analysis module 6, the semantic analysis module 6 classifies the extracted finger joint feature parts according to the sign language standard meaning data in the database 4, outputs the sign language meaning, and sends the sign language meaning to the text processing module 7, the text processing module 7 uses a machine learning model to output the meaning of the sign language meaning in the text, and sends the data to the display module 3, and the display module 3 displays the meaning of the sign language meaning in the text.

[0018] The sign language image acquisition module 1 uses a high-definition camera to acquire images. The high-definition camera collects each frame of the image. The semantic analysis module 6 performs semantic analysis on each frame of the image, and automatically deletes images that cannot recognize semantics.

[0019] The text processing module 7 uses a reinforcement learning model to process data into strategies, states, and strategy evaluation values. The meaning of the sign language in the text is used as the strategy, and the text is used as the state. If the meaning of the output sign language in the text is consistent with the original meaning, the strategy evaluation value of the strategy is increased. If the meaning of the output sign language in the text is inconsistent with the original meaning, the strategy evaluation value of the strategy is reduced.

[0020] The feature extraction module 5 uses a residual convolutional neural network to extract features. The residual convolutional neural network uses a network structure in the form of a convolution unit that is convolution-batch normalization-activated ReLU function.

[0021] Specifically, in the process of dynamic gesture recognition, the image of each frame is collected, and the collected image data is subjected to sign language recognition in turn. The feature part of each sign language is identified by the feature extraction module 5 in the sign language recognition and compared with the data in the database 4. If the corresponding semantics is not found in the database 4, the image of the frame is deleted, and the semantics in the image are connected in turn to form text. Since each semantics has multiple meanings, the reinforcement learning model is used in combination with the overall text to recognize the semantics of each gesture, thereby forming text and sending it to the display module 3 for display.

[0022] The present invention recognizes gestures through a sign language recognition module and uses reinforcement learning to understand different semantics in the same gesture. It can accurately express the body language of the deaf-mute, greatly improve the communication of the deaf-mute, and the operation is relatively simple.

[0023] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A barrier-free sign language communication system, comprising a sign language image acquisition module (1), a sign language recognition module (2), a display module (3) and a database (4), characterized in that: The database (4) stores the standard meaning data of sign language. The sign language image acquisition module (1) is used to acquire the sign language image and transmit the acquired data to the sign language recognition module (2). The sign language recognition module (2) comprises a feature extraction module (5), a semantic analysis module (6), and a text processing module (7). The feature extraction module (5) extracts the features of the finger joint part in the sign language image and transmits the extracted data to the semantic analysis module (6). The semantic analysis module (6) classifies the extracted finger joint feature parts according to the standard meaning data of sign language in the database (4), outputs the sign language meaning, and transmits the sign language to the semantic analysis module (7). The meaning is sent to a text processing module (7), the text processing module (7) uses a machine learning model to output the meaning of the sign language in the text, and sends the data to a display module (3), the display module (3) displays the meaning of the sign language in the text, the text processing module (7) uses a reinforcement learning model to process the data into strategies, states, and strategy evaluation values, takes the meaning of the sign language in the text as a strategy, and takes the text as a state. If the meaning of the output sign language in the text is consistent with the meaning itself, the strategy evaluation value of the strategy is increased; if the meaning of the output sign language in the text is inconsistent with the meaning itself, the strategy evaluation value of the strategy is reduced.

2. The barrier-free sign language communication system according to claim 1, characterized in that: The sign language image acquisition module (1) uses a high-definition camera to acquire images, and the high-definition camera collects images of each frame. The semantic analysis module (6) performs semantic analysis on the images of each frame, and automatically deletes images whose semantics cannot be recognized.

3. The barrier-free sign language communication system according to claim 1, characterized in that: The feature extraction module (5) uses a residual convolutional neural network to extract features. The residual convolutional neural network uses a network structure in the form of a convolution unit that is convoluted to batch normalized to activate a ReLU function.

Images