Sign language translation device and method and storage medium

Abstract

The invention discloses a sign language translation device and method and a storage medium. The sign language translation device comprises a controller, a first display screen, a camera, a second display screen and a microphone, and the first display screen, the camera, the second display screen and the microphone are all electrically connected to the controller; wherein the first display screen faces the deaf-mute and is used for the deaf-mute to watch information; the second display screen is arranged facing the healthy hearing person and is used for the healthy hearing person to watch information; the camera is used for collecting sign language actions of the deaf-mute, the controller processes sign language information collected by the camera so that corresponding characters can be displayed on the first display screen and the second display screen at the same time, the microphone is used for collecting voice information of a healthy listener, and the controller further processes the voice information collected by the microphone so that the voice information can be displayed on the first display screen and the second display screen at the same time. And displaying the corresponding sign language video on the second display screen.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sign language translation, and in particular to a sign language translation device, method and storage medium. BACKGROUND

[0002] With the increasing demand for barrier-free communication in society, the communication barrier between the deaf and the hearing is increasingly concerned. Sign language is the main communication method of the deaf community, while the hearing generally use spoken language for communication. The difference in language expression leads to significant communication barriers between the two parties in daily interaction.

[0003] At present, although there are some auxiliary communication devices or applications, such as simple picture-text exchange boards, speech recognition software or sign language recognition systems, these technologies often have problems such as single function, incomplete interaction or poor real-time performance. For example, some devices can only realize one-way conversion from speech to text or one-way recognition from sign language to text, lack of bidirectional real-time translation capability, and cannot meet the needs of natural communication between the deaf and the hearing in face-to-face communication.

[0004] In addition, there is a lack of integrated hardware devices designed for bidirectional communication in the prior art, which leads to the need for users to rely on multiple devices or complex operation processes during use, affecting communication efficiency and user experience. Especially in public places, educational environments or medical scenarios, the lack of efficient and intuitive sign language and speech interpretation tools seriously limits the social participation of the deaf community. Therefore, there is an urgent need for a sign language translation device that can realize real-time, bidirectional and efficient communication between the deaf and the hearing, which can accurately convert the sign language of the deaf into text for the hearing to read, and can also convert the speech of the hearing into visual sign language information for the deaf to understand, so as to truly break down the communication barrier and improve the naturalness and convenience of communication. SUMMARY

[0005] The embodiments of the present application provide a sign language translation device, method and storage medium, which can solve at least one of the above technical problems.

[0006] In a first aspect, the embodiments of the present application provide a sign language translation device, which comprises a controller, a first display screen, a camera, a second display screen and a microphone, wherein the first display screen, the camera, the second display screen and the microphone are electrically connected to the controller.

[0007] The first display screen is arranged to face the deaf-mute and is used for the deaf-mute to watch information; the second display screen is arranged to face the hearing person and is used for the hearing person to watch information; the camera is used to collect the sign language action of the deaf-mute; the controller processes the sign language information collected by the camera to display corresponding text on the first display screen and the second display screen at the same time; and the microphone is used to collect the voice information of the hearing person, and the controller further processes the voice information collected by the microphone to display corresponding sign language video on the second display screen.

[0008] In some embodiments, the first display screen and the second display screen are arranged at an included angle.

[0009] In some embodiments, the first display screen and the second display screen are rotatably connected.

[0010] In some embodiments, the first display screen and the second display screen are both transparent display screens.

[0011] In some embodiments, the camera is rotatably arranged on the top of the first display screen.

[0012] In some embodiments, the sign language translation device further comprises a loudspeaker, and the controller is electrically connected to the loudspeaker and controls the loudspeaker to broadcast according to the text displayed on the second display screen.

[0013] In some embodiments, the second display screen is a touch screen.

[0014] In some embodiments, the controller comprises:

[0015] A sign language recognition module, which is used to extract key points and recognize sign language semantics from the video stream collected by the camera;

[0016] A voice recognition module, which is used to convert the voice collected by the microphone into text;

[0017] A semantic recognition module, which is used to perform context analysis and semantic completion on the voice recognition result;

[0018] A text-to-speech module, which is used to synthesize natural speech from text information and output through the loudspeaker;

[0019] A virtual semantic generation module, which is used to synthesize a three-dimensional sign language animation video in accordance with Chinese sign language grammar rules in real time according to the text content.

[0020] In a second aspect, the embodiments of the present application further provide a sign language translation method, which is applied to the sign language translation device of any of the above embodiments, and the method comprises:

[0021] Collecting the sign language action of the deaf-mute collected by the camera;

[0022] The sign language information collected by the camera is processed to display corresponding text on the first display screen and the second display screen.

[0023] The microphone is used to collect voice information of the hearing person.

[0024] The voice information collected by the microphone is processed to display corresponding sign language video on the second display screen.

[0025] In a third aspect, the embodiments of the present application further provide a computer readable storage medium, which stores program codes. The program codes can be invoked by a processor to execute the sign language translation method.

[0026] In the sign language translation device provided by the embodiments of the present application, the sign language translation device includes a controller, a first display screen, a camera, a second display screen and a microphone. The first display screen, the camera, the second display screen and the microphone are electrically connected to the controller. The first display screen is arranged to face the deaf-mute and is used to display information for the deaf-mute. The second display screen is arranged to face the hearing person and is used to display information for the hearing person. The camera is used to collect the sign language actions of the deaf-mute. The controller processes the sign language information collected by the camera to display corresponding text on the first display screen and the second display screen simultaneously. The microphone is used to collect voice information of the hearing person. The controller further processes the voice information collected by the microphone to display corresponding sign language video on the second display screen. Thus, compared with the sign language translation device in the related art, the sign language translation device in the present application integrates the camera, the microphone, the controller and the double display screen structure. The sign language translation device can collect the sign language actions of the deaf-mute in real time, accurately recognize and convert the sign language actions into text information, and display the text information on the second display screen facing the hearing person, so that the hearing person can understand the expression content of the deaf-mute immediately. At the same time, the sign language translation device collects the voice information of the hearing person through the microphone, processes the voice information through the controller, and plays corresponding sign language video on the first display screen facing the deaf-mute, so that the deaf-mute can understand the speech of the other party in an intuitive visual way. The first display screen and the second display screen face different user groups respectively, which optimizes the human-computer interaction layout and improves the privacy and pertinence of information transmission. The whole system completes the bidirectional conversion of voice and sign language in one hardware platform, avoids the complexity of multi-device collaborative operation, and improves the communication efficiency and user experience. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0028] Figure 1 A structural schematic diagram of a sign language translation device provided by an embodiment of the present application.

[0029] Figure 2 A flowchart of a sign language translation method provided by an embodiment of the present application.

[0030] Explanation of reference numerals:

[0031] 10, sign language translation device; 100, controller; 200, first display screen; 300, camera; 400, second display screen; 500, microphone; 600, speaker.

[0032] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present application more clear, the following will further describe the embodiments of the present application in combination with the accompanying drawings.

[0034] The following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all the embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0035] In the description of the present application, it should be understood that the terms "first", "second" and the like are used only for the purpose of description and should not be understood as indicating or implying relative importance. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" means two or more. "And / or", the association between the associated objects, means that there can be three kinds of relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents that the associated objects before and after are a kind of "or" relationship.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0037] Please refer to Figure 1The embodiment of the application provides a sign language translation device 10, which comprises a controller 100, a first display screen 200, a camera 300, a second display screen 400 and a microphone 500, and the first display screen 200, the camera 300, the second display screen 400 and the microphone 500 are electrically connected to the controller 100; wherein the first display screen 200 is arranged to face a deaf-mute and is used for the deaf-mute to watch information; the second display screen 400 is arranged to face a hearing person and is used for the hearing person to watch information; the camera 300 is used for collecting sign language actions of the deaf-mute, the controller 100 processes sign language information collected by the camera 300, so as to simultaneously display corresponding text on the first display screen 200 and the second display screen 400, and the microphone 500 is used for collecting voice information of the hearing person, and the controller 100 further processes the voice information collected by the microphone 500, so as to display corresponding sign language video on the second display screen 400.

[0038] Therefore, compared with the sign language translation device 10 in the related art, the sign language translation device 10 in the application integrates the camera 300, the microphone 500, the controller 100 and the double-display screen structure, can collect sign language actions of the deaf-mute in real time, accurately recognize and convert the sign language actions into text information, and synchronously display the text information on the second display screen 400 facing the hearing person, so that the hearing person can immediately understand the expression content of the deaf-mute. Meanwhile, the sign language translation device 10 collects voice information of the hearing person through the microphone 500, processes the voice information through the controller 100, and plays corresponding sign language video on the first display screen 200 facing the deaf-mute, so that the deaf-mute can understand the speech of the other party in an intuitive visual manner. The first display screen 200 and the second display screen 400 face different user groups respectively, optimize the human-computer interaction layout, and improve the privacy and pertinence of information transmission. The whole system completes the bidirectional conversion of voice and sign language in one hardware platform, avoids the complexity of multi-device collaborative operation, and improves the communication efficiency and user experience.

[0039] In addition, the double-display screen design is adopted, the first display screen 200 faces the deaf-mute, and the second display screen 400 faces the hearing person, so that the role separation and information directional output in the physical space are realized. The text information is synchronously displayed on the double screens, so that the expression content is convenient for both parties to confirm; the sign language video is played only on the second display screen 400 on the hearing person side, so that the deaf-mute can receive information in the mother tongue form, the perception habits of different users are met, and the interaction intuitiveness and accuracy are enhanced.

[0040] Furthermore, all signal processing is completed in the local controller 100, without relying on network transmission and cloud computing, so that the delay and privacy leakage risk caused by data uploading are avoided. The full offline operation mode guarantees stable work in a network-free environment, greatly reduces the end-to-end response time, and realizes low-delay interaction close to natural conversation.

[0041] In some embodiments, the first display screen 200 and the second display screen 400 are arranged at an angle.

[0042] In some embodiments, the first display screen 200 and the second display screen 400 are arranged at an angle. In some embodiments, the first display screen 200 and the second display screen 400 are arranged at an angle. In some embodiments, the first display screen 200 and the second display screen 400 are arranged at an angle. In some embodiments, the first display screen 200 and the second display screen 400 are arranged at an angle. In some embodiments, the first display screen 200 and the second display screen 400 are arranged at an angle. In some embodiments, the first display screen 200 and the second display screen 400 are arranged at an angle. In some embodiments, the first display screen 200 and the second display screen 400 are arranged at an angle.

[0043] In some embodiments, the first display screen 200 and the second display screen 400 are rotatably connected.

[0044] In some embodiments, the first display screen 200 and the second display screen 400 are rotatably connected. In some embodiments, the first display screen 200 and the second display screen 400 are rotatably connected. In some embodiments, the first display screen 200 and the second display screen 400 are rotatably connected. In some embodiments, the first display screen 200 and the second display screen 400 are rotatably connected. In some embodiments, the first display screen 200 and the second display screen 400 are rotatably connected. In some embodiments, the first display screen 200 and the second display screen 400 are rotatably connected.

[0045] In some embodiments, the first display screen 200 and the second display screen 400 are transparent display screens.

[0046] The first display screen 200 and the second display screen 400 both adopt transparent display technology, which allows ambient light to penetrate while presenting image information, maintaining visual transparency. The transparent display screen is a transparent OLED or LCD structure, with high contrast, high response speed and low power consumption characteristics. The first display screen 200 faces the deaf-mute and is used to play the standard sign language video converted from voice information, and the second display screen 400 faces the hearing person and is used to display the text content recognized from the sign language action. In the configuration that both screens are transparent, the deaf-mute and the hearing person can directly observe each other's facial expressions, mouth changes and body language through the screen, maintaining natural eye contact and emotional interaction, and avoiding the sense of communication alienation caused by the blocking of the transparent display technology. The driving circuit and pixel array of the transparent display screen adopt miniaturized design to reduce the impact on the light transmittance, ensuring clear image display while the background scenery is not distorted.

[0047] In some embodiments, the camera 300 is rotatably arranged at the top of the first display screen 200.

[0048] The camera 300 is arranged at the top edge of the first display screen 200 and is rotatably connected to the display screen through a rotating holder. The holder is composed of a driving motor, a transmission assembly and an angle sensor, supporting flexible adjustment of the camera 300 in horizontal and vertical directions. This design enables the camera 300 to dynamically adjust the shooting angle according to the user's position, adapting to deaf-mute users of different heights, sitting or standing postures, and ensuring that the sign language action is always within the optimal shooting range.

[0049] The horizontal rotation range of the camera 300 reaches ±120°, and the pitch adjustment angle is ±60°, covering a wide field of view. In actual use, when the deaf-mute is located at different positions in front of the device, the camera 300 can adjust the direction through automatic or manual mode to avoid missing gesture information due to angle deviation. The rotation process of the holder is smooth and continuous, with fast response speed, which can complete angle calibration in a short time, ensuring the continuity and smoothness of interaction.

[0050] The camera 300 integrates automatic hand tracking function, with built-in image processing algorithm to analyze video stream in real time, recognize key point positions of both hands, and actively drive the holder to rotate according to the hand movement trajectory, realizing continuous locking and tracking of dynamic sign language action. This function effectively solves the problem of missing gestures of traditional fixed-angle cameras 300 and the need for strict user alignment, significantly improving the usability and robustness of the system.

[0051] The camera 300 adopts a wide-angle lens design with a field of view angle not less than 100°, and cooperates with a high-frame-rate CMOS image sensor to clearly capture a sequence of fast-changing sign language actions. The lens surface is provided with an anti-glare coating to reduce environmental light interference and improve the imaging quality in strong light or low-illumination conditions. The camera 300 housing adopts an integrated sealing structure, which has dustproof, anti-shake and anti-interference capabilities to ensure stable operation in various public environments.

[0052] The camera 300 is integrated with the first display screen 200, so that the deaf-mute can directly observe the screen feedback when performing sign language and confirm whether he / she is in the effective shooting area. This design optimizes the human-computer interaction path and reduces the use threshold. The camera 300 is connected to the controller 100 through a high-speed data interface to transmit video signals in real time, support low-delay sign language recognition by a local AI model, and provide high-quality input guarantee for realizing natural and smooth bidirectional translation.

[0053] In some embodiments, the sign language translation device 10 further comprises a loudspeaker 600, and the controller 100 is electrically connected to the loudspeaker 600 to control the loudspeaker 600 to broadcast the text displayed on the second display screen 400.

[0054] In some embodiments, the sign language translation device 10 further comprises a loudspeaker 600, and the controller 100 is electrically connected to the loudspeaker 600 to control the loudspeaker 600 to broadcast the text displayed on the second display screen 400.

[0055] After the controller 100 completes the recognition of the sign language actions, the generated text information is input into a text-to-speech module. The text-to-speech module adopts a lightweight speech synthesis algorithm to convert the text into natural and fluent speech signals in real time. The speech signals are transmitted to the loudspeaker 600 after being processed by an audio processing unit for noise reduction, gain and timbre optimization. The speech output content is strictly synchronized with the text information displayed on the second display screen 400, so that the hearing person can receive information through the auditory and visual channels to improve the understanding accuracy.

[0056] The loudspeaker 600 supports volume adjustment function, and the user can set the output volume level through physical buttons or touch interface. The device is provided with a mute switch, allowing the user to manually turn off the voice broadcast function according to the use scene. This design is suitable for libraries, hospital clinics and other environments that need to maintain quietness, and takes into account the communication efficiency and acoustic order of public space. The loudspeaker 600 uses a full-frequency sound unit, with a frequency response range of 300Hz to 3400Hz, meeting the requirements of the main frequency band of human voice, ensuring clear and distinguishable speech.

[0057] In some embodiments, the loudspeaker 600 integrates a voice feedback mechanism, which can emit a prompt sound when the system state changes, such as device startup, successful identification, low battery, etc. This function enhances the instant feedback capability of human-computer interaction and improves the operation intuitiveness. The loudspeaker 600 shell is provided with a dustproof and breathable mesh cover to prevent foreign matter from entering and affecting the sound performance, and also has certain water splash resistance, suitable for various use environments.

[0058] The loudspeaker 600 works with the controller 100 to form a complete voice output channel, so that the device can not only display text, but also restore the expression intention of the deaf-mute in spoken form. This design makes up for the limitations of pure visual information transmission, and is especially suitable for hearing users with limited vision or reading difficulties, further expanding the application population and service scenarios of the device.

[0059] In some embodiments, the second display screen 400 is a touch screen.

[0060] Among them, the second display screen 400 adopts capacitive or resistive touch technology, integrates a touch sensor layer and a display panel to form an integrated touch display screen. The touch screen supports multi-point touch operation and has high sensitivity and fast response characteristics, and can accurately identify gestures such as clicking, sliding and long pressing of the fingers or touch pen of the hearing person. The touch signal is transmitted in real time to the controller 100 through a dedicated interface, and the system performs instruction analysis and function execution.

[0061] The touch screen is used to realize the active input function of the hearing person. The hearing person can input text, select preset sentences or call common expression templates on the second display screen 400 through touch operation. The device is built-in with a common dialogue library, covering greeting, inquiry, indication, emergency help and other scenario-based sentences, and the hearing person can trigger voice broadcast and sign language video playback by clicking the corresponding icon without relying on real-time voice input. This function is particularly suitable when the environment is noisy or the hearing person cannot speak, improving the flexibility and reliability of communication.

[0062] The touch interface adopts graphical design, with clear layout and clear icon and text annotations, conforming to human-computer interaction specifications. The interface supports multi-language switching and can adapt to the language habits of users in different regions. The operation logic is simple and intuitive, and no professional training is required to quickly get started. The controller 100 uniformly schedules touch input and voice input, allowing both input methods to coexist or switch, meeting diverse interaction needs.

[0063] The touch screen surface is covered with high-strength scratch-resistant glass, with anti-fingerprint coating and anti-glare treatment to reduce the impact of environmental light reflection on visual recognition. The touch layer is designed with electromagnetic shielding to avoid signal interference with internal electronic components, ensuring stable response performance over a long period of operation. The display supports gesture calibration, allowing users to calibrate touch accuracy and adapt to different operation habits.

[0064] The introduction of touch functionality expands the interaction dimension of the device, allowing hearing individuals to not only express themselves through speech, but also to accurately convey information through touch.

[0065] In some embodiments, the controller 100 includes a sign language recognition module, a voice recognition module, a semantic recognition module, a text-to-speech module, and a virtual semantic generation module. The sign language recognition module is used to extract key points and recognize sign language semantics from the video stream captured by the camera 300. The voice recognition module is used to convert the voice collected by the microphone 500 into text. The semantic recognition module is used to perform context analysis and semantic completion on the voice recognition result. The text-to-speech module is used to synthesize natural speech from text information and output it through the speaker 600. The virtual semantic generation module is used to synthesize three-dimensional sign language animation videos that conform to Chinese sign language grammar rules in real time based on text content.

[0066] The controller 100 is the core processing unit of the sign language translation device 10, using an embedded system architecture, equipped with high-performance processor chips, and integrating neural network processing units (NPUs) or graphics processing units (GPUs) to support high-speed inference of local artificial intelligence models. The controller 100 runs a lightweight operating system internally, coordinating and managing data collection, processing, and output processes of hardware modules such as the camera 300, microphone 500, dual display screens, and speaker 600, achieving full offline operation without relying on external network connections, ensuring data privacy and system response speed.

[0067] The controller 100 is built-in with a sign language recognition module, which is constructed based on a deep learning model and uses a spatio-temporal sequence analysis algorithm to process the video stream collected by the camera 300. The model extracts human key points through a convolutional neural network (CNN) and a graph convolutional network (GCN) or a Transformer structure, focusing on recognizing hand gestures, arm movement trajectories, facial expressions, and mouth shape changes. The sign language recognition module models continuous action sequences and, in combination with a Chinese sign language dictionary database, converts dynamic gestures into corresponding semantic text, supporting accurate recognition of common vocabulary, phrases, and simple sentence patterns.

[0068] The speech recognition module receives voice signals collected by the microphone 500 and uses an end-to-end automatic speech recognition technology to convert analog audio into digital text. This module integrates noise reduction algorithms and acoustic models and can work stably under different environmental noise conditions. The semantic understanding module performs natural language processing on the received text information, analyzes the context, identifies the speaker's intent, performs semantic completion and ambiguity resolution, and improves the completeness and accuracy of language expression, avoiding misinterpretation due to colloquial omissions or dialect differences.

[0069] The text-to-speech module synthesizes the processed text information into natural and fluent speech output. This module uses a lightweight speech synthesis model, supports multiple voice tones and speech speed adjustments, and generates speech that is played through the loudspeaker 600 for hearing people to listen to. The speech output is strictly synchronized with the text display on the second display screen 400, achieving integrated audio-visual information transmission and enhancing understanding effectiveness.

[0070] The virtual sign language generation module drives a three-dimensional virtual character model based on text content and generates sign language animation videos that comply with Chinese sign language grammar specifications in real time. This module follows the non-hand-controlled characteristics of sign language, synchronously controls gesture actions, facial expressions, mouth shape changes, and body postures, ensuring that the output sign language expression is natural, accurate, and understandable. The generated videos are played smoothly on the first display screen 200, and deaf people understand the expression content of hearing people by watching the animation. All AI models have been optimized and compressed to adapt to embedded platform operation, ensuring low latency and high frame rate for real-time interaction experience.

[0071] In some embodiments, the controller 100 of the sign language translation device 10 is configured to support multiple interaction modes, including AI unattended mode, local attended mode, and remote attended mode. The controller 100 switches between different interaction modes based on real-time detection of user interaction status, environmental context information, or manual instructions issued by the user, achieving adaptive response to diverse use scenarios.

[0072] In the AI unattended mode, the controller 100 starts the built-in lightweight artificial intelligence question-answering model. The camera 300 continuously captures the sign language actions of the deaf-mute user, and the sign language recognition module parses the gesture, expression and lip shape information in the video stream into semantic text. The semantic recognition module performs context understanding and intent analysis on the text to determine the type of question raised by the user. The controller 100 calls the AI question-answering model to generate a natural language reply text that conforms to the context. The virtual semantic generation module receives the reply text and drives the three-dimensional virtual character model to generate an animation sequence that conforms to the grammar structure of Chinese sign language. The sign language animation is played to the deaf-mute user through the first display screen 200, completing the automatic answering process. The second display screen 400 synchronously displays the corresponding text content, facilitating onlookers to understand the interaction status. This mode is suitable for unattended scenarios such as government self-service terminals, hospital guide desks and library service stations, and realizes 7x24-hour uninterrupted barrier-free service.

[0073] In the local attended mode, the hearing operation personnel are located in front of the second display screen 400 and participate in face-to-face communication. The camera 300 captures the sign language actions of the deaf-mute user, and the controller 100 displays the translation results in text form on the second display screen 400 after sign language recognition and semantic processing. After the hearing operation personnel read the text content, they can respond through voice input or touch operation. The microphone 500 picks up the voice signal, and the voice recognition module converts it into text; if touch input is used, the text is directly generated through a preset sentence library or soft keyboard. After semantic optimization, the text information is converted into three-dimensional sign language animation by the virtual semantic generation module and played on the first display screen 200 for the deaf-mute user to watch and understand. The first display screen 200 and the second display screen 400 adopt an included angle or rotatable structure design to ensure that both parties maintain natural line of sight communication. The configuration of the transparent display screen allows users to observe the facial expressions of the opposite party through the screen, maintaining emotional connection. This mode is suitable for interactive scenarios that require human intervention, such as bank counters, hospital consultations and government service windows.

[0074] In the remote manned mode, the controller 100 establishes a safe and stable network connection through the integrated communication module. The communication module supports wireless communication protocols such as Wi-Fi, 4G / 5G, etc., and can transmit the high-definition sign language video stream collected by the camera 300 to the remote server or designated terminal device in real time. The remote terminal is operated by a translator or a field expert (such as a doctor or a lawyer) with professional sign language skills. The remote personnel understand the expression content of the deaf-mute user by watching the real-time video and send a reply in the form of voice or text. The controller 100 receives the remote reply data, and the voice information is converted into text by the voice recognition module, and the text information is directly entered into the processing flow. The text content is synthesized into standard sign language animation by the virtual semantic generation module and played on the first display screen 200. At the same time, the remote voice can be output through the loudspeaker 600, and the text information is displayed on the second display screen 400 for reference by the local hearing person. This mode is suitable for complex medical consultation, legal assistance, emergency alarm and other occasions that require high-precision translation and professional judgment.

[0075] The mode switching module built-in the controller 100 is responsible for managing the conversion logic of the interaction mode. The system is started by default or automatically enters the AI unmanned mode after a long time of no interaction. When the microphone 500 detects continuous voice input or the second display screen 400 receives touch control operation, the mode switching module determines that the local personnel is present and automatically switches to the local manned mode. The user can actively initiate a remote connection request through physical buttons, voice commands or touch menus, triggering the system to enter the remote manned mode. The controller 100 can also judge the complexity of the problem based on the semantic analysis result, and when professional terms or emergency keywords are recognized, it actively prompts the user whether to call a remote expert. After the remote conversation ends, the system automatically returns to the AI unmanned mode. All mode switching processes are accompanied by visual or vibration prompts to ensure that the user is informed. This multi-mode adaptive mechanism significantly improves the intelligent level, application range and user experience of the sign language translation device 10.

[0076] Please refer to Figure 2 The embodiment of the application also provides a sign language translation method, which can be applied to the above-mentioned sign language translation device 10. The sign language translation method comprises steps 101, 102, 103 and 104.

[0077] Step 101: Obtain the sign language action of the deaf-mute person collected by the camera 300.

[0078] The controller 100 acquires the sign language action performed by the deaf-mute through the camera 300 installed on the device. The camera 300 is located at the top edge of the first display screen 200, adopts a wide-angle lens design, and has a field of view angle of not less than 100°, which can cover the activity range of the user's hands and upper body. The camera 300 continuously collects image data at a rate of 30 frames per second or higher, forming a continuous video stream, and ensuring complete capture of dynamic gestures.

[0079] The camera 300 is equipped with an automatic focusing and low-light enhancement function, which can generate clear images under different environmental light conditions. The image sensor combines digital signal processing technology to complete real-time exposure adjustment, white balance correction and motion deblurring processing, improving the quality of the input video. The camera 300 is connected to the controller 100 through a high-speed serial interface, and stably transmits the original video data to the processing unit, with a delay controlled at the millisecond level.

[0080] After the controller 100 receives the video stream, it starts the preprocessing process. The image frames successively undergo noise suppression, background segmentation and hand region extraction operations, highlighting the key action area. The system uses a deep learning-driven hand detection model to quickly locate the position of the hands in the picture and establish a tracking trajectory, providing basic data for subsequent key point recognition.

[0081] This step realizes efficient acquisition and preliminary processing of sign language actions, and builds a complete input channel from physical gestures to digital signals. High-quality video input provides a reliable guarantee for accurate analysis of the sign language recognition module and is a prerequisite for achieving high-precision translation. The entire acquisition process does not require the user to actively trigger, supporting a natural interaction mode of doing and recognizing, significantly improving the convenience of use.

[0082] Step 102: Process the sign language information of the deaf-mute collected by the camera 300 to display corresponding text on the first display screen 200 and the second display screen 400.

[0083] The controller 100 performs semantic recognition processing on the received video stream, converting the sign language action of the deaf-mute into readable natural language text. The sign language recognition module calls a deep learning model deployed locally, which is built based on a spatiotemporal feature extraction network, combining a 3D convolutional neural network (3D-CNN) with a graph convolutional network (GCN) or a Transformer architecture, and can capture both the spatial configuration and the temporal evolution law of gestures. The model input is a sequence of continuous video frames, and the output is the corresponding sign language semantic label.

[0084] The system extracts human key point coordinates from the pre-processed image sequence, focusing on analyzing the positions of double-hand joints, the orientation of the palm, the bending state of the fingers, and the movement trajectory of the arms. At the same time, the model synchronously captures non-hand-controlled features, including facial expression changes, mouth shape actions, and head poses, and makes a comprehensive judgment in combination with the context. The recognition process uses a sliding window mechanism to segment the dynamic action sequence, matches the standard expression units in the Chinese sign language dictionary, and realizes dual recognition at the word level and the syntax level.

[0085] After the recognition module outputs the preliminary semantic results, the semantic understanding unit performs context completion and ambiguity resolution. Based on a lightweight natural language processing algorithm, this unit analyzes the sentence structure, restores omitted components, and corrects misrecognition caused by incomplete actions or environmental interference. The optimized text information has good grammatical integrity and semantic accuracy, conforming to the habits of natural language expression.

[0086] The generated text information is transmitted synchronously to the first display screen 200 and the second display screen 400 for visual presentation. The first display screen 200 is located on the side of the deaf-mute, used to display their own expression content, supporting self-confirmation and expression correction; the second display screen 400 is located on the side of the hearing person, enabling the hearing person to read the translation results in real time and understand the intentions of the other party. The dual-screen synchronous display mechanism ensures that both parties have consistent cognition of the communication content, forms a closed-loop feedback, and improves the reliability of communication.

[0087] The text display format is optimized for human-computer interaction, using large font and high-contrast color schemes to adapt to reading needs under different visual conditions. The system supports multi-language switching function, which can select display languages such as Chinese and English according to the use scene. The entire processing process is completed in the local controller 100, with a response time controlled within 300 milliseconds, realizing a real-time translation experience close to natural conversation rhythm.

[0088] Step 103: Obtain the microphone 500 for collecting the voice information of the hearing person.

[0089] The system collects the voice information of the hearing person through the microphone 500 integrated on the device. The microphone 500 adopts a multi-unit array structure, distributed on the front side or top area of the device shell, forming a directional pickup array. The array structure supports beamforming technology, which can dynamically focus on the direction where the hearing person is located, enhancing the reception strength of the target voice signal while suppressing environmental noise interference from other directions.

[0090] The microphone 500 has a wide frequency response characteristic, and the effective pickup frequency range covers 80Hz to 16kHz, which can completely capture the main frequency band of human voice. The built-in preamplifier and analog-to-digital conversion module can convert analog sound wave signals into digital audio streams in real time, with a sampling rate of not less than 16kHz and a quantization accuracy of more than 16 bits, ensuring high-fidelity restoration of voice data. The audio signal is transmitted to the controller 100 through a dedicated audio bus, and the transmission process adopts an anti-electromagnetic interference design to ensure data integrity.

[0091] After the controller 100 receives the audio data, it starts the voice preprocessing process. The system performs automatic gain control to adjust the volume to an appropriate level; applies frequency domain filtering algorithm to remove background noise, air conditioner sound, equipment running sound and other non-voice components; identifies the start and end positions of the effective voice segment through voice endpoint detection technology, separates the silent interval, and improves the subsequent recognition efficiency. The preprocessed audio stream provides a clear and stable input source for voice recognition.

[0092] This step realizes high-quality collection of language expression of people with normal hearing, and builds a complete input channel from acoustic signal to digital information. The cooperation of array microphone 500 and noise reduction algorithm ensures stable pickup of voice content in complex acoustic environments such as hospitals, government offices and transportation hubs. The entire collection process does not need to be triggered by keys, supports continuous voice input in natural conversation mode, and improves the fluency of interaction and user experience.

[0093] Step 104: Process the voice information collected by the microphone 500 to display the corresponding sign language video on the second display screen 400.

[0094] The controller 100 performs semantic analysis and format conversion on the voice information collected by the microphone 500 to generate instruction data that can drive sign language animation. The voice recognition module receives the preprocessed digital audio stream and calls the locally deployed lightweight automatic speech recognition model to convert the voice signal into corresponding text content. The model is built based on deep neural network and supports continuous speech recognition, which can accurately recognize words, phrases and simple sentences in daily communication, and adapt to different pronunciation habits and speech speed changes.

[0095] The text output result is transmitted to the semantic understanding module for context analysis and semantic optimization. The semantic understanding module combines the dialogue scene and language logic to complete, correct and structure the recognized text, eliminating redundant, omitted or ambiguous expressions in spoken language, ensuring that the generated semantic information is complete and accurate. The processed text is used as the input source for virtual sign language generation and enters the animation synthesis stage.

[0096] The virtual sign language generation module drives the three-dimensional virtual character model to generate an animation video that conforms to the grammar specifications of Chinese sign language (CSL) according to the optimized text content. The module has a built-in sign language action library that covers basic gestures, word order rules, non-hand-controlled features (such as facial expressions, lip shapes, and eyebrow movements), and spatial grammar structures. The system decomposes the text into semantic units, matches corresponding gesture action units, and arranges sequences according to the temporal and spatial organization rules of sign language to generate natural and fluent sign language expressions.

[0097] The three-dimensional animation is rendered and played in real time on the first display screen 200. The rendering process uses hardware acceleration technology to ensure that the video frame rate is stable at 25 fps or above, and the motion transition is smooth without lag. The sign language video contains complete body movements, facial expressions, and lip shape changes, faithfully reproducing the sign language expression methods used daily by the deaf and mute community, improving the accuracy and understandability of information transmission. During playback, the system supports pause, playback, and speech speed adjustment functions to facilitate the deaf and mute to repeatedly confirm key information.

[0098] The processing flow is completed locally on the device without relying on external networks or cloud services, ensuring data privacy and response speed. The end-to-end delay from voice input to sign language video output is controlled within 300 milliseconds, achieving real-time interaction close to natural conversation rhythm. The entire mechanism establishes a complete output channel from spoken language to sign language, enabling the language of hearing people to be accurately presented in the form of the deaf and mute mother tongue, and truly realizing two-way equal and barrier-free communication.

[0099] The computer readable storage medium provided by the embodiments of the present application has program codes stored therein, and the program codes can be called and executed by a processor to execute the methods described in the above embodiments.

[0100] The computer readable storage medium can be an electronic storage such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer readable storage medium includes a non-volatile computer readable storage medium. The computer readable storage medium has a storage space for program codes for executing any of the method steps described above. The program codes can be read from or written into one or more computer program products. The program codes can be compressed in an appropriate form, for example.

[0101] The same or similar reference numerals in the drawings of the embodiments correspond to the same or similar components; in the description of the present application, it is understood that if the orientations or positional relationships indicated by the terms "upper", "lower", "left", "right" and the like are based on the orientations or positional relationships shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationships in the drawings are only used for exemplary illustration and cannot be understood as a limitation on the present application, for those skilled in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0102] The above is only a preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A sign language translation device, characterized in that, It includes a controller, a first display screen, a camera, a second display screen, and a microphone, wherein the first display screen, the camera, the second display screen, and the microphone are all electrically connected to the controller; The first display screen is oriented towards deaf-mute individuals for viewing information; the second display screen is oriented towards hearing individuals for viewing information; the camera is used to capture the sign language gestures of the deaf-mute individuals; the controller processes the sign language information captured by the camera to display corresponding text simultaneously on the first and second display screens; the microphone is used to capture the voice information of the hearing individuals; the controller also processes the voice information captured by the microphone to display corresponding sign language video on the second display screen.

2. The sign language translation device according to claim 1, characterized in that, The first display screen and the second display screen are set at an angle.

3. The sign language translation device according to claim 2, characterized in that, The first display screen and the second display screen are rotatably connected.

4. The sign language translation device according to claim 1, characterized in that, Both the first and second displays are transparent displays.

5. The sign language translation device according to claim 1, characterized in that, The camera is rotatably mounted on top of the first display screen.

6. The sign language translation device according to claim 1, characterized in that, The sign language translation device also includes a speaker, and the controller is electrically connected to the speaker. The controller controls the speaker to play text according to the text displayed on the second display screen.

7. The sign language translation device according to claim 1, characterized in that, The second display screen is a touch screen.

8. The sign language translation device according to claim 7, characterized in that, The controller includes: The sign language recognition module is used to extract key points from the video stream captured by the camera and recognize sign language semantics; A speech recognition module, which is used to convert the speech collected by the microphone into text; A semantic recognition module, which is used to perform contextual analysis and semantic completion on the speech recognition results; A text-to-speech module, which is used to synthesize text information into natural speech and output it through a speaker; A virtual semantic generation module is used to synthesize a three-dimensional sign language animation video that conforms to the grammatical rules of Chinese sign language in real time based on the text content.

9. A sign language translation method, characterized in that, The method, applied to the sign language translation device according to any one of claims 1 to 9, comprises: Acquire the sign language gestures of the deaf-mute person captured by the camera; The sign language information of the deaf and mute person captured by the camera is processed to display the corresponding text on the first display screen and the second display screen; The microphone is used to collect voice information from a hearing person; The voice information collected by the microphone is processed to display the corresponding sign language video on the second display screen.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code, which can be called by a processor to execute the sign language translation method according to claim 9.

Images