Real-time sign language expression method and system based on native action model

Abstract

The embodiment of the invention provides a real-time sign language expression method and system based on a native action model, and is applied to the technical field of digital data processing. The method comprises the following steps: acquiring a character identity label and a corresponding audio stream from an audio / video in real time; converting the audio stream into text data; performing natural language processing on the text data to obtain a standardized word sequence; converting the standardized word sequence into a structured sign language sequence conforming to sign language expression habits; converting the structured sign language sequence into sign language action data with the character style based on a pre-trained personalized sign language model corresponding to a character identity identifier; and controlling a virtual human body model to make corresponding sign language actions in real time by using the sign language action data, and fusing the sign language actions with the characters in the original video. In this way, seamless conversion from the speech content of the target person to the sign language expression of the target person can be achieved, and therefore the sense of reality of watching and emotional connection are greatly enhanced.

Description

Technical Field

[0001] This disclosure relates to the field of digital data processing technology, and in particular to a real-time sign language expression method and system based on native motion models. Background Technology

[0002] With the increasing demand for information accessibility in society, how to provide high-quality, immersive film and television and live broadcast content for the hearing impaired has become the focus of the industry. The current mainstream solutions can be roughly divided into three categories: (1) External subtitles or pop-up text. Although they are simple to implement, the hearing impaired audience must frequently switch their eyes, resulting in a fragmented reading experience and difficulty in capturing the emotions and context of the actors. (2) "Digital human sign language translation" based on a universal sign language dictionary, which involves inserting a computer-generated virtual human in the corner of the screen to translate sign language. (3) Human simultaneous interpreters appearing on screen. Although this is closest to real communication, it is costly, limited in the number of sessions, and may obscure the main subject of the screen, making it difficult to promote on a large scale.

[0003] Therefore, the industry urgently needs a technical solution that is "real-time, low-latency, high-fidelity, and personalized" so that hearing-impaired people can directly see the actors themselves expressing themselves in natural and fluent sign language, thereby significantly improving information acquisition efficiency and emotional resonance. Summary of the Invention

[0004] This disclosure provides a real-time sign language expression method and system based on native motion models, which solves the unfamiliarity brought by third-party digital humans and avoids the technical problem of the "unnatural valley" effect that may exist in digital humans.

[0005] According to a first aspect of this disclosure, a real-time sign language expression method based on a native action model is provided. The method includes: Real-time acquisition of personal identification and corresponding audio stream from audio and video recordings; Convert the audio stream into text data; Natural language processing is performed on the text data to obtain a standardized word sequence; The standardized word sequence is converted into a structured sign language sequence that conforms to the habits of sign language expression; Based on a pre-trained personalized sign language model corresponding to a person's identity identifier, the structured sign language sequence is converted into sign language action data with the person's style. The virtual human model is controlled in real time to perform corresponding sign language actions using the sign language data, and then merged with the character in the original video.

[0006] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the personalized sign language model based on the pre-trained identity identifier is trained through the following steps: Acquire all aspects of the target person’s fine hand movements, arm trajectories, and body postures when expressing different structured sign language sequences; A dataset is constructed based on the structured sign language sequence and the obtained raw data. The dataset is divided into a training set, a validation set, and a test set. The training set was used to train a personalized sign language model based on the Transformer model architecture to obtain a personalized sign language model specific to the person's identity.

[0007] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the step of using the sign language motion data to control a virtual human model to perform corresponding sign language actions in real time and merging it with the character in the original video includes: The sign language motion data is used to drive a virtual human body model to generate hand movements; The generated hand gestures are used to replace the corresponding hand gestures of the people in the original video, resulting in a naturally blended output video.

[0008] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the method further includes: Adjust the shadow and reflection effects of the virtual human model based on the lighting information from the original video.

[0009] In addition to the aspects and any possible implementations described above, a further implementation is provided in which adjusting the shadow and reflection effects of the virtual sign language gestures based on the lighting information of the original video includes: Extract lighting information from the original video frames; Based on the lighting information, shadows and reflections are calculated to make the lighting of the virtual human body model consistent with the lighting of the real scene.

[0010] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the method further includes background restoration of the background obscured by the hand movements of the corresponding person in the original video, specifically comprising the following steps: From the first N frames of the video stream, select non-interference frames in which the proportion of actor occlusion of the scene background meets the preset low occlusion condition, and generate a background baseline map. The spatial alignment parameters between the current frame and the background reference map are obtained by a feature point matching algorithm, and a background supplement block that matches the background transparency area is extracted from the background reference map. The background supplement block is superimposed onto the transparent area of ​​the current frame background, and an edge blending algorithm is used to process the splicing edges to eliminate visual discontinuity.

[0011] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the method further includes: Sign language gesture data, emotion data, and timestamps are encoded into a lightweight metadata stream and then transmitted. Before rendering each frame of video, the corresponding metadata stream is extracted based on the current timestamp, and the pre-stored virtual human model is loaded. The virtual human model is driven to generate sign language actions based on the metadata stream, and then integrated and displayed with the video stream in real time.

[0012] According to a second aspect of this disclosure, a real-time sign language expression system based on a native action model is provided. The system includes: The acquisition module is used to obtain the person's identity identifier and corresponding audio stream from the audio and video in real time; A conversion module is used to convert the audio stream into text data; The analysis module is used to perform natural language processing on the text data to obtain a standardized word sequence; The translation module is used to convert the standardized word sequence into a structured sign language sequence that conforms to the expression habits of sign language; The generation module is used to convert the structured sign language sequence into sign language action data with the style of the person based on the pre-trained personalized sign language model corresponding to the person's identity identifier. The synthesis module is used to control the virtual human model to perform corresponding sign language actions in real time using the sign language action data, and to merge it with the character in the original video.

[0013] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0014] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods according to the first and / or second aspects of this disclosure.

[0015] This disclosure obtains the character's identity and corresponding audio stream from audio and video in real time and converts it into text data. Then, it uses Natural Language Processing (NLP) to obtain a standardized word sequence, and then uses a neural machine translation model to convert it into a structured sequence that conforms to sign language habits. Combined with a pre-trained personalized sign language model, it generates sign language movement data with the character's style. Finally, it integrates the hand movements of the virtual human model with the original video stream in real time, achieving a seamless transformation of the actor's own sign language expression. This not only ensures the personalization and realism of the sign language movements, enhancing the emotional connection and immersion for hearing-impaired viewers, but also improves interactivity and reduces server pressure. It effectively avoids the "unnatural valley" effect brought by third-party digital humans, making the sign language expression and the actor's performance naturally integrated, and significantly improving the viewing experience.

[0016] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A flowchart is shown for a real-time sign language expression method based on a native motion model according to an embodiment of the present disclosure; Figure 2 A flowchart illustrating the process of obtaining a pre-trained personalized sign language model based on a person's identity identifier, according to an embodiment of the present disclosure, is shown. Figure 3 A block diagram of a real-time sign language expression system based on a native motion model according to an embodiment of the present disclosure is shown; Figure 4 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0019] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0020] This disclosure obtains the character's identity and corresponding audio stream from audio and video in real time and converts it into text data. Then, it uses Natural Language Processing (NLP) to obtain a standardized word sequence, and then uses a neural machine translation model to convert it into a structured sequence that conforms to sign language habits. Combined with a pre-trained personalized sign language model, it generates sign language movement data with the character's style. Finally, it integrates the hand movements of the virtual human model with the original video stream in real time, achieving a seamless transformation of the actor's own sign language expression. This not only ensures the personalization and realism of the sign language movements, enhancing the emotional connection and immersion for hearing-impaired viewers, but also improves interactivity and reduces server pressure. It effectively avoids the "unnatural valley" effect brought by third-party digital humans, making the sign language expression and the actor's performance naturally integrated, and significantly improving the viewing experience.

[0021] Figure 1 A flowchart of a real-time sign language expression method 100 based on a native motion model according to an embodiment of the present disclosure is shown. Figure 1 As shown, method 100 includes: S110 can obtain the person's identity identifier and corresponding audio stream in real time from audio and video.

[0022] In some embodiments, a multi-channel audio and video acquisition device is used to capture the target person's identity identifier and corresponding audio stream in real time. Based on the identity recognition result, the spatial location of the target person is determined. A spatial audio separation algorithm is used in combination with semantic segmentation to obtain a person region mask. The target person's individual audio stream is separated from the mixed audio stream. Other people's voices and environmental noise are filtered out. An identity ID tag is added to the separated target person's audio stream, and the timestamp sequence of the audio stream is recorded, so that the audio stream is accurately associated with the target person's identity.

[0023] S120 converts the audio stream into text data.

[0024] In some embodiments, the preprocessed audio frames are fed into the streaming ASR engine in 20ms packets. After feature extraction and language model processing, the results are output, and the final text fragment is output when a complete semantic unit (such as a comma, period, or a pause of more than 300ms) is detected.

[0025] In some embodiments, the raw text output by ASR is subsequently processed as follows: Remove colloquial filler words; Consistent capitalization and standardized punctuation; Timestamp alignment: Each text segment is stamped with a timestamp (millisecond level) that precisely corresponds to the original audio and video frame, ensuring that subsequent sign language actions are synchronized with the audio stream content.

[0026] S130 performs natural language processing on the text data to obtain a standardized word sequence.

[0027] In some embodiments, the text data obtained from speech recognition is first thoroughly cleaned, and continuous sentences are segmented into independent lexical units using a word segmentation algorithm to ensure that the boundaries of each word are clearly distinguishable. Subsequently, the segmented text is standardized, including: Remove punctuation marks, special characters, and extra spaces from the text to avoid irrelevant symbols interfering with subsequent processing; Further filter and remove stop words from the text, such as common function words and modal words, words with no actual meaning or weakened meaning, to streamline the text content; Perform word form restoration and stem extraction to transform the text into a refined and standardized word sequence; The core meaning and context of the text are extracted through advanced semantic analysis, and sign language semantics are generated.

[0028] S140 converts standardized word sequences into structured sign language sequences that conform to sign language expression habits.

[0029] In some embodiments, a neural machine translation model is used to reorganize the word order and structure of a standardized word sequence to obtain a structured sign language sequence that conforms to the expression habits of sign language. The neural machine translation model is obtained through the following steps: To acquire data for specific sign language grammar rules; The dataset is labeled and constructed, and the dataset is divided into training set, validation set and test set; The neural machine translation model to be trained is trained using the training set to obtain the trained neural machine translation model.

[0030] In some embodiments, for a target sign language (such as a common sign language of a specific country or region), data containing its unique grammatical rules is collected. This data needs to cover the core features of the sign language, such as spatial logic (e.g., using hand position changes to reflect the spatial relationship of things), action sequence (e.g., the expression habit of describing the subject first and then the action), and semantic association, to ensure that the data can fully reflect the grammatical system of the sign language. The collected data is meticulously labeled: the natural language text is accurately associated with the corresponding structured sign language expression, clarifying the meaning of each word and phrase in the natural language, as well as its corresponding action combination, execution order, and spatial parameters in the sign language. The labeled data is then divided into training set, validation set, and test set according to a preset ratio. The training set is used to train the neural machine translation model to be trained, resulting in the trained neural machine translation model (NMT).

[0031] Finally, the standardized word sequence is input into the trained NMT model. Based on the learned sign language grammar rules, the model will automatically complete word order adjustment and structural reconstruction, and output a structured sign language sequence containing information such as the execution order of each sign language action, spatial location parameters, and emotional expression markers (such as the intensity of emotion reflected by the amplitude and speed of hand movements). This provides a basic framework that conforms to the sign language expression habits for subsequent conversion into personalized sign language action data for actors.

[0032] S150, based on a pre-trained personalized sign language model corresponding to a person's identity, converts structured sign language sequences into sign language action data with the person's style.

[0033] like Figure 2 As shown, the personalized sign language model pre-trained based on the person's identity is obtained through the following steps: S151, acquire all aspects of the target person’s fine hand movements, arm trajectories, and body postures when expressing different structured sign language sequences.

[0034] In some embodiments, high-precision inertial sensors and optical tracking systems are used to perform large-scale, all-around motion capture of the target person. The capture range includes not only fine hand movements (such as the bending angle of finger joints and the trajectory of hand rotation) and the movement path of the arm, but also body postures (such as the torso tilt angle and the range of shoulder movement) when expressing different words and phrases, ensuring comprehensive recording of the actor's movement details and stylistic characteristics in various semantic and emotional scenarios.

[0035] S152, a dataset is constructed based on the structured sign language sequence and the obtained raw data. The dataset is divided into a training set, a validation set, and a test set.

[0036] In some embodiments, the neural machine translation (NMT) module outputs diverse structured sign language sequences covering three major semantic scenarios to ensure data generalization: Basic semantic sequence: contains sign language structures corresponding to commonly used words (e.g., "hello" - "hello / gesture"); Complex sentence sequences: containing sign language structures corresponding to everyday short sentences (such as "today I went to the park" - "today / park / go", which conforms to the sign language word order of "time + place + action"); Emotional association sequence: includes sign language structures corresponding to 5 core emotions (happiness, anger, sadness, surprise, and calmness) (e.g., “get into university” → “university / get into / happy”, labeled with the emotion tag “happy-level 4”).

[0037] Based on the action data of the target task and the structured sign language sequence, a dataset with a one-to-one correspondence between the structured sign language sequence and the target person's action data is obtained. A random hierarchical partitioning strategy is adopted to ensure that the semantic and emotional distribution of each dataset is consistent with the original data, and to avoid the model evaluation distortion caused by partitioning bias.

[0038] In some embodiments, data cleaning is performed using specialized tools to remove potential noise and inaccurate movements, ensuring data quality. To address interference that may be introduced during sensor acquisition, filtering algorithms (such as Kalman filtering) are used to smooth the motion trajectory data and eliminate outliers exceeding reasonable ranges. Simultaneously, time-series consistency checks are performed to remove broken or repetitive motion segments caused by temporary equipment malfunctions. Human assistance is used to verify the motion data by combining labeled text and semantic information.

[0039] Specifically, when the text corresponding to a certain action data is "goodbye", but the action trajectory deviates too much from the common "goodbye" sign language action (such as the hand not completing the waving gesture), it is judged as an inaccurate action and is removed; for atypical actions caused by the actor's fatigue or mistakes (such as action pauses or abnormal amplitude), by comparing multiple action records under the same semantics, samples that conform to the actor's regular expression habits are retained to ensure that the remaining data can truly reflect his stable action style.

[0040] S153, using the training set to train a personalized sign language model based on the Transformer model architecture, to obtain a personalized sign language model specific to the person's identity.

[0041] In some embodiments, a model is built based on the Transformer architecture, taking the training set as input and the corresponding target actor's motion data as output. The model learns the mapping relationship between the two through backpropagation. During training, the model focuses on capturing details of the actor's movement style, such as habitual gesture amplitude for specific words and rhythmic features of body language during emotional expression. Simultaneously, the deviation between the model's output motion sequence and the real data is monitored in real time using a validation set, dynamically adjusting hyperparameters such as the learning rate and the number of network layers to avoid overfitting or underfitting.

[0042] In some embodiments, the trained model is independently evaluated using a test set to calculate core metrics such as motion reproduction accuracy (e.g., joint coordinate error, motion trajectory overlap) and emotion matching accuracy. When these metrics reach a preset threshold (e.g., motion reproduction accuracy ≥ 95%), the model training is considered complete, resulting in a personalized sign language model specific to the character's identity. At this point, the model can convert the input structured sign language sequence (containing information such as motion sequence and spatial logic) into sign language motion data that highly matches the style of the target character actor, including the precise coordinates and rotation angles of the hands, arms, and key joints of the body in three-dimensional space, providing a personalized foundation for subsequent virtual motion generation.

[0043] The S160 uses sign language motion data to control a virtual human model to perform corresponding sign language actions in real time, and then merges them with the characters in the original video.

[0044] In some embodiments, sign language motion data is used to drive a virtual human model to generate hand movements; The generated hand gestures are used to replace the corresponding hand gestures of the people in the original video, resulting in a naturally blended output video.

[0045] Specifically, the three-dimensional spatial coordinates and rotation angles of key joints of the hand and fingers (such as the palm base, knuckles, and fingertips) are extracted from the data. The joint posture of the model is updated in real time through the skeletal binding system, so that the virtual human model can accurately reproduce the details of the actor's sign language movements (such as the degree of finger bending, the angle of palm rotation, and the trajectory of arm movement).

[0046] In some embodiments, the shadow and reflection effects of the virtual human model are adjusted based on the lighting information of the original video.

[0047] In some embodiments, lighting information is extracted from the original video frames; shadows and reflections are calculated based on the lighting information to make the lighting of the virtual human model consistent with the lighting of the real scene.

[0048] In some embodiments, a semantic segmentation algorithm is first used to identify the actor's body regions (especially the real hand and arm positions) and background elements (such as tables, chairs, walls, etc.) in the original video frames, generating an accurate foreground mask; simultaneously, the depth information of the video frames is calculated to establish the spatial hierarchy of the scene. Based on the above information, when rendering the virtual human body model, the actor's real hand regions in the original video are made transparent to avoid visual conflicts between the virtual human body model and real limbs.

[0049] In some embodiments, the lighting information of the original video stream is analyzed in real time, including the direction of the light source (e.g., strong light from the left), intensity (e.g., bright environment or dim scene), and color tone (e.g., warm light, cool light). Based on this information, matching shadow effects (e.g., when the light source shines from the upper left, the virtual human model produces a corresponding shadow on the lower right) and light reflections (e.g., the highlight position of metallic nails changes with the light source) are calculated and rendered for the virtual human model to ensure that the lighting and shadow performance of the virtual human model is completely consistent with the original video environment, avoiding a sense of "floating" or "digital prosthetics" and other incongruities.

[0050] In some embodiments, background restoration is performed on the background obscured by the hand movements of the corresponding person in the original video. The specific steps are as follows: From the first N frames of the video stream, select non-interference frames in which the proportion of actor occlusion of the scene background meets the preset low occlusion condition, and generate a background baseline map. The spatial alignment parameters between the current frame and the background reference map are obtained through a feature point matching algorithm, and background supplementary blocks that match the background transparent area are extracted from the background reference map. The background supplement block is overlaid onto the transparent area of ​​the current frame background, and an edge blending algorithm is used to process the stitching edges to eliminate visual discontinuity.

[0051] In some embodiments, the first N frames of the video stream are selected, and the scene background region and the target person region in each frame are extracted by a semantic segmentation algorithm. The proportion of the target person region to the scene background region is calculated. A preset low occlusion condition is set as the target person occluding the background by less than 30%. Frames that meet this condition are selected as interference-free frames, that is, frames in which the actor's body occludes the background less and the background details are complete. At least 3 interference-free frames are retained to ensure sufficient background information. For the selected interference-free frames, a multi-frame mean fusion + noise reduction optimization strategy is used to generate a background baseline map.

[0052] In some embodiments, a semantic segmentation algorithm is used to identify the background region (i.e., the transparent background region) occluded by the target person's hand movements in the current frame. The boundary coordinates (e.g., top-left corner coordinates (x1, y1), bottom-right corner coordinates (x2, y2)) and area of ​​this region are marked by pixel transparency detection (judging regions with pixel alpha channel values ​​< 255). An ORB (Oriented Fast and Rotated BRIEF) feature point matching algorithm is used to obtain the spatial alignment parameters between the current frame and the background reference image: ORB feature points are extracted from the non-transparent background region of the current frame and the background reference image (500-800 feature points are extracted per frame to ensure coverage of key background details, such as wall textures and furniture edges). The feature points are matched using a FLANN (Fast Library for Approximate Nearest Neighbors) matcher, and incorrect matches are eliminated. Based on the matched feature point pairs, a global transformation matrix (including translation, rotation angle, and scaling) is calculated between the current frame and the background reference image to ensure that the transformed background reference image is completely aligned with the background spatial position of the current frame (alignment error ≤ 2 pixels).

[0053] In some embodiments, the extracted background supplement block is directly superimposed onto the transparent region of the current frame according to its coordinates, covering the transparent pixels. During the superposition process, the alpha channel value of the background supplement block is kept at 255 (completely opaque) to ensure that there is no residual transparency in the supplemented background. The Poisson blending algorithm is used to smooth the gradient at the stitching edges between the supplement block and the current frame background, making the repaired background visually completely coherent. The Canny algorithm is used to perform edge detection on the fused current frame. If the gradient value of the detected stitching edge is ≤10 (the smaller the gradient value, the smoother the edge), the repair is considered qualified; if the gradient value is >10, the edge width or smoothing parameters of the Poisson blending are readjusted until the visually seamless requirement is met.

[0054] In some embodiments, during live streaming or real-time playback, the system processes and renders each frame of video with a millisecond-level response time, ensuring that the movements of the virtual human model are completely synchronized with the actor's lip movements and body postures in time. In the final output video, the virtual human model blends naturally with the actor's original image, visually presenting the effect of the target person using sign language. This preserves the target person's real expression while conveying information to hearing-impaired viewers through virtual sign language movements, enhancing the realism and immersion of the viewing experience.

[0055] In some embodiments, to further enhance the completeness and realism of sign language expression, professional 3D scanning technology (such as photogrammetry) can be used to perform high-precision scans of the actor's entire body and face, thereby obtaining their realistic geometric shape and skin texture data. The raw data obtained from the scan is then imported into 3D modeling software (such as Blender) for optimization processing, constructing a 3D digital double that is highly consistent with the actor's appearance. This digital double has a built-in complex digital skeletal system and facial binding system, which can accurately respond to movement and facial expression commands.

[0056] The numerical data stream, including joint coordinates, rotation angles, and facial expression parameters, generated during the motion synthesis stage, is directly input into the skeletal and facial rigging system of the 3D digital avatar. Based on these numerical instructions, the real-time rendering engine drives the digital avatar to accurately reproduce every gesture, finger bending detail, and subtle facial muscle twitches, achieving precise synchronization between full-body sign language expression and facial expressions.

[0057] In some embodiments, sign language gesture data, emotion data, and timestamps are encoded into a lightweight metadata stream and transmitted. Before rendering each frame of video, the corresponding metadata stream is extracted based on the current timestamp, and the pre-stored virtual human model is loaded. The virtual human model is driven to generate sign language actions based on the metadata stream, and then integrated and displayed with the video stream in real time.

[0058] Specifically, the sign language motion data (including the 3D coordinates and rotation angles of key hand and body joints), emotional data (facial expression parameters matching sign language expressions, such as eyebrow and mouth shape control values), and timestamp information (accurate to milliseconds, used for synchronizing video frames) generated during the motion synthesis stage are first integrated and encoded. A dedicated compression algorithm converts this data into a lightweight metadata stream, significantly reducing data size while ensuring data integrity and transmission efficiency. During transmission, either the RTMP protocol's metadata channel or HTTP Live Streaming (HLS)'s in-band metadata channel is used to synchronously encapsulate the encoded metadata stream with the actor's original video stream and transmit it in real time, ensuring strict alignment of the two on the timeline.

[0059] In some embodiments, the following operations are performed during video playback: Timestamp synchronization: Before rendering each video frame, the timestamp of the current video frame is automatically queried, and based on the timestamp, the sign language gesture instructions, emotional parameters and other information at the corresponding moment are accurately extracted from the metadata stream to ensure that the time synchronization error between virtual actions and video content is controlled within milliseconds.

[0060] Model loading and driving: Load the pre-stored personalized sign language model that matches the target person's movement style, input the extracted sign language movement data into the skeletal driving system of the virtual human model, update the hand joint posture in real time, and generate sign language movements consistent with the metadata instructions (such as finger bending, arm swing trajectory, etc.); at the same time, the facial expression parameters of the virtual model can be adjusted according to the emotional data to achieve the coordination of expression and movement.

[0061] Environment Adaptation and Integration: The AR engine combines the original video lighting information (direction, intensity, color) contained in the metadata stream to calculate and render matching shadows and light reflection effects for the virtual human model, ensuring that the virtual human model and the lighting environment of the real video scene are consistent. At the same time, based on the foreground occlusion information in the metadata, the real hand area of ​​the target person in the original video is made transparent, preserving the virtual human model in the foreground display, and the background occluded by the corresponding person's hand movements in the original video is repaired through background restoration technology.

[0062] Final presentation: A motion-driven, environment-adapted virtual human model is superimposed in real time onto the currently playing video frame to form a naturally integrated picture, which is displayed on the user's device (such as a mobile phone or computer) screen, achieving the visual effect of "the actor himself giving sign language in real time" when the audience watches.

[0063] According to the embodiments of this disclosure, hearing-impaired individuals can directly watch the sign language communication of the actors themselves, enhancing the realism, emotional connection, and immersion of the viewing experience. This avoids the "unnatural valley" effect brought by third-party digital humans and improves the naturalness and human touch of the communication. At the same time, the personalized model ensures the high fidelity and personal style of the sign language movements, while client-side rendering reduces server pressure and improves the interactivity of the live broadcast, bringing a better viewing experience to hearing-impaired individuals.

[0064] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0065] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0066] Figure 3 A block diagram of a real-time sign language expression system 300 based on an actor's native motion model, according to an embodiment of the present disclosure, is shown. Figure 3 As shown, system 300 includes: The acquisition module 301 is used to acquire the person's identity identifier and corresponding audio stream from the audio and video in real time; Conversion module 302 is used to convert audio streams into text data; Analysis module 303 is used to perform natural language processing on text data to obtain standardized word sequences; Translation module 304 is used to convert standardized word sequences into structured sign language sequences that conform to sign language expression habits; The generation module 305 is used to convert structured sign language sequences into sign language action data with the person's style based on the pre-trained personalized sign language model corresponding to the person's identity identifier. The compositing module 306 is used to control the virtual human body model to perform corresponding sign language actions in real time using sign language motion data, and to merge it with the person in the original video.

[0067] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0068] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0069] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0070] Figure 4 A schematic block diagram of an electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0071] Electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in ROM 402 or a computer program loaded into RAM 403 from storage unit 408. RAM 403 can also store various programs and data required for the operation of electronic device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. I / O interface 405 is also connected to bus 404.

[0072] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of displays, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0073] Computing unit 401 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 401 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by computing unit 401, one or more steps of method 100 described above may be performed. Alternatively, in other embodiments, computing unit 401 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0074] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0075] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0076] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0077] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0078] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0079] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0080] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0081] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A real-time sign language expression method based on a native action model, comprising: Real-time acquisition of personal identification and corresponding audio stream from audio and video recordings; Convert the audio stream into text data; Natural language processing is performed on the text data to obtain a standardized word sequence; The standardized word sequence is converted into a structured sign language sequence that conforms to the habits of sign language expression; Based on a pre-trained personalized sign language model corresponding to a person's identity identifier, the structured sign language sequence is converted into sign language action data with the person's style. The virtual human model is controlled in real time to perform corresponding sign language actions using the sign language data, and then merged with the character in the original video.

2. The method according to claim 1, characterized in that, The personalized sign language model, pre-trained based on the person's identity identifier, is trained through the following steps: Acquire all aspects of the target person’s fine hand movements, arm trajectories, and body postures when expressing different structured sign language sequences; A dataset is constructed based on the structured sign language sequence and the obtained raw data. The dataset is divided into a training set, a validation set, and a test set. The training set was used to train a personalized sign language model based on the Transformer model architecture to obtain a personalized sign language model specific to the person's identity.

3. The method according to claim 1, characterized in that, The step of using the sign language motion data to control the virtual human model to perform corresponding sign language movements in real time and merging it with the character in the original video includes: The sign language motion data is used to drive a virtual human body model to generate hand movements; The generated hand gestures are used to replace the corresponding hand gestures of the people in the original video, resulting in a naturally blended output video.

4. The method according to claim 3, characterized in that, The method further includes: Adjust the shadow and reflection effects of the virtual sign language gestures based on the lighting information of the original video.

5. The method according to claim 4, characterized in that, The adjustment of shadow and reflection effects of virtual sign language movements based on the lighting information of the original video includes: Extract lighting information from the original video frames; Based on the lighting information, shadows and reflections are calculated to make the lighting of the virtual human body model consistent with the lighting of the real scene.

6. The method according to claim 3, characterized in that, The method also includes background restoration for the background obscured by the hand movements of the corresponding person in the original video. The specific steps are as follows: From the first N frames of the video stream, select non-interference frames in which the proportion of actor occlusion of the scene background meets the preset low occlusion condition, and generate a background baseline map. The spatial alignment parameters between the current frame and the background reference map are obtained by a feature point matching algorithm, and a background supplement block that matches the background transparency area is extracted from the background reference map. The background supplement block is superimposed onto the transparent area of ​​the current frame background, and an edge blending algorithm is used to process the splicing edges to eliminate visual discontinuity.

7. The method according to claim 1, characterized in that, The method further includes: Sign language gesture data, emotion data, and timestamps are encoded into a lightweight metadata stream and then transmitted. Before rendering each frame of video, the corresponding metadata stream is extracted based on the current timestamp, and the pre-stored virtual human model is loaded. The virtual human model is driven to generate sign language actions based on the metadata stream, and then integrated and displayed with the video stream in real time.

8. A real-time sign language expression system based on a native action model, comprising: The acquisition module is used to obtain the person's identity identifier and corresponding audio stream from the audio and video in real time; A conversion module is used to convert the audio stream into text data; The analysis module is used to perform natural language processing on the text data to obtain a standardized word sequence; The translation module is used to convert the standardized word sequence into a structured sign language sequence that conforms to the expression habits of sign language; The generation module is used to convert the structured sign language sequence into sign language action data with the style of the person based on the pre-trained personalized sign language model corresponding to the person's identity identifier. The synthesis module is used to control the virtual human model to perform corresponding sign language actions in real time using the sign language action data, and to merge it with the character in the original video.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images