Sign language correction feedback based on machine learning model

Abstract

In various examples, machine learning model-based sign language correction feedback is provided. Machine learning model-based recognition of sign language symbols (e.g., body poses and / or motions) can be used to generate real-time kinematic feedback to signers to guide them to adjust their hand poses to align correctly with established sign language standard symbols. A sign language feedback framework can process video data representing conversational sign language symbol sequences to extract kinematic keypoints to search a sign language lexicon representing an established vocabulary of sign language symbols. Based on selecting an expected sign language symbol from the lexicon and computing a deviation between a kinematic keypoint pattern of the expected sign language symbol in the lexicon and the extracted kinematic keypoint symbol, the sign language feedback framework can provide real-time kinematic feedback to the gesture signer describing how to adjust their gesture signing.

Description

Background Technology

[0001] Sign language is a natural language developed to convey meaning using a visual-manual approach. Compared to spoken language, which relies on auditory signals such as sounds and verbal information, sign language uses hand shapes, movements, facial expressions, and body postures to communicate. Sign languages ​​have their own unique grammar and syntax (which may differ significantly from spoken language in the same region) and can be influenced by a variety of factors, including cultural and regional differences. They provide a means of communication for individuals with varying degrees of hearing impairment and also play a role in the cultural identity and community cohesion of these individuals and their supporting communities. Learning a particular version of sign language, such as American Sign Language, can be achieved through a variety of methods and resources. Formal courses can be offered by community colleges, universities, or various organizations, while internet-based courses and video tutorials provide learners with flexibility and accessibility. Summary of the Invention

[0002] Embodiments of this disclosure relate to sign language correction feedback based on machine learning models. Systems and methods for providing real-time kinematic feedback to sign language signifiers are disclosed to illustrate how sign language signifiers adjust their gestures.

[0003] Compared to conventional systems, embodiments of this disclosure relate to techniques for generating real-time kinematic feedback to sign language speakers (referred to herein as signifiers) using the recognition of conversational sign language symbols (e.g., body postures including hand shapes, curvatures, and / or movements) based on machine learning models. This kinematic feedback guides them to adjust their hand postures (e.g., hand and finger positions and / or movements) to correctly align with established standard sign language symbols (e.g., from authoritative sign language dictionaries). In some embodiments, the communication platform may include a sign language feedback framework comprising one or more machine learning models and processing video data representing sequences of conversational sign language symbols performed by the signifier. One or more machine learning models may be trained to perform human posture recognition (such as skeletal kinematics recognition), which extracts the relative positions of predefined kinematic keypoints (e.g., bones, joints, and / or other features) from one or more images of an individual's one or more hands, one or more arms, face, and / or torso. A sign language feedback framework can detect and / or extract the relative positions of predefined kinematic keypoints displayed by a signer from video data to construct two-dimensional kinematic keypoint symbols. These extracted kinematic keypoint symbols can provide patterns of kinematic keypoints that can be used to query a sign language dictionary (e.g., a database) representing an established vocabulary of conversational sign language symbols. The sign language feedback framework can perform a similarity algorithm that computes one or more alignment metrics that associate the extracted kinematic keypoint symbols with those that can be found in the sign language dictionary. The similarity algorithm can then select from the sign language dictionary the kinematic keypoint pattern that has the highest similarity to the extracted kinematic keypoint symbols as the expected sign language symbol for the signer. By selecting expected sign language symbols from a dictionary and calculating the deviation between the kinematic keypoint patterns of the expected sign language symbols in the dictionary and the extracted kinematic keypoint symbols, the sign language feedback framework can provide real-time kinematic feedback to sign language users (describing how to adjust their sign language gestures (e.g., how to adjust the alignment of their hands and / or fingers)) to improve their sign language gesture proficiency. Attached Figure Description

[0004] The system and method for sign language correction feedback based on a machine learning model are described in detail below with reference to the accompanying drawings, wherein:

[0005] Figure 1 This is a data flow diagram of an example process for a sign language-based communication system according to some embodiments of this disclosure;

[0006] Figure 2 This is a data flow diagram illustrating an example sign language feedback framework according to some embodiments of the present disclosure;

[0007] Figure 3A and Figure 3B This is a diagram illustrating an example of evaluating sign language communication to provide real-time kinematic feedback according to some embodiments of the present disclosure;

[0008] Figure 4 This is a diagram illustrating an example user interface of a client application used in conjunction with a sign language feedback framework according to some embodiments of the present disclosure;

[0009] Figure 5 This is a flowchart illustrating an example method for providing sign language kinematic feedback according to some embodiments of the present disclosure;

[0010] Figure 6 This is a block diagram of an example computing device applicable to implementing some embodiments of the present disclosure; and

[0011] Figure 7 This is a block diagram of an example data center applicable to implementing some embodiments of this disclosure. Detailed Implementation

[0012] Systems and methods related to sign language correction feedback based on machine learning models have been disclosed. Learning to communicate in sign language involves learning to recognize hand shapes, movements, and facial expressions, as well as developing the corresponding manual skills to reproduce hand shapes and movements with sufficient proficiency so that they can be recognized by others as conveying sign language messages. Mastering sign language skills is generally best achieved through continuous practice and exposure to the language in real-life environments. Current technology-based solutions for facilitating sign language learning typically target cloud-based tutorial platforms and / or applications where sequences of conversational sign language symbols (and / or alphabetic sign language symbols) can be presented on a display, and users attempt to learn to understand the presented sign language content, for example, based on closed-text translation. By using these techniques, users can learn to follow sign language content and develop basic skills in forming conversational sign language symbols with their own hands by imitating what they see. That is to say, users can assess their own ability to accurately reproduce conversational sign language symbols, which has a significant impact on how well they will be able to use sign language to communicate their thoughts to others. With this in mind, forms of sign language recognition techniques based on convolutional neural network (CNN) models have been proposed. However, those techniques are primarily designed for classifying handwritten images, rather than generating true translations of conversational sign language symbol sequences. Furthermore, such sign language recognition techniques, as proposed to date, are limited by the dynamic context of real-life sign language conversations, where distorted sign language symbols can lead to ambiguity or misunderstanding from the recipient's perspective.

[0013] Compared to current technologies used to support sign language communication, embodiments of this disclosure involve techniques for generating real-time kinematic feedback to sign language speakers (referred to herein as signifiers) using the recognition of conversational sign language symbols (e.g., hand shapes, body postures, curvatures, and / or movements) based on machine learning models. This kinematic feedback guides them to adjust their body postures (e.g., the position and / or movement of their hands and fingers) to correctly align with established standard sign language symbols (e.g., from authoritative sign language dictionaries). In some embodiments, they may be guided to adjust facial expressions. As further discussed herein, it should be understood that sign language symbols can include static symbols (e.g., where the posture is held in a position) and temporal symbols (e.g., those relating to movement and / or changes in posture over time).

[0014] For example, in some embodiments, the communication platform may include a sign language feedback framework that includes one or more machine learning models and processes video data representing sequences of conversational sign language symbols performed by a signifier. One or more machine learning models may be trained to perform human pose recognition (such as skeletal kinematics recognition) that extracts the relative positions of predefined kinematic keypoints (e.g., bones, joints, and / or other features) from images of an individual's hands, arms, face, and / or torso. The sign language feedback framework may detect and / or extract the relative positions of the predefined kinematic keypoints displayed by the signifier from the video data to construct two-dimensional kinematic keypoint symbols. Such extracted kinematic keypoint symbols can provide patterns of kinematic keypoints that can be used to query a sign language dictionary (e.g., a database) representing an established vocabulary of conversational sign language symbols. For example, the sign language feedback framework may perform a similarity algorithm that computes one or more alignment metrics that associate the extracted kinematic keypoint symbols with those kinematic keypoint symbols that can be found in a sign language dictionary. As mentioned above, it should be understood that in some embodiments, one or more of the kinematic keypoint symbols of conversational sign language symbols defined in the sign language dictionary may be static in nature (e.g., representing static hand and / or body postures) and / or temporal in nature (e.g., representing hand movement patterns opposite to or other than strictly static hand postures). A similarity algorithm can select the kinematic keypoint pattern with the greatest similarity to the extracted kinematic keypoint symbols from the sign language dictionary as the expected sign language symbol for the signifier. Based on the selection of the expected sign language symbol from the dictionary and the calculation of the deviation between the kinematic keypoint pattern of the expected sign language symbol from the dictionary and the extracted kinematic keypoint symbols, the sign language feedback framework can provide real-time kinematic feedback to the signifier (describing how to adjust their signification (e.g., how to adjust the alignment of their hands and / or fingers)) to improve their signification proficiency.

[0015] In some embodiments, the sign language feedback framework can generate animated kinematic feedback (e.g., real-time and / or near-real-time animated visual feedback) to the signifier by presenting, for example, an overlay of kinematic keypoint patterns or similar graphics (highlighting kinematic keypoints whose positional deviation from the dictionary-defined kinematic keypoint patterns exceeds a threshold amount). For example, the sign language feedback framework can determine the boundary shape (e.g., a bounding box) around the detected keypoints and calculate changes to determine when keypoints extracted from video data exceed a predetermined tolerance. The sign language feedback framework can also determine corrections (e.g., orientation and / or distance) that indicate how keypoints exceeding the tolerance should be adjusted to align the signifier's hand gestures to a more accurate representation of the expected sign language symbol. For example, real-time kinematic feedback can display arrows or other graphics showing how the signifier moves (e.g., adjusts one or more hands, one or more fingers, one or more arms, torso, and / or facial expressions) to better align the symbols they present with the expected sign language symbols. In some embodiments, real-time kinematic feedback can display visual correction feedback for multiple different keypoints, where keypoints out of tolerance are clearly highlighted in real time, along with instructions on how to improve their alignment. For example, in some embodiments, real-time kinematic feedback can include a kinematic keypoint pattern overlay, where keypoints out of tolerance can be easily distinguished from keypoints within tolerance, such as by different colors and / or other markers. The kinematic keypoint pattern overlay can include an animated representation of dynamically updated kinematic keypoint patterns. For example, the position of the kinematic keypoints in the pattern can be dynamically adjusted to follow the changing hand gestures of the gesturing user. In some embodiments, real-time kinematic feedback can present a kinematic keypoint pattern overlay where keypoints within tolerance are displayed using a first color (e.g., green), and keypoints out of tolerance are displayed using a second color (e.g., red). As the gesturing user adjusts their hand gestures based on the displayed graphical guidance, those keypoints that move from out of tolerance to within tolerance alignment can change color (e.g., from red to green) to provide positive reinforcement to the gesturing user that they are correctly adjusting their hand gestures. If the adjustments made by the signer cause a key point that was previously within tolerance to become outside tolerance, the color of that key point can be changed (e.g., from green to red) to provide feedback to the signer that their attempt to correct their hand posture has resulted in a further misalignment with the intended sign language symbol.

[0016] While real-time kinematic feedback to a signer is described as including graphical overlays, it should be understood that the discussion of overlays is intended to provide a non-limiting example of real-time kinematic feedback, and other forms of real-time kinematic feedback may be used in other embodiments. For example, in some embodiments, real-time kinematic feedback may include enhancements (e.g., modifications) to a modified (e.g., using a generative artificial intelligence (AI) model) real-time video feed to the signer to alter the signer's hand pose to illustrate the deviation between the kinematic keypoint pattern of the expected sign language symbol and the extracted kinematic keypoint symbol. That is, kinematic feedback may include real-time video feedback showing the signer how to adjust their hand pose using a modified image of the signer's hand, and may illustrate the deviation between their actual hand pose and a target hand pose that places their extracted keypoints within tolerance to match the expected sign language symbol selected from a dictionary. In some embodiments, real-time kinematic feedback may include enhanced and / or extended realistic representations that display a combination of generative AI hand pose modifications and keypoint pattern overlays.

[0017] In some embodiments, a sign language dictionary can be selected from multiple different sign language dictionaries to reconfigure the sign language feedback framework for different versions of sign language. That is, while American Sign Language (ASL) is the prevalent language used by hearing-impaired individuals in the United States, British Sign Language (BSL), Spanish Sign Language (SSL), Japanese Sign Language (JSL), and French Sign Language (LSF) are examples of different sign languages, each with its own vocabulary and grammatical rules. In some embodiments, a sign language dictionary for a specific sign language version can be loaded into memory and accessed by the sign language feedback framework to provide the signifier with real-time kinematic feedback corresponding to the sign language version used by the signifier. In some embodiments, the sign language feedback framework can access different sign language dictionaries from a library of available sign language dictionaries to tailor the sign language feedback framework for a specific version of sign language. In some embodiments, the sign language feedback framework can evaluate an input video feed that captures the signifier. The signifier can register their sign language user preferences in the sign language feedback framework by gesturing in their preferred sign language. The sign language feedback framework can detect the sign language version that the signifier is using (e.g., by using a trained machine learning model to infer and classify the sign language version) and use that determination to establish the sign language version preference of the signifier.

[0018] In some embodiments, a sign language feedback framework may include and / or access one or more language models (e.g., micro-language models (TLM), small language models (SLM), large language models (LLM), etc.) and / or retrieve augmented generative (RAG) artificial intelligence models. Such RAG models can be used to at least partially implement a sign language dictionary for determining kinematic keypoint patterns based on extracted kinematic keypoint symbols, which represent the expected sign language symbols of a signifier. The RAG model may access one or more data sources as authoritative knowledge to augment the training-based data sources when generating responses to input cues. Therefore, in some embodiments, the sign language feedback framework may include a RAG model that accesses data sources comprising one or more authoritative sign language dictionaries. The sign language feedback framework may input cues to the RAG model based on the extracted kinematic keypoint symbols, and in response, the RAG model may select kinematic keypoint patterns identified from one or more authoritative sign language dictionaries that are similar to the extracted kinematic keypoint symbols. In some embodiments, based on video data capturing the hand gestures of the signifier and kinematic keypoint patterns identified by the RAG model, the sign language feedback framework can generate a response indicating the deviation between the kinematic keypoint patterns of the expected sign language symbol and the kinematic keypoint symbol extracted from the signifier. The sign language feedback framework can then provide real-time kinematic feedback to the signifier describing how to adjust their hand gestures (e.g., how to adjust the alignment of their hands and / or fingers) to conform to the expected sign language symbol. In one or more embodiments, the RAG model can be configured to use a data source comprising a sign language dictionary corresponding to the sign language version used by the signifier. For example, the RAG model can select a sign language dictionary corresponding to the sign language version based on sign language user preference indications.

[0019] In some embodiments, the sign language feedback framework as described herein may be implemented at least in part as a plug-in or module software component of an application that executes locally on the computing device of the person gesturing (e.g., a user computing device such as a desktop computer, laptop, tablet, smartphone, or other computing device). As a non-limiting example, such an application may include a client application for a communication platform, a sign language training application, or a precise hand gesture training application for another skill. In some embodiments, the sign language feedback framework may be implemented at least in part as a service (e.g., a microservice) exposed from a networked cloud computing platform (e.g., at least in part hosted by a data center 700). For example, in some embodiments, one or more functions of the sign language feedback framework (such as one or more machine learning models providing sign language detection and / or translation functions and / or generating real-time kinematic feedback (e.g., overlaying and / or modifying video feeds)) may be accessed as a service by a client application using, for example, application programming interface (API) function calls, Hypertext Transfer Protocol (HTTP) control channels, and / or WebRTC sender and receiver clients for audio, video, and / or text data. In some embodiments, the sign language feedback framework functionality can be implemented as an optional service of the underlying video conferencing communication platform (e.g., as...). The functionality provided by Maxine and / or implemented as a web application by one or more servers of a communication platform. In some embodiments, the sign language feedback framework functionality can be distributed across client applications, exposed web services, and / or web applications hosted by one or more servers. For example, the client application may include a front-end of the sign language feedback framework that communicates and interacts with a user interface on the speaker's device and communicates with a back-end that includes computing hardware resources for executing machine learning models in the sign language feedback framework and / or generating real-time kinematic feedback data that is transmitted back to the front-end for presentation on the user interface. Furthermore, one or more sign language dictionaries may be hosted and accessed from one or more cloud-based computing platforms.

[0020] In some embodiments, one or more models of the sign language feedback framework can be executed by various different neural network architectures. For example, one or more models may include machine learning model architectures (e.g., one or more encoder-decoder-based machine learning models, CNN models, deep neural network (DNN) models, LLM-based models, generative AI models, etc.) trained to perform operations (such as, but not limited to, sign language symbol detection, hand pose detection, skeletal kinematics recognition, kinematic keypoint pattern detection and / or extraction) and / or to instantiate and control graphical overlays for real-time kinematic feedback. In some embodiments, the sign language feedback framework may be implemented using an artificial intelligence (AI)-based software framework (e.g., a suite of cloud-hosted AI models) (such as, but not limited to, NVIDIA's Tokkio). In some embodiments, the sign language feedback framework may include a first communication channel processing path for processing video stream feeds that capture representations of the signer captured by one or more image sensors. The sign language feedback framework may include a second communication channel processing path for processing outgoing real-time kinematic feedback data used to present to a user interface of a client application.

[0021] In some embodiments, the sign language feedback framework described herein can be implemented more generally as a posture feedback framework for other use case applications that can benefit from user learning to align a user's body posture with established standards. For example, the posture feedback framework may reference one or more sign language-based body posture dictionaries comprising one or more standardized kinematic keypoint patterns, wherein the posture feedback framework applies queries based on kinematic keypoint symbols extracted from video data to find similar (e.g., most similar) kinematic keypoint patterns and generate real-time kinematic feedback describing how the user adjusts their hand and / or body postures to conform to the expected posture. Such use cases may include body posture dictionaries associated with tasks such as playing musical instruments (e.g., correct hand positions for piano, guitar, violin, etc.), sports (e.g., proper grip and stance for tennis, golf, baseball, etc.) or other tasks involving obtaining accurate hand postures upon successful completion of the task. Such sign language and / or body posture feedback frameworks enhance the capabilities of the underlying technologies to more efficiently and effectively accomplish their task of presenting training content.

[0022] refer to Figure 1 , Figure 1This is an example data flow diagram of the process for a sign language-based communication system 100 according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or as alternatives to those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or combined with other components in any suitable combination and location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be performed by one or more processors that include processing circuitry and execute instructions stored in memory.

[0023] like Figure 1 As shown, the sign language-based communication system 100 may include one or more client devices 105 coupled to the communication platform 120 to instantiate one or more virtual communication channels. One or more functions and / or components of the sign language-based communication system 100 described herein may at least partially utilize computing devices (such as...) Figure 6 The computing device 600 shown) and / or data center (such as about Figure 7 The resources of the data center 700 shown are used for implementation. As a non-limiting example, the communication platform 120 may include conferencing services (e.g., Microsoft Teams, Zoom, Cisco Webex, GoToMeeting, etc.), cloud-based collaborative content creation platforms (e.g., NVIDIA Omniverse or other multi-user virtual environments), peer-to-peer communication systems, and / or other platforms that support real-time audio / video communication between client device 105 and one or more other user client devices 105' (e.g., which may be operated by human users and / or virtual users (such as, but not limited to, AI-based users)). In some embodiments, one or more users may communicate via their respective client devices (e.g., client device 105 and other client devices 105') (such as, but not limited to, [other platforms]). Figure 6 The corresponding client application of the computing device 600 shown in the figure accesses the communication platform 120 separately (e.g., via a network connection).

[0024] As a non-limiting example, in some embodiments, the communication platform 120 may include a collaboration platform through which client devices 105 and 105' can exchange audiovisual content in a virtual environment session 122 (e.g., a virtual meeting or conference), a virtual reality space (e.g., a virtual reality in which users, represented by avatars, can interact), or another virtual environment (e.g., NVIDIA Omniverse), hosted by the communication platform 120. Typically, when the communication platform 120 initiates a meeting (e.g., a “call”), it can establish an instance of the virtual environment session 122. The virtual environment session 122 defines a channel or other shared logical infrastructure that carries audio, video, text, and / or other forms of communication between multiple user participants who are participants in the virtual environment session 122. In such embodiments, one or more user client applications 110 for accessing the virtual environment session 122 may include, for example, standalone applications (e.g., Microsoft Teams, Apple FaceTime, or other applications), or web browser applications (e.g., Microsoft Edge, Safari, etc.) that access the virtual environment session 122 via the web server (HTTP) protocol.

[0025] like Figure 1As shown, client device 105 may include a human-machine interface 106 through which a user interacts with one or more user client applications 110. For example, HMI 106 may include one or more of a keyboard, pointing device, touchscreen, microphone, and / or other input interfaces for providing input to user client applications 110, and / or a display screen, one or more speakers, and / or other output interfaces for providing content from user client applications 110 to the user. In some embodiments, client device 105 and / or HMI 106 may include one or more cameras 108 that capture image data of the user for uplink transmission, transmitting the captured image data as uplink content data feed 115 to communication platform 120. Uplink content data feed 115 may include audio and / or video data captured by the user from user client applications 110. That is, the uplink content data feed 115 may include communication content data, which, among other data, includes video data representing sign language communication (e.g., content data generated by a user gesturing in sign language), which may be distributed, for example, to one or more other user client devices 105' via the communication platform 120. The user client application 110 may control the HMI 106 to display at least one user interface (UI) 107 to display content received from the downlink content data feed 116 obtained from other users via the communication platform 120.

[0026] User client application 110 can also control UI 107 to display a representation of kinematic feedback data 117 generated by at least one sign language feedback framework 130. The sign language feedback framework 130 may include an enhancement manager 132 based on generative artificial intelligence and one or more sign language proficiency engines 134 based on machine language models. In some embodiments, the sign language feedback framework 130 may include and / or have access to one or more body posture-based sign language dictionaries 136, which the sign language proficiency engines 134 use to evaluate sign language communications included in uplink content data feed 115.

[0027] As this article is about Figure 3A and Figure 3BIn more detail, the sign language feedback framework 130 can evaluate sign language communication provided by uplink content data feed 115 to provide kinematic feedback data 117 (e.g., animated kinematic feedback data providing visual feedback) to the signifier via UI 107, describing how to adjust their sign language gestures (e.g., how to adjust the posture of their hands and / or body) to improve their sign language proficiency. In some embodiments, the sign language feedback framework 130 can instantiate at least one sign language proficiency engine instance 134 based on video data from uplink content data feed 115 by detecting whether uplink content data feed 115 includes sign language data (and if so, which sign language is being used). Based on the detected sign language, the sign language proficiency engine instance 134 can extract kinematic keypoint location patterns from the video data to define one or more sign language body posture kinematic keypoint symbols (e.g., where sign language symbols may include static symbols that hold a posture in position and / or temporal symbols that involve movement and / or change of posture over time). The sign language proficiency engine instance 134 can calculate one or more kinematic keypoint positional deviations between one or more kinematic body posture kinematic keypoint symbols extracted from video data and standardized kinematic keypoint patterns obtained from one or more sign language dictionaries 136. An augmentation manager 132 based on generative artificial intelligence can input kinematic keypoint positional deviation data from the sign language proficiency engine instance 134 to calculate kinematic feedback data 117 representing an adjustment to one or more individual body postures. As discussed herein, the kinematic feedback data 117 can indicate an adjustment to one or more individual body postures that will align the extracted sign language body posture kinematic keypoint symbols with standardized kinematic keypoint patterns obtained from one or more sign language dictionaries 136 (e.g., based on similarity within a similarity threshold). The user client application 110 can then receive the kinematic feedback data 117 and control the user interface 107 to output a visual presentation based on the kinematic feedback data 117.

[0028] Now for reference Figure 2 , Figure 2 This is a data flow diagram illustrating an example sign language feedback framework 130 according to some embodiments of the present disclosure. (See also: Regarding...) Figure 1The sign language feedback framework 130 discussed can process sign language content within the uplink content data feed 115 to generate kinematic feedback data 117. In some embodiments, the sign language feedback framework 130 can detect or otherwise determine specific sign languages ​​appearing in the uplink content data feed 115, thereby configuring the sign language proficiency engine instance 134 and / or selecting one or more corresponding sign language dictionaries 136 that can be used by the sign language proficiency engine instance 134 (e.g., finding standardized kinematic key point patterns). In some embodiments, the sign language proficiency engine instance 134 can input instructions from sign language preference setting data 210 received from the user client application 110. For example, a user of the user client application 110 can set sign language user preferences that indicate the preferred sign languages ​​the user will use for gestural indication. The sign language proficiency engine instance 134 can input sign language user preferences and access one or more sign language dictionaries 136, which it will use to generate keypoint position deviation data (representing the deviation between kinematic keypoint symbols extracted from video data and standardized kinematic keypoint patterns obtained from one or more sign language dictionaries 136). For example, in some embodiments, sign language preference setting data 210 can indicate the selection of a preferred sign language (e.g., ASL). In this case, the sign language feedback framework 130 can initialize (e.g., instantiate) one or more sign language proficiency engine instances 134 with a target sign language pattern configuration that generates keypoint position deviation data representing the degree of conformity between the signifier and the selected preferred sign language (e.g., ASL), or representing a lack of conformity between the signifier and the selected preferred sign language.

[0029] As another example, in some embodiments, the sign language feedback framework 130 may infer sign language user preferences based on processing uplink content data feeds 115 received from the user client application 110. For example, the sign language feedback framework 130 includes a sign language detection model 224, which includes a machine learning model trained to infer when sign language is being used from video data and to classify the sign language being used. Based on the sign language detection model 224 determining when and which sign language is being used, the sign language feedback framework 130 may instantiate one or more sign language proficiency engine instances 134 (e.g., to access corresponding sign language dictionaries 136) with a target sign language pattern configuration for that preferred sign language. In some embodiments, a corresponding sign language dictionary 136 may be selected from a sign language library 236, which includes multiple sign language dictionaries 136 defining standard sign language kinematic keypoint symbols for various different sign languages. The sign language library 236 may be part of or separate from the sign language feedback framework 130.

[0030] The sign language proficiency engine instance 134 may include one or more machine learning models trained to perform human pose recognition (such as skeletal kinematics recognition) that extracts the relative positions of predefined kinematic keypoints (e.g., skeletal bones and joints and / or other features) from one or more images of an individual's one or more hands, one or more arms, face, and / or torso. The sign language proficiency engine instance 134 may detect and / or extract video data from uplink content data feed 115 that shows the relative positions of predefined kinematic keypoints displayed by the signer to establish two-dimensional kinematic keypoint symbols, such as... Figure 3A As illustrated in the diagram. For example... Figure 3A The figure illustrates an example of a two-dimensional kinematic keypoint symbol 300 based on human posture, which includes kinematic keypoints for hand postures (shown at 310), kinematic keypoints for torso postures (shown at 312), and / or kinematic keypoints for facial postures (shown at 314). A sign language proficiency engine instance 134 can extract one or more keypoints of such human postures to determine a global two-dimensional kinematic keypoint symbol that can indicate a pattern conveying sign language-based information. As mentioned herein, the two-dimensional kinematic keypoint symbol determined by the sign language proficiency engine instance 134 can be a static symbol (e.g., a static human posture determined from an image frame), and / or the two-dimensional kinematic keypoint symbol determined by the sign language proficiency engine instance 134 can be a temporal symbol (e.g., a moving human posture determined from a sequence of image frames). The kinematic keypoint symbols extracted from the captured human posture can be used as a query by the sign language proficiency engine instance 134 to search the sign language dictionary 136 to find a corresponding standard version of the sign language symbol. This standard version can be used as the basis for comparison with the extracted kinematic keypoint symbols to generate kinematic keypoint position deviation data. In some embodiments, the sign language proficiency engine instance 134 can perform the search of the sign language dictionary 136 based on performing a similarity algorithm. For example, the similarity algorithm can select kinematic keypoint patterns from the sign language dictionary 136 that have similarity (e.g., within a similarity threshold) to the extracted kinematic keypoint symbols and define them as representing the expected sign language symbol of the person gesturing. Based on the expected sign language symbol determined from the sign language dictionary 136, the sign language proficiency engine instance 134 calculates the deviation between the kinematic keypoint pattern of the expected sign language symbol and the extracted kinematic keypoint symbol. In some embodiments of the sign language dictionary 136 that include multiple variations of sign language symbols, the similarity algorithm can select a kinematic keypoint pattern from the sign language dictionary 136 that has the closest similarity to the obviously expected sign language symbol of the signer, given the extracted kinematic keypoint symbol.

[0031] In some embodiments, the similarity algorithm may incorporate contextual information to search the sign language dictionary 136 to determine the expected sign language symbol for the person gesturing. For example, even where sign language symbols can be defined based on hand and / or torso posture, the similarity algorithm may utilize facial key points to classify facial expressions (e.g., happiness, sadness, anger, confusion, etc.) to help identify the expected sign language symbol and its corresponding normalized kinematic key point patterns from the sign language dictionary 136. For example, when a facial expression conveying happiness is detected, this detection can help the similarity algorithm narrow its search to sign language symbols associated with happiness. In some embodiments, contextual information may include detected changes in facial features (e.g., puffed-out cheeks). For example, when a facial expression conveying happiness is detected, this detection can help the similarity algorithm narrow its search to sign language symbols associated with happiness.

[0032] In some embodiments, the use of contextual information by the similarity algorithm may be sensitive to cultural differences. For example, the sign language dictionary 136 may include culturally relevant facial expression variations that can be used to help identify expected sign language symbols. Furthermore, in some embodiments, the sign language proficiency engine instance 134 may monitor body language and emotions (e.g., as indicated by detected body postures and / or facial expressions) and provide feedback to the signifier (e.g., to UI 107) when the observed body language and emotions, given extracted kinematic keypoint symbols, appear to mismatch with the sign language symbols that the signifier clearly expects. In some embodiments, body language and emotion detection and classification may be performed by an emotion detection model (e.g., a machine learning model trained to extract emotion data based on kinematic keypoints and / or other factors). In some embodiments, the emotion detection and classification performed by the sign language proficiency engine instance 134 may also be configured to adapt to cultural differences in how various aspects of emotion can be expressed.

[0033] The sign language proficiency engine instance 134 can then generate keypoint position deviation data, which represents one or more deviations between the kinematic keypoint symbols extracted from the video data and the standardized kinematic keypoint patterns obtained from one or more sign language dictionaries 136. For example, as Figure 3BAs illustrated, the sign language proficiency engine instance 134 can process image data (shown at 320) to obtain extracted kinematic keypoint symbols (shown at 322). Based on the extracted kinematic keypoint symbols, the sign language proficiency engine instance 134 searches one or more sign language dictionaries 136 to obtain standard kinematic keypoint symbols (shown at 330) that have similarity to the extracted kinematic keypoint symbols 322 (e.g., within a similarity threshold). The sign language proficiency engine instance 134 defines the standard kinematic keypoint symbols as representing the expected sign language symbols for the signifier. The sign language proficiency engine instance 134 can then perform kinematic keypoint deviation analysis 340 to generate keypoint position deviation data 350. The keypoint position deviation data 350 can represent the deviation of the position of the keypoint in the extracted kinematic keypoint symbols 322 from where the keypoint position should be located to produce the expected sign language symbols according to one or more sign language dictionaries 136. In some embodiments, minor deviations (e.g., deviations less than a deviation threshold) can be ignored. The deviation data 350 can indicate the locations of extracted keypoints that have deviated from standard keypoint locations (e.g., deviated beyond a deviation threshold). For example, sign language proficiency engine instance 134 can determine the boundary shape (e.g., bounding box) around the detected keypoints and calculate the changes to determine when keypoints extracted from video data deviate from a predetermined tolerance.

[0034] In some embodiments, the keypoint position deviation data 350 may include a spatial representation of the positions of conforming and non-conforming keypoints, and / or may include adjustment data representing human posture adjustments that can be made to bring non-conforming keypoints to their correct positions, which are defined, for example, by one or more sign language dictionaries 136. For example, in Figure 3B In the diagram, deviation data 350 at 352 indicates that a set of kinematic keypoints associated with the signer's index finger is misaligned, too far to the right (shown at 354), and further indicates where those kinematic keypoints should be adjusted to (shown at 356) to properly align with the standard kinematic keypoint pattern 330. Deviation data 350 may also indicate what body posture adjustments need to be applied to the misaligned keypoints to align them correctly with the standard kinematic keypoint pattern 330. Furthermore, in some embodiments, sign language proficiency engine instance 134 may generate and include one or more text instructions (shown at 358) in the deviation data 350, which instruct the speaker what posture adjustments need to be applied to align the misaligned keypoints.

[0035] One or more sign language proficiency engine instances 134 can be implemented using one or more sign language verification machine learning models 222, which may include one or more different neural network architectures. For example, the sign language proficiency engine instance 134 can be implemented using sign language verification machine learning models 222, which include one or more trained machine learning model architectures (e.g., encoder-decoder based models) (for performing sign language detection, body posture detection, and / or kinematic keypoint extraction), one or more generative artificial intelligence models (e.g., small language model (SLM) based models, LLM based models, etc.), and / or one or more retrieval augmentation generation (RAG) artificial intelligence models. The sign language verification machine learning model 222 may generate output including kinematic keypoint positional deviation data based on searching for sign language symbols in a sign language dictionary 136 (which can be used as a basis for comparison with the extracted kinematic keypoint symbols). The sign language proficiency engine instance 134 may include a sign language recognition software module that operates in conjunction with a dialogue manager (DM) software module and / or natural language processing artificial intelligence (AI). For example, the sign language proficiency engine may be implemented using NVIDIA Riva. In some embodiments, the sign language proficiency engine instance 134 may be implemented using a set of graphics processing unit (GPU)-accelerated multilingual speech recognition and translation microservices, including a sign language neural machine translation service, and in some implementations may generate prompts for interfacing with one or more LLMs, which can be accessed by the sign language feedback framework 130 and / or one or more sign language proficiency engine instances 134.

[0036] Based on keypoint position deviation data 350 generated by the sign language proficiency engine instance 134, the enhancement manager 132 can generate kinematic feedback data 117 that is provided back to the user client application 110. The user client application 110 can use the kinematic feedback data 117 to present a visual representation of the keypoint position deviation data 350 on the UI 107. The enhancement manager 132 can generate animated kinematic feedback (e.g., real-time and / or near-real-time animated visual feedback) to the signer by presenting, for example, a kinematic keypoint pattern overlay or similar graphics that highlight kinematic keypoints whose position deviations from the dictionary-defined kinematic keypoint patterns exceed a threshold amount. The kinematic feedback data 117 may include corrections (e.g., orientation and / or distance) that indicate how keypoints exceeding the tolerance should be adjusted to align the signer's hand gesture with a more accurate representation of the expected sign language symbol. For example, based on real-time kinematic feedback data 117, the user client application 110 can control the UI 107 to display arrows or other graphics showing how the signifier can move and adjust their body posture (e.g., adjust one or more hands, one or more fingers, one or more arms, torso, and / or facial expressions) to better align the sign language symbols they are presenting with the intended sign language symbols. In some embodiments, the UI 107 can display visual correction feedback for multiple different key points, where key points exceeding the tolerance are clearly highlighted in real time and indicated how to improve the alignment of key points exceeding the tolerance.

[0037] For example, in some embodiments, based on real-time kinematic feedback data 117, UI 107 may display a kinematic keypoint pattern overlay, where keypoints out of tolerance can be easily distinguished from keypoints within tolerance (e.g., by different colors and / or other markers). The kinematic keypoint pattern overlay may include an animated representation of dynamically updated kinematic keypoint patterns. For example, the position of the kinematic keypoints in the pattern may be dynamically adjusted to follow the changing hand gestures of the gesturing user. In some embodiments, real-time kinematic feedback may present a kinematic keypoint pattern overlay where keypoints within tolerance are displayed using a first color (e.g., green), and keypoints out of tolerance are displayed using a second color (e.g., red). When the gesturing user adjusts their body and / or hand gestures based on the graphical guidance displayed on UI 107, the colors of those out-of-tolerance keypoints that become aligned within tolerance may change (e.g., from red to green) to provide positive reinforcement to the gesturing user that they are correctly adjusting their hand gestures. If the adjustments made by the signer cause a key point that was previously within tolerance to become outside tolerance, then the color of that key point can be changed (e.g., from green to red) to provide feedback to the signer that their attempt to correct their hand gesture has resulted in a further misalignment with the intended sign language symbol.

[0038] In other embodiments, other forms of real-time kinematic feedback may be used. The enhancement manager 132 may receive uplink content data feed 115 and generate video data that enhances the video data from the uplink content data feed 115 to generate kinematic feedback data 117. For example, the enhancement manager 132 may generate a version of the uplink content data feed 115 that illustrates body posture adjustments and / or instructions for guiding adjustments by the signifier based on the kinematic feedback data 117. The real-time kinematic feedback data 117 may include enhancements (e.g., modifications) to the uplink content data feed 115 (e.g., using a generative artificial intelligence (AI) model) to alter the signifier's hand posture to illustrate deviations between the expected kinematic keypoint pattern of the sign language symbol and the extracted kinematic keypoint symbol. Specifically, the kinematic feedback data 117 may include real-time video feedback showing how the signifier should adjust their hand posture using a modified image of their hand, and illustrating the deviation between their actual hand posture and the target hand posture. This will ensure that their extracted keypoints are within tolerance to match the expected sign language symbols selected from the dictionary. In some embodiments, the kinematic feedback data 117 may include enhanced and / or extended realistic representations showing a combination of generative AI hand posture modifications and keypoint pattern overlays.

[0039] In some embodiments, supplementing or replacing UI 107, user client application 110 may use kinematic feedback data 117 to control one or more training peripherals 109. For example, training peripheral 109 (e.g., coupled to or integrated with client device 105) may include a robotic hand that responds to control from user client application 110 based on kinematic feedback data 117, enabling the robotic hand to provide a standard of correct hand gestures. For example, a human user may place their hand on the robotic hand (or alternatively, the robotic hand may rest on a human hand) while the robotic hand adjusts its posture (e.g., hand and fingers) to provide bodily feedback to the user, improving their sign language proficiency. The robotic hand can then provide feedback for better hand gesture techniques, including manual manipulation of the human hand, audio cues, etc.

[0040] like Figure 2 As shown, in some embodiments, the enhancement manager 132 may include one or more generative artificial intelligence models 220 (e.g., small language model (SLM) based models, LLM-based models, video and / or audio generation models, avatar managers (e.g., for instantiating and controlling avatars), etc.) to generate kinematic feedback data 117. For example, in some embodiments, one or more generative artificial intelligence models 220 may serve as prompts for keypoint position deviation data from one or more instantiated sign language proficiency engine instances 134 to generate the kinematic feedback data 117 described herein. In some embodiments, one or more models of the sign language feedback framework 130 may be implemented at least in part using an artificial intelligence (AI) based software framework (e.g., a suite of cloud-hosted AI models) (such as, but not limited to, NVIDIA's Tokkio).

[0041] In some embodiments, the sign language feedback framework 130 as described herein can be implemented as a plug-in or module component of the client application 110 and executed locally on the client device 105. In some embodiments, one or more functions of the sign language feedback framework 130 can be exposed to the client application 110 as a cloud computing platform-based network service (e.g., a microservice). For example, in some embodiments, one or more functions of the sign language feedback framework 130, such as one or more sign language verification machine learning models 222 of the sign language proficiency engine instance 134, the generative AI model 220 of the enhancement manager 132, and / or the sign language detection model 224, can be implemented as a network service accessed by the client application 110 using, for example, application programming interface (API) function calls, HTTP control channels, and / or WebRTC sender and receiver clients for audio, video, and / or text data. In some embodiments, one or more aspects of the sign language feedback framework 130 functions can be implemented as an optional service of the underlying communication platform 120 (e.g., as a cloud computing platform). The sign language feedback framework 130 functionality is provided by Maxine and implemented as a web application by one or more servers of the communication platform 120. In some embodiments, the sign language feedback framework 130 functionality may be distributed across client applications, exposed web services, and / or web applications hosted by one or more servers. For example, client application 110 may include a front-end of the sign language feedback framework 130 that communicates and interfaces with user interface 107 and with a back-end that includes computing hardware resources to execute sign language proficiency engine instance 134, enhancement manager 132, and / or sign language detection model 224.

[0042] Now for reference Figure 4 , Figure 4 This is a diagram illustrating an example user interface 410 (such as UI 107 generated and controlled by a user client application 110). In this example, UI 410 represents a UI used for interacting with a video conferencing session (e.g., a video teleconference) associated with a virtual environment session 122. In this example UI 410, the UI includes a sign language presenter feedback screen 420 that presents an enhanced version of the uplink content data feed 115, which includes feedback to the presenter based on kinematic feedback data 117 generated by the sign language feedback framework 130.

[0043] like Figure 4As shown, the user interface 410 may also include a participant area 430 displaying other participants in the session. As described herein, the virtual environment session 122 includes a logical infrastructure established by the communication platform 120 for the real-time transmission of content data between participants. Therefore, the UI 410 can be controlled to present downlink content data feeds 116 received from other participants as separate content feeds, such as those shown at 432 and 433. In this example, many user participants have chosen to share their live local video feeds so that those participants are presented as video in participant area 430 using those live local video feeds, as shown via windows 432 and 433. Other user participants, indicated at 434, have chosen not to share live local video feeds and are instead presented in participant area 430 using still profile images or a default image.

[0044] In some embodiments, the video of the presenting user displayed in the sign language presenter feedback screen 420 can be generated based on kinematic feedback data 117 generated by the sign language feedback framework 130 and uplink content data feed 115 from the presenting user's client application 110. In this example, the user client application 110 may have already been configured with sign language user preferences for indicated sign language (e.g., ASL). Therefore, the sign language feedback framework 130 generates kinematic feedback data 117 that provides the presenter with adjustments to their gesture indications to help them conform to the selected sign language.

[0045] As previously discussed, the language proficiency engine instance 134 can perform kinematic keypoint deviation analysis 340 to generate keypoint position deviation data 350. Keypoint position deviation data 350 can represent the deviation of the position of a keypoint in the extracted kinematic keypoint symbol 322 from the position where the keypoint should be located to generate the expected sign language symbol according to one or more sign language dictionaries 136. In some embodiments, minor deviations (e.g., deviations less than a deviation threshold) can be ignored. The deviation data 350 can indicate those extracted keypoint positions that have deviated from the standard keypoint position (e.g., deviated beyond the deviation threshold). The deviation data 350 can also indicate what adjustments need to be applied to non-compliant keypoints to align them correctly with the standard kinematic keypoint pattern 330. Furthermore, in embodiments, the sign language proficiency engine instance 134 can generate and include one or more text instructions in the deviation data 350, instructing the speaker what postural adjustments need to be applied to align non-compliant keypoints. Based on the key point position deviation data 350 generated by the sign language proficiency engine instance 134, the enhancement manager 132 can generate kinematic feedback data 117 that is provided back to the user client application 110. The user client application 110 can use the kinematic feedback data 117 to present a visual representation of the key point position deviation data 350 within the sign language presenter feedback screen 420.

[0046] In some embodiments, the sign language presenter feedback screen 420 can present animated kinematic feedback (e.g., real-time and / or near-real-time animated visual feedback) to the signer by presenting video data enhanced from the video data in the uplink content data feed 115. The sign language presenter feedback screen 420 can also present AI-generated enhancements from the uplink content data feed 115 illustrating body posture adjustments and / or adjustment instructions to guide the signer based on kinematic feedback data 117, such as enhancing the signer's hand posture to illustrate deviations between the expected kinematic keypoint patterns of the sign language symbol and the extracted kinematic keypoint symbols. That is, the sign language presenter feedback screen 420 can present animated feedback, such as in the form of real-time video (showing the signer how to adjust their body and / or hand posture using a modified image depicting the signer), and can illustrate deviations between their actual posture and the target posture, which will keep their extracted keypoints within tolerance to match the expected sign language symbol selected from the dictionary. For example, the sign language presenter feedback screen 420 displays a video image of the signer generated based on the uplink content data feed 115 at 421. Based on kinematic feedback data 117, the image of the signer can be enhanced to display extracted key points that conform to the standard kinematic key point pattern of the expected sign language symbol, as shown at 440. As shown at 442, the image of the signer can be enhanced to display extracted key points that do not conform to the standard kinematic key point pattern (shown at 442) and / or where these key points should be located to obtain a conforming kinematic key point pattern (shown at 444). In some embodiments, the image of the signer can be enhanced with further graphical cues (such as shown at 446) illustrating the direction and / or movement that the signer can perform to obtain a more accurate representation of the expected sign language symbol. For example, based on real-time kinematic feedback data 117, the sign language presenter feedback screen 420 may display arrows or other graphics showing how the signer can move and adjust their body posture (e.g., adjust one or more hands, one or more fingers, one or more arms, torso, and / or facial expressions) to better align the sign language symbols they are presenting with the intended sign language symbols. In some embodiments, the UI 410 may display visual correction feedback for multiple different key points, where key points 442 that are out of tolerance (e.g., non-compliant) are clearly highlighted in real time, and instructions 446 are given on how to improve their alignment. Furthermore, in some embodiments, the sign language presenter feedback screen 420 may present one or more text instructions (shown at 422) to guide the speaker in applying posture adjustments to transform non-compliant key points 442 into compliant key points 444.In some embodiments, the sign language presenter feedback screen 420 may present the speaker's image without modification (e.g., uplink content data feed 115), but will present one or more of the non-conforming extracted keypoints 442, conforming extracted keypoints 444, and / or graphic cues 446 as graphic overlays on or near the speaker's image. In some embodiments, the sign language presenter feedback screen 420 may include enhanced and / or extended reality presentations that display a combination of generative AI hand gesture modifications and keypoint pattern overlays.

[0047] In some embodiments, the sign language presenter feedback screen 420 may use an animated avatar 423 to enhance the image of the signer, the animated avatar 423 performing the gesture and providing one or more of text instructions 422, non-compliant extracted key points 442, compliant extracted key points 444 and / or graphic cues 446.

[0048] As previously discussed, in some embodiments, the communication platform 120 may include a cloud-based virtual reality space (e.g., a virtual reality in which a user, represented by an avatar, can interact) or another virtual environment (e.g., NVIDIA Omniverse). Therefore, in some such embodiments, the sign language feedback framework 130 may generate sign language video data used by the platform to render the avatar in such a virtual environment where sign language can be used for gesturing. In some embodiments, one or more aspects of UI 107 and / or UI 410 may be implemented by HMI 106 as immersive augmented reality (AR) / virtual reality (VR) rendering, HMI 106 including AR / VR goggles, glasses, or headphones, where each participant will see avatars of other participants they are conversing with and / or gesturing to, where the viewer's personalized rendering is presented as communication in sign language based on the viewer's sign language preferences.

[0049] Figure 5 This is a diagram illustrating a method for providing sign language kinematic feedback according to some embodiments of the present disclosure. It should be understood that the details herein refer to… Figure 5 The features and elements described in method 500 may be used in combination with, in combination with, or substituted for elements of any other embodiment discussed herein, and vice versa. Furthermore, it should be understood that... Figure 5 The functionality, structure, and other descriptions of the elements of the embodiments described herein can be applied to similar or like named or described elements in any of the figures and / or embodiments described herein, and vice versa.

[0050] Each block of the method 500 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be executed by one or more processors, including processing circuitry and the execution of instructions stored in memory. The method can also be implemented as computer-usable instructions stored on a computer storage medium. The method can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or a plug-in to another product, to name just a few. Furthermore, as an example, regarding... Figure 1 The sign language-based communication system 100 describes method 500. However, these methods may also be performed by any system or any combination of systems (including, but not limited to, those described herein) in addition to or alternatively.

[0051] As discussed in more detail herein, the method may typically include generating output comprising animated kinematic feedback data representing instructions for performing one or more pose adjustments, the one or more pose adjustments being calculated based on one or more keypoint position deviations between a standardized kinematic keypoint pattern and one or more extracted kinematic keypoint symbols extracted from video data including representations of human poses.

[0052] In block B502, method 500 includes receiving video data including representations of sign language communication. In some embodiments, client device 105 and / or HMI 106 may include one or more cameras 108 that capture image data of the user for uplink transmission to communication platform 120 as uplink content data feed 115. Uplink content data feed 115 may include audio and / or video data captured by the user from user client application 110. That is, uplink content data feed 115 may include communication content data, which includes video data representing sign language communication among other data.

[0053] In block B504, method 500 includes extracting one or more kinematic keypoint symbols for sign language body postures from video data. The one or more kinematic keypoint symbols for sign language body postures include kinematic keypoints corresponding to at least one of the skeletal bones or joints (e.g., body postures including hand shape, curvature, and / or movement). As previously discussed, sign language proficiency engine instance 134 may include one or more machine learning models trained to perform human posture recognition, such as skeletal kinematic recognition, which extracts the relative positions of predefined kinematic keypoints (e.g., skeletal bones and joints and / or other features) from images of one or more hands, one or more arms, face, and / or torso of an individual. Sign language proficiency engine instance 134 may detect and / or extract from uplink content data feed 115 video data showing the relative positions of predefined kinematic keypoints displayed by a signer to establish two-dimensional kinematic keypoint symbols, such as in Figure 3A As illustrated in the figure. In some embodiments, the method may implement a framework comprising one or more machine learning models that extract one or more hand pose kinematic keypoint symbols from video data.

[0054] In block B506, method 500 includes determining a standardized kinematic keypoint pattern corresponding to a sign language symbol based on one or more extracted sign language body posture kinematic keypoint symbols. The method may include performing a search of one or more body posture dictionaries based on the extracted sign language body posture kinematic keypoint symbols to determine the standardized hand posture kinematic keypoint pattern. The one or more body posture dictionaries include one or more sign language-based body posture dictionaries based on one or more sign language versions. In some embodiments, the method may execute one or more retrieval-enhanced generative (RAG) artificial intelligence models that access one or more data sources including one or more body posture dictionaries. The method may include an execution framework comprising one or more machine learning models that generate kinematic feedback data based at least on video data and select a sign language dictionary based at least on the sign language version indicated by the video data.

[0055] As discussed herein, the kinematic keypoint sign language symbols extracted from captured human postures can be used as queries by the sign language proficiency engine instance 134 to search the sign language dictionary 136 to find corresponding standard versions of the sign language symbols. These standard versions can be used as a basis for comparison with the extracted kinematic keypoint symbols to generate kinematic keypoint positional deviation data. In some embodiments, the sign language proficiency engine instance 134 can perform the search of the sign language dictionary 136 based on performing a similarity algorithm. For example, the similarity algorithm can select kinematic keypoint patterns from the sign language dictionary 136 that have similarity (e.g., within a similarity threshold) to the extracted kinematic keypoint symbols and define them as representing the expected sign language symbol for the person gesturing. Based on the expected sign language symbol determined from the sign language dictionary 136, the sign language proficiency engine instance 134 calculates the deviation between the kinematic keypoint pattern of the expected sign language symbol and the extracted kinematic keypoint symbol. In some embodiments, the similarity algorithm can combine the use of contextual information to search the sign language dictionary 136 to determine the expected sign language symbol for the person gesturing. For example, even where sign language symbols can be defined based on hand and / or torso postures, similarity algorithms can utilize facial key points to classify facial expressions (e.g., happiness, sadness, anger, confusion, etc.) to help identify the expected sign language symbols and their corresponding standardized kinematic key point patterns from the sign language dictionary 136.

[0056] In block B508, method 500 includes calculating one or more kinematic keypoint positional deviations based on a first set of kinematic keypoint patterns and a second set of kinematic keypoints of one or more extracted sign language body posture kinematic keypoint symbols. For example, based on the expected sign language symbol determined from sign language dictionary 136, sign language proficiency engine instance 134 calculates the deviation between the kinematic keypoint pattern of the expected sign language symbol and the extracted kinematic keypoint symbol. In some embodiments, the similarity algorithm may incorporate contextual information to search sign language dictionary 136 to determine the expected sign language symbol for the signifier. For example, even where sign language symbols can be defined based on hand posture and / or torso posture, the similarity algorithm may utilize facial keypoints to classify facial expressions (e.g., happy, sad, angry, confused, etc.) to help identify the expected sign language symbol and its corresponding standardized kinematic keypoint pattern from sign language dictionary 136. Sign language proficiency engine instance 134 may then perform kinematic keypoint deviation analysis 340 to generate keypoint positional deviation data 350. Keypoint position deviation data 350 can represent the deviation of the position of a keypoint in the extracted kinematic keypoint symbol 322 from its intended keypoint position to generate the expected sign language symbol based on one or more sign language dictionaries 136. In some embodiments, minor deviations (e.g., deviations less than a deviation threshold) can be ignored. The deviation data 350 can indicate those extracted keypoint positions that have deviated from standard keypoint positions (e.g., deviated beyond a deviation threshold). For example, the sign language proficiency engine instance 134 can determine the boundary shape (e.g., bounding box) around the detected keypoints and calculate changes to determine when keypoints extracted from video data deviate from a predetermined tolerance.

[0057] In block B510, method 500 includes: causing (e.g., controlling) a user interface to present kinematic feedback data based on one or more keypoint position deviations, the kinematic feedback data indicating one or more adjustments to align a first set of kinematic keypoints with a second set of kinematic keypoints, at least based on an established tolerance. The kinematic feedback data may include an animated kinematic keypoint pattern including at least one indication of body posture adjustments for aligning one or more sign language body posture kinematic keypoint symbols with a standardized kinematic keypoint pattern. In some embodiments, the animated kinematic keypoint pattern includes one or more visual indications of kinematic keypoint positions exceeding the tolerance. In some embodiments, the kinematic feedback data may include modifications to video data to alter the hand posture of the signifier, thereby illustrating one or more deviations between the standardized hand posture kinematic keypoint pattern and the extracted one or more sign language body posture kinematic keypoint symbols.

[0058] As discussed herein, keypoint position deviation data 350 may include spatial representations of conforming and non-conforming keypoint positions, and / or may include adjustment data representing adjustments made to bring non-conforming keypoints to their correct positions as defined by one or more sign language dictionaries 136. Deviation data 350 may also indicate what adjustments need to be applied to non-conforming keypoints to align them correctly with standard kinematic keypoint patterns 330. Furthermore, in embodiments, sign language proficiency engine instance 134 may generate and include one or more text instructions in deviation data 350 that indicate what postural adjustments the speaker needs to apply to conform non-conforming keypoints. User client application 110 may use kinematic feedback data 117 to present a visual representation of keypoint position deviation data 350 on UI 107. Enhancement manager 132 may generate animated kinematic feedback (e.g., real-time and / or near-real-time animated visual feedback) to the signer by presenting, for example, a kinematic keypoint pattern overlay or similar graphics (highlighting kinematic keypoints whose position deviations from the kinematic keypoint patterns defined by the dictionary exceed a threshold amount). Kinematic feedback data 117 may include corrections (e.g., direction and / or distance) indicating how key points exceeding tolerance should be adjusted to align the signifier's hand gestures to a more accurate representation of the intended sign language symbol. For example, based on real-time kinematic feedback data 117, user client application 110 may control UI 107 to display arrows or other graphics showing how the signifier can move and adjust their body posture (e.g., adjust one or more hands, one or more fingers, one or more arms, torso, and / or facial expressions) to better align the presented sign language symbol with the intended sign language symbol. In some embodiments, UI 107 may display visual correction feedback for multiple different key points, clearly highlighting key points exceeding tolerance in real time and indicating how to improve their alignment.

[0059] In some embodiments, the method may control one or more robotic peripherals based at least on kinematic feedback data. For example, in some embodiments, in addition to or as an alternative to UI 107, user client application 110 may use kinematic feedback data 117 to control one or more training peripherals 109. For example, training peripheral 109 (e.g., coupled to or integrated with client device 105) may include a robotic hand responding to control from user client application 110 based on kinematic feedback data 117, enabling the robotic hand to provide a standard of correct hand gestures.

[0060] In some embodiments, the machine (e.g., a robot or ego machine) may be trained to communicate in sign language based at least on kinematic feedback data. For example, in some embodiments, client device 105 may include an AI-based user who learns to communicate in sign language based on kinematic feedback data 117.

[0061] In some embodiments, the method can control a multi-user virtual environment to render one or more avatars within the multi-user virtual space to present communication content data, at least based on sign language video data. As previously mentioned, the communication platform 120 may include a cloud-based collaborative content creation platform, such as, but not limited to, NVIDIA Omniverse or other augmented reality (AR) / virtual reality (VR) / mixed reality (MR) multi-user virtual environments (e.g., virtual reality). In some such embodiments, the sign language feedback framework 130 may generate sign language video data used by the platform to render avatars in the virtual environment, which can use sign language represented by the sign language video data to gesture. UI 107 can be implemented as immersive AR / VR / MR rendering via HMI 106, which includes AR / VR / VR goggles, glasses, or headphones, where each participant will see avatars of other participants they are conversing with and / or gesturing to, where the viewer's personalized rendering is presented as communication in sign language based on the viewer's sign language preferences.

[0062] In some embodiments, the systems and methods described herein may be executed within or in conjunction with a simulated environment (e.g., NVIDIA's DriveSIM, NVIDIA's Omniverse, etc.) that uses simulated data (e.g., simulated sensor data from simulated sensors of a virtual machine or simulator). For example, sign language data may be used to perform actions associated with virtual machines and / or participants within the environment (e.g., navigation, communication, etc.). In some embodiments, a simulated environment and / or one or more of its objects, features, or components may be generated or managed within a 3D content collaboration platform (e.g., NVIDIA's Omniverse) for industrial digitization, generative physical artificial intelligence (AI), and / or other use cases, applications, or services. For example, the content collaboration platform or system may include a system for using or developing generic scene descriptors (USD) (e.g., OpenUSD) data for managing objects, features, scenes, etc., in simulated environments, digital environments, etc. The platform may include realistic physical simulations, such as using NVIDIA's PhysX SDK, to simulate real physics and physical interactions with simulations hosted by the platform. The platform can integrate OpenUSD along with ray tracing / path tracing / light transport simulations (e.g., NVIDIA's RTX rendering technology) into software tools and simulation workflows for building, training, deploying, or testing AI systems (such as systems for testing, validating, training (e.g., machine learning models, neural networks, etc.)) and / or other tasks related to automobiles, robots, machines, or other applications.

[0063] In some embodiments, the systems and methods described herein can be deployed in in-vehicle infotainment (IVI) systems or in-cabin experience (IX) applications. For example, an infotainment system within a vehicle (e.g., a car, truck, drone, construction equipment, robot, semi-autonomous vehicle, or autonomous vehicle) may include one or more onboard processors (e.g., a central processing unit (CPU), graphics processing unit (GPU), hardware-based deep learning accelerator (DLA), software-based programmable vision accelerator (PVA) that may include one or more vector processing units (VPU), direct memory access (DMA) systems and / or pixel processing engines (PPE), software-based optical flow accelerators (OFA), SoCs, etc.), memory and / or storage devices (e.g., for storing control algorithms, sensor data, and one or more machine learning models), and memory and / or storage devices (e.g., for storing entertainment content, navigation data, and user preferences). The system can use these processors to execute one or more machine learning models to achieve features such as passenger monitoring, gesture recognition, and real-time communication with other services via network connectivity. In-vehicle infotainment systems can also use natural language processing (NLP) models to enable voice-based and / or sign language-based interactions. One or more machine learning models can be stored locally or accessed via one or more APIs connected to cloud services, enabling the system to process requests in real-time or near real-time.

[0064] In some embodiments, the systems and methods described herein can be deployed in robotic applications. For example, a robot or robotic system may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), hardware-based programmable vision accelerator (PVA) that may include one or more vector processing units (VPU), direct memory access (DMA) systems, and / or pixel processing engines (PPE), software-based optical flow accelerators (OFA), SoCs, etc.) and memory and / or storage devices (e.g., for storing control algorithms, sensor data, and one or more machine learning models). The robotic system can use these processors to execute one or more machine learning models (e.g., language models) that allow it to autonomously or semi-autonomously perform complex tasks, such as interacting with and / or manipulating static and / or dynamic objects, or navigating its environment using sensors such as cameras, lidar, radar, ultrasonic sensors, etc. The system may use sensor fusion techniques to combine data from multiple sensors (e.g., cameras, infrared devices, lidar, radar, accelerometers, etc.) to create a comprehensive model of the robot's surrounding environment. This data can be processed locally on the robot or sent to a remote server for more computationally intensive tasks, such as 3D mapping or SLAM (Simultaneous Localization and Mapping). In one or more embodiments, data from individual robots (e.g., sensor data, task status, or environmental conditions) can be uploaded to the cloud, where a centralized AI model can analyze optimized commands and distribute them throughout the fleet. In some embodiments, the machine learning models described herein (e.g., language models, visual language models (VLM), large language models (LLM), small language models (SLM), multimodal language models (MMLM), diffusion models, neural radiation field (NeRF) models, DNNs, etc.) can be used to allow robots to perceive and reason about their environment and / or communicate with one or more other robots and / or humans in the environment. In some embodiments, robots can (e.g., using one or more network interface cards (NICs) and / or data processing units (DPUs)) communicate with one or more locally hosted servers / computing devices and / or with one or more remotely located servers / computing devices (e.g., in one or more data centers).

[0065] In some examples, one or more machine learning models described in this paper (e.g., deep neural networks, language models, LLM, SLM, VLM, multimodal language models, perceptual models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural radiation field (NeRF) models, etc.) can be packaged into one or more cloud-hosted microservices, such as inference microservices (e.g., NVIDIA...). NIM (Network Inference Microservice) can include containers (e.g., an operating system (OS-level virtualization package) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine". For example, an inference microservice can include the container itself and one or more models (e.g., weights and biases). In some cases, such as when one or more machine learning models are small enough (e.g., have a sufficiently few parameters), one or more models can be contained within the container itself. In other examples, such as when one or more models are large, one or more models can be hosted / stored in the cloud (e.g., in a data center) and / or can be hosted on-premises and / or at the edge (e.g., on a local server or computing device, but outside the container). In such embodiments, one or more models are accessible via one or more APIs, such as a representative state transfer (REST) ​​API. Therefore, in some embodiments, one or more machine learning models described herein can be deployed as inference microservices to enhance performance. Accelerate the deployment of one or more models on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., built using standardized AI model deployment and execution software, such as NVIDIA's Triton Inference Server), and / or one or more APIs for high-performance deep learning inference, which may include an inference runtime and model optimizations for low latency and high throughput for production applications (such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). One or more machine learning models described herein may be included as part of a microservice along with accelerated infrastructure that has the ability to be deployed with a single command and / or orchestrated and autonomously scaled using a container orchestration system on the accelerated infrastructure (e.g., from single-device to data center scale).Therefore, an inference microservice may include one or more machine learning models (e.g., optimized for high-performance inference), inference runtime software that executes one or more machine learning models and provides output / response to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, identity verification, and / or other monitoring. In some embodiments, the inference microservice may include software for in-situ replacement and / or updating of one or more machine learning models. When replacement or updating occurs, the software performing the replacement / update may maintain the user configuration of the inference runtime software and the enterprise management software.

[0066] The systems and methods described herein can be used for a variety of purposes, including but not limited to machine control, machine motion, machine actuation, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and supervision, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational artificial intelligence, optical transmission simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, generative AI, and / or any other suitable application.

[0067] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, shipboard systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems implementing one or more language models (such as one or more large language models (LLMs), one or more small language models (SLMs), one or more visual language models (VLMs), one or more multimodal language models (MMLMs)), systems for performing optical transmission simulations, systems for performing collaborative content creation for 3D assets, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0068] Example computing device

[0069] Figure 6The block diagram is provided for use in implementing one or more example computing devices 600 of some embodiments of this disclosure. The computing device 600 may include an interconnect system 602 directly or indirectly coupled to the following devices: memory 604, one or more central processing units (CPUs) 606, one or more graphics processing units (GPUs) 608, a communication interface 610, input / output (I / O) ports 612, input / output components 614, a power supply 616, one or more presentation components 618 (e.g., one or more displays), and one or more logic units 620. In at least one embodiment, one or more computing devices 600 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 608 may include one or more vGPUs, one or more CPUs 606 may include one or more vCPUs, and / or one or more logic units 620 may include one or more virtual logic units. Therefore, computing device 600 may include discrete components (e.g., a complete GPU dedicated to computing device 600), virtual components (e.g., a portion of the GPU dedicated to computing device 600), or a combination thereof. In some embodiments, one or more functionalities of the user client application 110, HMI 106, UI 107, and / or sign language feedback framework 130 described herein may be implemented at least in part using one or more computing devices 600.

[0070] although Figure 6 The various blocks are shown connected via an interconnect system 602 with wiring, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component 618, such as a display device, may be considered an I / O component 614 (e.g., if the display is a touchscreen). As another example, CPU 606 and / or GPU 608 may include memory (e.g., memory 604 may represent a storage device other than the memory of GPU 608, CPU 606, and / or other components). Thus, Figure 6 The computing devices mentioned are merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all of these are considered within the same category. Figure 6 Within the scope of computing devices.

[0071] Interconnect system 602 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 602 may include one or more link or bus types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Fast (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 606 may be directly connected to memory 604. Furthermore, CPU 606 may be directly connected to GPU 608. In cases where there is a direct or point-to-point connection between components, interconnect system 602 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required in computing device 600.

[0072] The memory 604 may include any medium of a wide variety of computer-readable media. A computer-readable medium can be any available medium that can be accessed by the computing device 600. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. For example and without limitation, computer-readable media may include computer storage media and communication media.

[0073] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media, implemented in any way or by any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 604 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by computing device 600. As used herein, computer storage media does not include the signal itself.

[0074] Computer storage media may contain computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and include any information transport medium. The term "modulated data signal" can refer to a signal whose characteristics are set or altered in a manner that encodes information into that signal. For example and without limitation, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as sound, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer-readable media.

[0075] One or more CPUs 606 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 600 to perform one or more of the methods and / or processes described herein. Each of the one or more CPUs 606 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a large number of software threads simultaneously. The one or more CPUs 606 may include any type of processor and may include different types of processors depending on the type of computing device 600 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 600, the processor may be an advanced RISC mechanism (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors such as math coprocessors, computing device 600 may also include one or more CPUs 606.

[0076] In addition to or replacing one or more CPUs 606, one or more GPUs 608 may also be configured to execute at least some computer-readable instructions to control one or more components of computing device 600 to perform one or more of the methods and / or processes described herein. One or more GPUs 608 may be integrated GPUs (e.g., having one or more CPUs 606) and / or one or more GPUs 608 may be discrete GPUs. In embodiments, one or more GPUs 608 may be coprocessors of one or more CPUs 606. Computing device 600 may use one or more GPUs 608 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, one or more GPUs 608 may be used for general-purpose computing on GPUs (GPGPU). One or more GPUs 608 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. One or more GPUs 608 may generate pixel data for outputting an image in response to rendering commands (e.g., rendering commands received via a host interface from one or more CPUs 606). One or more GPUs 608 may include graphics memory, such as display memory, for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 604. One or more GPUs 608 may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 608 may generate different portions of pixel data or GPGPU data for different outputs (e.g., the first GPU for the first image, the second GPU for the second image). Each GPU may include its own memory or may share memory with other GPUs.

[0077] In addition to or replacing one or more CPUs 606 and / or one or more GPUs 608, one or more logic units 620 may be configured to execute at least some computer-readable instructions to control one or more components of computing device 600 to perform one or more methods and / or processes described herein. In embodiments, one or more CPUs 606, one or more GPUs 608, and / or one or more logic units 620 may execute any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 620 may be part of and / or integrated into one or more CPUs 606 and / or one or more GPUs 608, and / or one or more logic units 620 may be discrete components of CPU 606 and / or GPU 608 or otherwise external thereto. In embodiments, one or more logic units 620 may be processors of one or more CPUs 606 and / or one or more GPUs 608.

[0078] In some embodiments, one or more functions of the user client application 110, HMI 106, UI 107, and / or sign language feedback framework 130 described herein may be implemented at least in part using one or more central processing units (CPUs) 606, one or more graphics processing units (GPUs) 608, and / or one or more logic units (620). For example, one or more machine learning models of the sign language feedback framework 130 may include neural networks executing on one or more GPUs 608 to perform the functions of the sign language detection model 224, the sign language proficiency engine 134, and / or the enhancement manager 132.

[0079] Examples of one or more logic units 620 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multiprocessor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating-point unit (FPU), input / output (I / O) elements, peripheral component interconnect (PCI) or peripheral component interconnect fast (PCIe) elements, etc.

[0080] The communication interface 610 may include one or more receivers, transmitters, and / or transceivers that allow the computing device 600 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. The communication interface 610 may include components and functions that allow communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 620 and / or the communication interface 610 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via the interconnect system 602 to one or more GPUs 608 (e.g., their memory).

[0081] I / O port 612 allows computing device 600 to be logically coupled to other devices, including I / O component 614, presentation component 618, and / or other components, some of which may be built into (e.g., integrated into) computing device 600. Illustrative I / O component 614 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, browsers, printers, wireless devices, and so on. I / O component 614 can provide a Natural User Interface (NUI) for processing user-generated air gestures, voice, or other physiological input. In some instances, the input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and adjacent-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 600 (described in more detail below). Computing device 600 may include depth cameras such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof for gesture detection and recognition. In addition, computing device 600 may include an accelerometer or gyroscope that allows motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by computing device 600 to render immersive augmented reality or virtual reality.

[0082] Power supply 616 may include hardwired power, battery power, or a combination thereof. Power supply 616 may supply power to computing device 600 to allow the components of computing device 600 to operate.

[0083] Presentation component 618 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. Presentation component 618 may receive data from other components (e.g., GPU 608, CPU 606, DPU, etc.) and output that data (e.g., as an image, video, sound, etc.). In some embodiments, HMI 106 may include one or more of presentation components 618, and / or include the UI 107 described herein and displayed through one or more presentation components 618.

[0084] Example Data Center

[0085] Figure 7 An example data center 700 is illustrated, which can be used in at least one embodiment of this disclosure. Data center 700 may include a data center infrastructure layer 710, a framework layer 720, a software layer 730, and an application layer 740. In some embodiments, one or more functionalities of the sign language feedback framework 130 described herein may be implemented at least in part using data center 700.

[0086] like Figure 7As shown, the data center infrastructure layer 710 may include a resource coordinator 712, grouped computing resources 714, and node computing resources (“nodes CR”) 716(1)-716(N), where “N” represents any complete positive integer. In at least one embodiment, nodes CR 716(1)-716(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state drives or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules and cooling modules, etc. In some embodiments, one or more nodes CR 716(1)-716(N) may correspond to servers having one or more of the aforementioned computing resources. Furthermore, in some embodiments, nodes CR716(1)-716(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more nodes CR716(1)-716(N) may correspond to virtual machines (VMs). In some embodiments, one or more functions of the sign language feedback framework 130 described herein may be implemented at least in part using one or more nodes CR716(1)-716(N). For example, one or more machine learning models of the sign language feedback framework 130 may include neural networks executing on one or more nodes CR716(1)-716(N) to perform the functions of the sign language detection model 224, the sign language proficiency engine 134, and / or the enhancement manager 132.

[0087] In at least one embodiment, the grouped computing resources 714 may include individual groups (not shown) of nodes CR716 housed in one or more racks, or a plurality of racks (also not shown) housed in data centers in various geographic locations. Individual groups of nodes CR716 within the grouped computing resources 714 may include computing, networking, memory, or storage resources that can be configured or allocated to support groups of one or more workloads. In at least one embodiment, several nodes CR716, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0088] Resource coordinator 712 may be configured or otherwise controlled to control one or more nodes CR716(1)-716(N) and / or grouped computing resources 714. In at least one embodiment, resource coordinator 712 may include a Software Design Infrastructure (“SDI”) management entity for data center 700. Resource coordinator 712 may include hardware, software, or some combination thereof.

[0089] In some embodiments, one or more functions of the sign language feedback framework 130 described herein may be hosted by data center 700 and provided as available web services to user client application 110 and / or communication platform 120.

[0090] In at least one embodiment, such as Figure 7 As shown, framework layer 720 may include a job scheduler 728, a configuration manager 734, a resource manager 736, and a distributed file system 738. Framework layer 720 may include a framework for software 732 supporting software layer 730 and / or one or more applications 742 of application layer 740. Software 732 or application 742 may respectively include web-based service software or applications, such as service software or applications provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 720 may be, but is not limited to, a free and open-source software web application framework, such as Apache Spark, which can use distributed file system 738 for large-scale data processing (e.g., "big data"). TM (Hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 728 may include a Spark driver for facilitating the scheduling of workloads supported by various layers of the data center 700. In at least one embodiment, the configuration manager 734 may be able to configure different layers, such as the software layer 730 and the framework layer 720, which includes Spark and a distributed file system 738 for supporting large-scale data processing. The resource manager 736 is able to manage cluster or group computing resources mapped to or allocated for supporting the distributed file system 738 and the job scheduler 728. In at least one embodiment, the cluster or group computing resources may include grouped computing resources 714 at the data center infrastructure layer 710. The resource manager 736 may coordinate with the resource coordinator 712 to manage these mapped or allocated computing resources. In some embodiments, one or more applications 742 and / or software 732 may at least partially include code that, when executed, implements one or more functions of the sign language feedback framework 130 described herein.

[0091] In at least one embodiment, the software 732 included in the software layer 730 may include software used by at least a portion of the nodes CR716(1)-716(N), the grouped computing resources 714, and / or the distributed file system 738 of the framework layer 720. One or more types of software may include, but are not limited to, Internet web page search software, email virus browsing software, database software, and streaming video content software.

[0092] In at least one embodiment, the application layer 740 may include one or more applications 742 that can be used by at least a portion of the nodes CR716(1)-716(N), the grouped computing resources 714, and / or the distributed file system 738 of the framework layer 720. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0093] In at least one embodiment, any of the configuration manager 734, resource manager 736, and resource coordinator 712 can perform any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. Self-modification actions can alleviate the risk of data center operators of data center 700 making potentially poor configuration decisions and can prevent underutilization and / or skewed portions of the data center.

[0094] Data center 700 may include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information according to one or more embodiments described herein. For example, a machine learning model can be trained by calculating weight parameters based on a neural network architecture using the software and computing resources described above with respect to data center 700. In at least one embodiment, by using weight parameters calculated through one or more training techniques, information can be inferred or predicted using trained machine learning models corresponding to one or more neural networks, such as, but not limited to, those described herein, using the resources described above with respect to data center 700.

[0095] In at least one embodiment, the data center 700 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as a service to allow a user to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0096] Example network environment

[0097] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may... Figure 6 The implementation is carried out on one or more instances of computing device 600—for example, each device may include similar components, features, and / or functions of computing device 600. Furthermore, in the case of implementing back-end devices (e.g., servers, NAS, etc.), the back-end devices may be included as part of data center 700, examples of which are described herein. Figure 7 To describe in more detail.

[0098] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks, or networks within multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In cases where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0099] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the server functionality described herein can be implemented on any number of client devices.

[0100] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for supporting software at the software layer and / or one or more applications at the application layer. The software or applications may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software network application framework, such as one that can use a distributed file system for large-scale data processing (e.g., "big data").

[0101] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these various functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). If the connection to a user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0102] Client devices may include those described in this article. Figure 6 The example computing device 600 described includes at least some components, features, and functions. By way of example and not limitation, the client device may be a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, aircraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, in-vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these described devices, or any other suitable device.

[0103] This disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, which are executed by a computer or other machine such as a personal digital assistant or other handheld device. Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a wide variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. This disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices linked via a communication network.

[0104] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" could include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Furthermore, "at least one of element A or element B" could include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" could include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0105] The subject matter of this disclosure is described in detail herein to satisfy legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have envisioned that the claimed subject matter may also be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms “step” and / or “block” may be used herein to imply different elements of the method employed, these terms should not be construed as implying any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. One or more processors, including processing circuitry, for: Receive video data, the video data including representations of sign language communication; Extract one or more kinematic key points of sign language body posture from the video data; Based on one or more extracted kinematic key point symbols of sign language body posture, determine the standardized kinematic key point pattern corresponding to the sign language symbol; Calculate the positional deviation of one or more kinematic key points based on the first set of kinematic key points of the standardized kinematic key point pattern and the second set of kinematic key points of one or more extracted sign language body posture kinematic key point symbols; and The user interface displays kinematic feedback data based on the positional deviations of one or more kinematic key points, the kinematic feedback data indicating one or more adjustments to align the first set of kinematic key points with the second set of kinematic key points, at least based on a predetermined tolerance.

2. The one or more processors of claim 1, wherein, The one or more sign language body posture kinematic keypoint symbols include kinematic keypoints corresponding to at least one bone or joint.

3. The one or more processors of claim 1, wherein, The one or more processors are also used for: The kinematic feedback data is generated to include an animated kinematic keypoint pattern, the animated kinematic keypoint pattern including at least one indication of at least a portion of a body posture adjustment, the body posture adjustment being used to align the one or more sign language body posture kinematic keypoint symbols with the standardized kinematic keypoint pattern.

4. The one or more processors of claim 3, wherein, The one or more processors are also used for: Generate the animation kinematic keypoint pattern to include one or more visual indications of kinematic keypoint locations beyond the tolerance.

5. The one or more processors of claim 1, wherein, The one or more processors are also used for: Control one or more robot peripherals based at least on the kinematic feedback data.

6. The one or more processors of claim 1, wherein, The one or more processors are also used for: The machine was trained to communicate using sign language, at least based on the aforementioned kinematic feedback data.

7. The one or more processors of claim 1, wherein, The one or more processors are also used for: The kinematic feedback data is generated based at least on modifications to the video data to alter the hand posture of the person gesturing, thereby illustrating one or more deviations between the standardized kinematic keypoint pattern and one or more extracted sign language body posture kinematic keypoint symbols.

8. The one or more processors of claim 1, wherein, The one or more processors are also used for: A search of one or more body posture dictionaries is performed based on one or more extracted sign language body posture kinematic keypoint symbols to determine the standardized kinematic keypoint pattern.

9. The one or more processors of claim 8, wherein, The one or more body posture dictionaries include one or more sign language-based body posture dictionaries based on one or more sign language versions.

10. The one or more processors of claim 8, wherein, The one or more processors are also used for: Execute one or more retrieval-enhanced generative AI models, wherein the one or more retrieval-enhanced generative AI models access one or more data sources including the one or more body pose dictionaries.

11. The one or more processors of claim 1, wherein, The one or more processors are also used for: A framework comprising one or more machine learning models is executed, which extract one or more kinematic keypoint symbols of sign language body postures from the video data.

12. The one or more processors of claim 1, wherein, The one or more processors are also used for: The framework includes one or more machine learning models that generate the kinematic feedback data based at least on the video data and a sign language dictionary, the sign language dictionary being selected based at least on the sign language version indicated by the video data.

13. The processor of claim 1 or more, wherein the processing circuitry is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system used to perform deep learning operations; A system used to perform remote operations; Systems used for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; Systems implemented using edge devices; Systems used to generate or present virtual reality (VR) content; A system for generating or presenting augmented reality (AR) content; A system for generating or presenting mixed reality (MR) content; Systems implemented using robots; A system for performing conversational AI operations; A system that implements one or more language models; A system that implements one or more small language models (SLMs); A system that implements one or more micro-language models (TLMs); A system that implements one or more large language model LLMs; A system that implements one or more Visual Language Models (VLMs); A system that implements one or more multimodal language models (MMLMs); A system for generating synthetic data; Systems for generating synthetic data using AI; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

14. A system comprising one or more processors, said one or more processors being used to: Extract one or more kinematic keypoint symbols from video data that includes representations of human posture; A standardized kinematic keypoint pattern is obtained based on one or more of the extracted kinematic keypoint symbols; Calculate one or more key point position deviations based on the first set of kinematic key points of the standardized kinematic key point pattern and the second set of kinematic key points of one or more extracted kinematic key point symbols; Calculate one or more body pose adjustments for the one or more extracted kinematic keypoint symbols, wherein the one or more body pose adjustments are calculated based on at least a predetermined tolerance to reduce the positional deviation of the one or more keypoints; and An output is generated that includes kinematic feedback data, which represents instructions for performing the one or more body posture adjustments.

15. The system of claim 14, wherein the one or more processors are further configured to: In response to receiving the video data via the user interface, the user interface is controlled to output a representation of the kinematic feedback data.

16. The system of claim 14, wherein the one or more processors are further configured to: The kinematic feedback data is generated to include an animated kinematic keypoint pattern, the animated kinematic keypoint pattern including at least one indication for body posture adjustment to align the one or more extracted kinematic keypoint symbols with the standardized kinematic keypoint pattern.

17. The system of claim 14, wherein the one or more processors are further configured to: The kinematic feedback data is generated based at least on modifications to the video data to alter hand posture, illustrating one or more deviations between the standardized kinematic keypoint pattern and the one or more extracted kinematic keypoint symbols.

18. The system of claim 14, wherein the one or more processors are further configured to: A search of one or more body posture dictionaries is performed based on one or more extracted kinematic keypoint symbols to determine the standardized kinematic keypoint pattern.

19. The system of claim 14, wherein the system comprises at least one of: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system used to perform deep learning operations; A system used to perform remote operations; Systems used for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; Systems implemented using edge devices; Systems used to generate or present virtual reality (VR) content; A system for generating or presenting augmented reality (AR) content; A system for generating or presenting mixed reality (MR) content; Systems implemented using robots; A system for performing conversational AI operations; A system that implements one or more language models; A system that implements one or more small language models (SLMs); A system that implements one or more micro-language models (TLMs); A system that implements one or more large language model LLMs; A system that implements one or more Visual Language Models (VLMs); A system that implements one or more multimodal language models (MMLMs); A system for generating synthetic data; Systems for generating synthetic data using AI; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

20. A method comprising: generating an output comprising animated kinematic feedback data representing instructions for performing one or more pose adjustments calculated based on one or more keypoint position deviations between a standardized kinematic keypoint pattern and one or more extracted kinematic keypoint symbols extracted from video data comprising a human pose representation.

Images