Method and apparatus for tuning a sign language library

Abstract

The application discloses a method for optimizing a sign language library, comprising the following steps: providing a sign language library with built-in basic data tables; providing a user data table in the sign language library, which is used for storing emotional sign language data with emotional types; obtaining a sign language sample provided by a target user, the sign language sample comprising a sample word, a sample emotional type and a sample sign language video; learning sample sign language actions and sample action execution time according to the sample sign language video; updating the user data table according to the sample word, the sample emotional type, the sample sign language actions and the sample action execution time; wherein, when providing a sign language library service, the user data table has a high priority, and the basic data table has a low priority. The technical scheme provided by the application can enable a user to match correct semantics in time and convey emotions at a lower cost.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a method, apparatus, computer device, and computer-readable storage medium for optimizing a sign language library. Background Technology

[0002] With the rapid development of computer technology, the field of sign language recognition is also developing rapidly. Sign language recognition uses computer vision technology to identify the meaning expressed by gestures. This technology can be combined with speech recognition, natural language processing, and voice broadcasting to be applied in social scenarios for deaf and mute individuals, enabling them to communicate normally with others.

[0003] While generative sign language simulations based on machine learning possess universality, like handwriting and speech, individual reading and writing habits vary. Users may require a considerable amount of time to adapt to sign language expressions within a single vocabulary. As the amount of information within a time limit increases and the rate of sign language simulation increases, users need faster reaction times, often failing to match the correct meaning in a timely manner. Furthermore, using simple word-gesture model animations for sign language expression suffers from incomplete information reception, such as the inability to recognize and express user emotions. More sophisticated sign language simulation technologies tend to use facial expressions to assist in conveying emotions, but capturing and representing micro-expressions is costly. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, computer device, and computer-readable storage medium for optimizing a sign language library, which can be used to solve the problems described above.

[0005] One aspect of this application provides a method for optimizing a sign language library, including:

[0006] A sign language library with a built-in basic data table is provided; wherein, the basic data table includes multiple basic sign language data, each basic sign language data includes a word and the sign language action mapped to that word and the action execution time;

[0007] The sign language library provides a user data table for storing emotional sign language data with emotion types.

[0008] Obtain sign language samples provided by target users, including sample words, sample emotion types, and sample sign language videos;

[0009] The sample sign language actions and their execution times are learned from the sample sign language videos.

[0010] The user data table is updated based on the sample words, sample emotion types, sample sign language actions, and sample action execution times.

[0011] When providing sign language library services, the user data table has high priority, while the basic data table has low priority.

[0012] Preferably, it further includes:

[0013] Based on the action execution time in each basic sign language data entry in the basic data table, the multiple basic sign language data entries are divided into multiple sign language groups;

[0014] The grouped base data table is copied multiple times to obtain multiple emotion data tables that correspond one-to-one with multiple emotion types.

[0015] Preferably, it further includes:

[0016] Based on the sample emotion type, determine the target emotion data table from the plurality of emotion data tables;

[0017] From the target emotion data table, determine the target sign language group to which the sample word belongs;

[0018] At least some of the words in the target sign language group are adjusted to obtain optimized sign language data for each of the at least some words;

[0019] The target user data table is updated based on the optimized sign language data corresponding to each word in at least some of the words.

[0020] Preferably, adjusting at least some words in the target sign language group to obtain optimized sign language data for each of the at least some words includes:

[0021] Based on the sample action execution time, the action execution time of each word in at least a portion of the words is adjusted in batches.

[0022] Preferably, the user data table includes multiple emotional sign language data entries, and at least some of the multiple emotional sign language data entries are configured with a freeze flag; the method further includes:

[0023] Identify a specific word in the target group and exclude that specific word from the adjustment scope;

[0024] The specific word is a word that is simultaneously located in the user data table and carries the frozen identifier.

[0025] Preferably, the step of learning the sample sign language actions and the execution time of the sample actions from the sample sign language video includes:

[0026] Identify the sample sign language video to obtain multiple sign language expression keyframes;

[0027] Based on the multiple sign language expression keyframes, the execution time of the target user's sample action under the sample emotion type is obtained.

[0028] Preferably, it further includes:

[0029] Based on a preset pattern, push task requests to record sign language samples to the target user and / or other users;

[0030] The task request includes the words to be recorded and the emotion types to be recorded corresponding to the sign language to be recorded.

[0031] Preferably, it also includes providing an electronic reward to the user who responded to the task request.

[0032] Preferably, it further includes:

[0033] When the vocabulary optimization rate in the sign language library reaches a threshold ratio, a sign language reading challenge task is pushed to the target user;

[0034] The responses from the target users to the sign language reading challenge are evaluated; and

[0035] Based on the evaluation results, electronic rewards will be provided to the target users.

[0036] Preferably, it further includes:

[0037] Provide the target user's sign language database to the download platform; and

[0038] Based on the number of times the target user's sign language library is downloaded, a corresponding electronic reward is provided to the target user.

[0039] One aspect of this application provides a sign language library optimization device, comprising:

[0040] The first providing module is used to provide a sign language library with a built-in basic data table; wherein, the basic data table includes multiple basic sign language data, each basic sign language data includes a word and the sign language action mapped to that word and the action execution time;

[0041] The second providing module is used to provide a user data table in the sign language library, the user data table being used to store emotional sign language data with emotion types;

[0042] The acquisition module is used to acquire sign language samples provided by the target user, the sign language samples including sample words, sample emotion types and sample sign language videos;

[0043] The learning module is used to learn sample sign language actions and the execution time of sample actions based on the sample sign language videos;

[0044] The optimization module is used to update the user data table based on the sample words, sample emotion types, sample sign language actions, and sample action execution times.

[0045] When providing sign language library services, the user data table has high priority, while the basic data table has low priority.

[0046] One aspect of this application provides a computer device, the computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described sign language library tuning method.

[0047] One aspect of this application provides a computer-readable storage medium storing a computer program that can be executed by at least one processor to perform the steps of the above-described sign language library tuning method.

[0048] The sign language library optimization method, apparatus, computer device, and computer-readable storage medium provided in this application have the following advantages:

[0049] (1) The sign language samples can be used to update the target user’s own habitual sign language to the user data table, that is, to personalize and optimize the sign language database, thereby helping the target user to quickly and accurately match the semantics of the sign language actions.

[0050] (2) Different emotion types correspond to different action rates (action execution times) in individual sign language movements. In this embodiment, based on the different action rates of sign language movements under different emotions, specifically, it can receive "sample sign language of sample words under the sample emotion type provided by the target user," obtain the sample word, sample emotion type, sample sign language movement, and sample action execution time based on the sample sign language, and then update the user data table based on the data obtained above. Therefore, when the sign language library provides services, it can prioritize matching emotion types and sign language movements / words through the user data table. If no match is found in the user data table, matching can be performed based on the basic data table. Compared to the "high computational cost required to obtain the target user's emotion type through micro-expression recognition," this embodiment conveys emotions through the action speed of sign language samples, reducing computational costs. When generating sign language simulation performances, matching emotions are used to allow the target user to receive multi-dimensional sign language information that conforms to their own expression habits. Attached Figure Description

[0051] Figure 1 This diagram illustrates an application environment for a sign language library optimization method according to an embodiment of this application.

[0052] Figure 2 A flowchart illustrating the optimization method of a sign language library according to Embodiment 1 of this application is shown in the schematic diagram.

[0053] Figure 3 A schematic diagram of the sub-flowchart for step S206 is shown;

[0054] Figure 4 The flowchart illustrating the optimization method of the sign language library according to Embodiment 1 of this application shows an additional step.

[0055] Figure 5 The flowchart for batch tuning is illustrated schematically;

[0056] Figure 6 A flowchart of an exemplary application 1 is shown schematically;

[0057] Figure 7 A flowchart of exemplary application 2 is illustrated schematically;

[0058] Figure 8 A block diagram of a sign language library optimization device according to Embodiment 2 of this application is shown schematically;

[0059] Figure 9 The illustration shows a schematic diagram of the hardware architecture of a computer device suitable for implementing a method for optimizing a sign language library according to Embodiment 3 of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0061] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0062] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0063] The following is a definition of the terminology used in this application:

[0064] Single sample: This refers to a gesture video sample generated by recording and uploading only once for a specific statement expressing a certain emotion.

[0065] Text: Here it refers to language text that has been converted into grammatically correct and communicative language through speech recognition.

[0066] To facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are described below:

[0067] The generated sign language is universally applicable based on machine learning. However, just like handwriting and speech, due to differences in everyone's reading and writing habits, users may need a long time to adapt to the sign language performance within a single sign language vocabulary. When the amount of information within a limited time increases and the rate of sign language simulation demonstration increases, users need to have a faster reaction speed. Since the system vocabulary provided by the service provider does not have individual adaptability, it is highly unlikely to help users match the correct semantics in their minds in a timely manner.

[0068] Related sign language optimization involves supplementing and correcting individual sign words in a vocabulary database based on multiple samples from machine learning. However, this optimization is inefficient and suffers from low utilization rates of the corrected and supplemented words in real-world scenarios.

[0069] Furthermore, using simple word-gesture model animations to simulate sign language expression suffers from incomplete information reception, such as the inability to recognize and express the user's emotions. More sophisticated sign language simulation technologies tend to use facial expressions to assist in conveying emotions, but capturing and representing micro-expressions is costly.

[0070] In view of the above problems, this application aims to provide a new personalized optimization scheme for sign language performance. This new scheme provides users with a personalized optimization mode based on an existing vocabulary database. Taking into account the different movement speeds of sign language under different emotions, it allows users to perform batch optimization of other sign language words and gestures of similar duration within a certain parameter range through a single input of specific commonly used words.

[0071] In the multi-dimensional expression of information through sign language, emotions can be conveyed not only through facial expressions but also through the speed and intensity of sign language movements. Therefore, the batch optimization of the database in this application allows users to input the same sign language words under multiple emotions and record them multiple times. When generating sign language simulation performances, the corresponding emotions are also used to ensure that users receive multi-dimensional sign language information that aligns with their own communication habits. As follows:

[0072] (1) The sign language library is copied into multiple copies according to the emotion and labeled. Sign languages ​​with the same duration in a single sign language library are grouped.

[0073] (2) Using the speed of sign language as an important reference indicator for the expression of sign language emotions, extracting effective key frames of sign language entered by users, using the duration of key frames of a single word as the parameter value for emotion matching, and optimizing the details of sign language expression under different emotions accordingly.

[0074] (3) Freeze the individual sign language that has been entered and optimized, and optimize other sign languages ​​in the same group accordingly.

[0075] The following provides an exemplary application environment for this application.

[0076] like Figure 1 As shown, the exemplary application environment may include a server 2 and a user device 4 connected via a network.

[0077] A network can include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or more. A network can include physical links, such as coaxial cable links, twisted-pair cable links, fiber optic links, combinations thereof, and / or the like. A network can include wireless links, such as cellular links, satellite links, Wi-Fi links, and / or the like.

[0078] Server 2 configures a sign language library for each user (account) and provides updates and downloads for the sign language library. Server 2 can consist of one or more computing devices, such as rack servers, blade servers, tower servers, or cabinet servers (including standalone servers or server clusters composed of multiple servers). The one or more computer devices can include virtualized computing instances. The computer devices can load virtual machines based on virtual images and / or other data that define specific software (e.g., operating systems, dedicated applications, servers) used for emulation. As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on the one or more computer devices.

[0079] User device 4 can interact with server 2. Examples include: smartphones, tablets, laptops, smart devices (e.g., smartwatches, smart glasses), virtual reality devices, gaming devices, set-top boxes, digital streaming devices, in-vehicle terminals, smart TVs, TV boxes, MP4 (Moving Picture Experts Group Audio Layer IV) players, etc.

[0080] User device 4 can run Windows and Android systems. TM Operating systems such as iOS or Android can be used. Based on these systems, various applications can run, such as video recording and uploading programs.

[0081] It should be noted that, Figure 1 The server 2 and user equipment 4 shown are merely illustrative and are not intended to limit the scope of patent protection of this application. Any number of servers and user equipment can be included depending on actual needs.

[0082] The following will provide several embodiments in the above exemplary application environment to illustrate the optimization scheme of the sign language library.

[0083] Example 1

[0084] It should be noted that the optimization method described in this embodiment can be executed on server 2.

[0085] like Figure 2 As shown, the optimization method for this sign language library may include steps S200 to S208, wherein:

[0086] Step S200 It provides a sign language library with a built-in basic data table; wherein, the basic data table includes multiple basic sign language data, each basic sign language data includes a word and the sign language action and action execution time that the word maps to.

[0087] Server 2 provides each user with a sign language library. Users can customize their own sign language library configuration.

[0088] The data content for each basic sign language data point can be as follows:

[0089] {

[0090] Id: The unique identifier of this word in the sign language library;

[0091] Word: the word itself;

[0092] Type: Ti (the mood type assigned to the word);

[0093] Action: The sign language action Act0 corresponding to the word;

[0094] Time: The execution time of the sign language action Act0;

[0095] Describe: to describe;

[0096] }

[0097] Step S202 The sign language library provides a user data table for storing emotional sign language data with emotion types.

[0098] Users can personalize their sign language settings according to different emotions and personal sign language habits. The resulting personalized emotional sign language data is stored in a user data table. The data content of each emotional sign language record may include:

[0099] {

[0100] Index: Index number, with no actual meaning;

[0101] Id: The Id in the basic data table;

[0102] Word: the word itself;

[0103] Type: Ti (the mood type assigned to the word);

[0104] Action: The sign language action Act1 corresponding to the word;

[0105] Time: The execution time of sign language action Act1;

[0106] IsOptimized: A freeze flag indicating whether the data has been optimized.

[0107] }

[0108] Step S204 The system obtains sign language samples provided by target users, including sample words, sample emotion types, and sample sign language videos.

[0109] When the target user needs to customize a specific word under a certain emotion (such as the sample word W):

[0110] Target users can be:

[0111] (1) Record a video of yourself demonstrating the sign language for the sample word W;

[0112] (2) Input or select via user interface: the emotion (sample emotion type) when recording sample word W.

[0113] (3) Package the sample word W, the sample sign language video of the sample word W, and the sample emotion type and upload them to server 2.

[0114] The data protocol definition for uploading to server 2 is as follows:

[0115] {

[0116] ID: Protocol identifier;

[0117] Word: Sample words (text) uploaded this time;

[0118] Type: Ti (the sentiment type assigned to the sample words);

[0119] Movie: Sample sign language video;

[0120] }

[0121] Step S206 The sample sign language actions and their execution times are learned from the sample sign language videos.

[0122] Sample sign language actions (e.g., Act1) can be learned from the sample sign language videos using algorithms such as imitation learning and behavior cloning.

[0123] Imitation Learning-Behavior Cloning Algorithm: The inductive problem from provided examples is often simplified into a supervised learning problem, without having to make assumptions about the teacher's intentions. Based on supervised learning, this algorithm learns a robot's policy from instructional data. By using the state *s* at any moment of any expert instructional trajectory as the algorithm's input, and the corresponding action *a* as the algorithm's output, supervised learning algorithms can be applied to solve this imitation learning problem. For example, to perform a specified action by controlling the angles of seven joints of a robotic arm, the neural network's output can be designed to be the angle values ​​of these seven joints or the probability distribution of these angles. If a Gaussian distribution is used, the neural network can be designed to output the mean and variance of the corresponding angles. Using the above supervised learning approach to solve the imitation learning problem, ideally, the learned policy can reproduce the example behavior in new scenarios. This imitation learning-behavior cloning algorithm can be used to clone sign language animations from video sign language.

[0124] The speed at which a target user demonstrates sign language can also be identified based on the sample sign language video using image recognition and other methods.

[0125] In optional embodiments, such as Figure 3As shown, step S206, "learning sample sign language actions and sample action execution times based on the sample sign language video," may include: step S300, identifying the sample sign language video to obtain multiple sign language expression keyframes; step S302, obtaining the sample action execution time of the target user under the sample emotion type based on the multiple sign language expression keyframes. Using the speed of sign language as a reference indicator for sign language emotion expression, keyframes provided by the target user are extracted, and the duration of a single word's keyframe is used as a parameter value for emotion matching, thereby optimizing the details of sign language performance under different emotions. Specifically, keyframes in the sample sign language video can be identified using deep neural networks, and the time difference between the first and last identified keyframes can be calculated. This time difference is then determined as the sample action execution time of the target user in demonstrating the sample sign language. In this optional embodiment, the sample action execution time used by the target user to represent the sample word using sign language under the sample emotion type can be accurately obtained.

[0126] Step S208 The user data table is updated based on the sample words, sample emotion types, sample sign language actions, and sample action execution times.

[0127] Since the sign language library includes a basic data table and a user data table, updating and optimizing the user data table is equivalent to optimizing the sign language library.

[0128] Continuing with the example of sample word W, the process of updating the user data table can be as follows:

[0129] The sample emotion type of the sample word W is T1, the sample sign language action is Act1, and the sample action execution time is Time1.

[0130] (1) Find the corresponding ID, etc. from the basic database based on the sample word W.

[0131] (2) Update the sample word W, sample emotion type T1, sample sign language action Act1, sample action execution time Time1, freeze flag, etc. to the user data table to add the above-mentioned customized content of the target user.

[0132] Through the above update process, target users can update a large number of their own habitual emotional sign language expressions into the sign language library.

[0133] When providing sign language library services, the user data table has high priority, while the basic data table has low priority.

[0134] For example, the optimized sign language database (basic data table + updated user data table) provides the following services: when the target user's sign language information to be recognized is obtained, the user data table is used first to obtain the sign language text and emotion type corresponding to the sign language information to be recognized; when the data table cannot be used to obtain the sign language text, the basic data table is used to obtain the sign language text.

[0135] Based on the above, the optimization method based on sign language database provided in this embodiment has the following advantages:

[0136] (1) The sign language samples can be used to update the target user’s own habitual sign language to the user data table, that is, to personalize and optimize the sign language database, thereby helping the target user to quickly and accurately match the semantics of the sign language actions.

[0137] (2) Different emotion types correspond to different action rates (action execution times) for individual sign language movements. In this embodiment, based on the premise that the action rate of sign language movements differs under different emotions, server 2 can receive "sample sign language for sample words under the sample emotion type provided by the target user," obtain the sample word, sample emotion type, sample sign language movement, and sample action execution time based on the sample sign language, and then update the user data table based on the data obtained above. Therefore, when the sign language library provides services, it can prioritize matching emotion types and sign language movements / words through the user data table. If no match is found in the user data table, matching can be performed based on the basic data table. It should be noted that since the basic data table does not contain emotion types, it cannot match emotion types.

[0138] Compared to the high computational cost required to obtain the target user's emotion type through micro-expression recognition, the embodiment conveys emotions through the movement speed of sign language samples, reducing computational costs. When generating sign language simulation performances, matching emotions are used to allow the target user to receive multi-dimensional sign language information that conforms to their own expression habits.

[0139] The following provides some optional embodiments to further optimize the technical effect and solve new technical problems.

[0140] Related sign language optimization involves supplementing and correcting individual sign language expressions in a vocabulary database based on multiple samples obtained through machine learning. However, this optimization is inefficient and suffers from low utilization rates of corrected and supplemented words in real-world scenarios.

[0141] In optional embodiments, such as Figure 4As shown, the method may further include: step S400, dividing the multiple basic sign language data into multiple sign language groups based on the action execution time in each basic sign language data in the basic data table; step S402, copying the grouped basic data table multiple times to obtain multiple emotion data tables corresponding one-to-one with multiple emotion types. That is, sign languages ​​with the same duration in the basic data table are grouped, and the grouped basic data tables are copied multiple times according to emotion and labeled with emotion to obtain multiple emotion data tables. When sign language data for a certain emotion type is optimized, other sign language data in the same group within the same emotion data table can be optimized accordingly. In other words, when a target user provides sample sign language for a specific word of a specific emotion to optimize the sign language data for that specific word, other sign language data with similar durations can be automatically optimized in batches within a certain parameter range, resulting in high optimization efficiency.

[0142] It should be noted that the base data table can also be copied multiple times to obtain multiple emotion data tables that correspond one-to-one with multiple emotion types. Then, each of the multiple emotion data tables can be internally grouped according to the action execution time. In other embodiments, grouping can also be based on multiple parameters such as action execution time and action amplitude.

[0143] In optional embodiments, such as Figure 5 As shown, the above-mentioned batch optimization can be achieved through the following steps: Step S500, determine the target emotion data table from the multiple emotion data tables according to the sample emotion type; Step S502, determine the target sign language group where the sample word belongs from the target emotion data table; Step S504, adjust at least some words in the target sign language group to obtain optimized sign language data for each of the at least some words; Step S506, update the target user data table according to the optimized sign language data corresponding to each of the at least some words. In this optional embodiment, once the target user supplements and corrects a single sign language, it can automatically supplement and correct at least some other sign languages ​​accordingly, resulting in high optimization efficiency and high utilization in real-world scenarios.

[0144] In an optional embodiment, step S504, "adjusting at least some of the words in the target sign language group to obtain optimized sign language data for each of the at least some words," may further include: batch adjusting the action execution time of each word in the at least some words according to the sample action execution time.

[0145] Continuing with the example word W:

[0146] The sample emotion type of the sample word W is T1, the sample sign language action is Act1, and the sample action execution time is Time1.

[0147] Find words with emotion type T1 that are in the same sign language group as the sample word W (e.g., words X, Y, Z). These words can be batch optimized with reference to the sample word W.

[0148] Example 1: Taking word X as an example, the sign language data of word X is located in the user data table. Its basic sign language data in the basic data table is: emotion type T1, sign language action Act0, action execution time Time0.

[0149] The execution time of the action for word X can be adjusted from Time0 to time:

[0150] time = (Time1 / Time0)*, where θ is a preset value;

[0151] Where Time0 is the execution time of the action before word X was optimized, and time is the execution time of the action after word X was optimized.

[0152] Example 2: Taking word Y as an example, the sign language data for word Y already exists in the user data table.

[0153] The execution time of the action for word Y is directly adjusted to time in the user data table.

[0154] In an optional embodiment, the user data table includes multiple emotional sign language data corresponding to multiple sign languages. To ensure stability after optimization, during batch optimization, a single sign language that has already been entered and optimized is frozen, while others are optimized accordingly. For example, at least some of the emotional sign language data are configured with a freeze flag. The method may further include: identifying a specific word in the target group and excluding that specific word from the adjustment scope; wherein, the specific word is a word that is simultaneously located in the user data table and carries the freeze flag.

[0155] Continuing with the example word W:

[0156] The sample emotion type of the sample word W is T1, the sample sign language action is Act1, and the sample action execution time is Time1.

[0157] Find words with emotion type T1 that are in the same sign language group as the sample word W (e.g., words X, Y, Z). However, if, for example, the sign language data for word Z has a "frozen flag" set, then do not optimize for word Z.

[0158] In addition, to enable rapid optimization of the sign language library, some incentive modes can be provided, such as:

[0159] In an optional embodiment, the method further includes: pushing task requests for recording sign language samples to the target user and / or other users according to a preset pattern; wherein the task request includes the words to be recorded and the emotion type to be recorded corresponding to the sign language to be recorded. For example, words from a sign language library can be pushed to users daily as a daily task, allowing users to continuously update their preferred sign language samples for a high degree of customization.

[0160] In an optional embodiment, the method may further include: providing electronic rewards to the user responding to the task request. Upon completion of the task, the user may be awarded points, which can be redeemed for various visual sign language models, etc.

[0161] In an optional embodiment, the method may further include: pushing a sign language reading challenge task to the target user when the word optimization rate in the sign language library reaches a threshold ratio; evaluating the response of the target user to the sign language reading challenge task; and providing an electronic reward to the target user based on the evaluation result. For example, when the word optimization rate in the sign language library reaches a threshold ratio, a sign language reading challenge is provided, selecting long and difficult sentences for the user to understand (including semantics and emotion), and awarding points based on the final evaluation result (such as a score) to promote feedback and subsequent optimization.

[0162] In an optional embodiment, the method may further include: providing the target user's sign language library to a download platform; and providing the target user with corresponding electronic rewards based on the number of times the target user's sign language library is downloaded. For example, users can upload their optimized sign language libraries for other users to download, and uploaders (such as the target user) with high download rates and long cumulative usage time can receive points rewards.

[0163] like Figure 6 , 7 As shown, in order to make the technical solution of this application clearer and easier to understand, exemplary applications 1 and 2 are provided below.

[0164] Example Application 1:

[0165] S600 configures a sign language library for target users, which includes a basic data table, multiple emotion data tables, and a user data table.

[0166] The emotion data table is copied from the base data table, and different emotion data tables correspond to different emotion types.

[0167] Each emotion data table includes multiple sign language groups, and the execution time of the actions in the sign language data within the same sign language group is roughly the same.

[0168] S602, Obtain sign language samples uploaded by the target user, the sign language samples include sample words W, sample emotion types T1, and sample sign language videos.

[0169] S604 uses the behavior cloning algorithm to learn the sample sign language action Act1 and the sample action execution time Time1 from the sample sign language video.

[0170] S606, update the sample word W, sample emotion type T1, sample sign language action Act1, and sample action execution time Time1 to the user data table.

[0171] S608, determine the target emotion data table with emotion type T1 from multiple emotion data tables.

[0172] S610, determine the target sign language group to which the sample word W belongs from the target emotion data table.

[0173] S612, use the sample action execution time Time1 to optimize the action execution time (playback speed) of all unoptimized words in the target sign language group corresponding to the sign language actions.

[0174] The above describes the optimization process for the sign language library.

[0175] Use 1: First, use the user data table to query the matching sign language actions and the execution time of the actions to obtain the corresponding words and emotion types. If the query in the user data table fails, then use the basic data table to query.

[0176] Use 2: When a target user searches for the sign language action corresponding to a word, first use the user data table to query the sign language action with the emotion type for that word. If the required sign language action is not found, then use the basic data table to query.

[0177] Example Application 2:

[0178] S700 configures a sign language library for target users, which includes a basic data table and a user data table.

[0179] S702, based on the action execution time in each basic sign language data in the basic data table, divide the multiple basic sign language data into multiple sign language groups;

[0180] S704, copy the grouped basic data table multiple times to obtain multiple emotion data tables that correspond one-to-one with multiple emotion types.

[0181] Different emotion data tables correspond to different emotion types.

[0182] Each emotion data table includes multiple sign language groups, and the execution time of the actions in the sign language data within the same sign language group is roughly the same.

[0183] S706, Obtain sign language samples uploaded by the target user, the sign language samples include sample word W, sample emotion type T1 and sample sign language video.

[0184] Regularly push out words (single words) plus emotion types for target users to input.

[0185] Target users actively select commonly used words and emotion types for input.

[0186] Using video recording, target users, based on their own sign language habits, input emotions into the provided recognition mode (the gesture area specified on the shooting interface), record clear and high-quality sample sign language videos, and upload them.

[0187] S708 uses the behavior cloning algorithm to learn the sample sign language action Act1 from sample sign language videos.

[0188] S710, Identify the sample sign language video to obtain multiple sign language expression keyframes.

[0189] S712, based on the multiple sign language expression keyframes, obtain the sample action execution time Time1 of the target user under the sample emotion type T1.

[0190] S714, update the sample word W, sample emotion type T1, sample sign language action Act1, and sample action execution time Time1 to the user data table.

[0191] S716, proportionally optimizes the action execution time of all unoptimized words in the same emotion type and sign language group.

[0192] Specifically:

[0193] Identify the target emotion data table with emotion type T1 from multiple emotion data tables.

[0194] From the target emotion data table, determine the target sign language group to which the sample word W belongs.

[0195] Optimize the execution time (playback speed) of the sign language actions corresponding to all unoptimized words in the target sign language group proportionally.

[0196] S718, freeze the action execution time of sample word W in the user data table so that it will not be changed by the batch optimization of later entered words.

[0197] It should be noted that the core concepts of table copying, grouping, and freezing in this exemplary application can also be applied to the batch optimization of simulated animation data that require more precise parameter acquisition methods, such as the range of motion of individual sign language words and user lip movements, thereby enabling a more comprehensive and customized design for the sign language system.

[0198] By utilizing sample sign language examples corresponding to individual words entered by the user, the sign language database is optimized in batches for individual sign language expressions and their corresponding lip movements. Multiple copies of the sign language database are created to express different emotions. Users only need to enter specific words individually and selectively under emotion commands. The server will then uniformly adjust the sign language actions and lip movements based on the duration and amplitude of the actions within the same group. In other words, by leveraging users' sign language usage habits, the user's own sign language database is optimized in batches, thereby enhancing their efficiency in reading simulated sign language and their emotional perception.

[0199] Example 2

[0200] Figure 8 A block diagram schematically illustrates a sign language library optimization apparatus according to Embodiment 2 of this application. This sign language library optimization apparatus can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program modules referred to in the embodiments of this application refer to a series of computer program instruction segments capable of performing specific functions. The following description will specifically introduce the functions of each program module in this embodiment. For example... Figure 8 As shown, the sign language library optimization device 800 may include a first providing module 810, a second providing module 820, an acquisition module 830, a learning module 840, and an optimization module 850, wherein:

[0201] The first providing module 810 is used to provide a sign language library with a built-in basic data table; wherein, the basic data table includes multiple basic sign language data, each basic sign language data includes a word and the sign language action mapped to that word and the action execution time;

[0202] The second providing module 820 is used to provide a user data table in the sign language library, the user data table being used to store emotional sign language data with emotion types;

[0203] The acquisition module 830 is used to acquire sign language samples provided by the target user, the sign language samples including sample words, sample emotion types and sample sign language videos;

[0204] The learning module 840 is used to learn the sample sign language actions and the execution time of the sample actions based on the sample sign language video.

[0205] The optimization module 850 is used to update the user data table based on the sample words, sample emotion types, sample sign language actions, and sample action execution times.

[0206] When providing sign language library services, the user data table has high priority, while the basic data table has low priority.

[0207] In an optional embodiment, the device further includes a copying module (unidentified) for:

[0208] Based on the action execution time in each basic sign language data entry in the basic data table, the multiple basic sign language data entries are divided into multiple sign language groups;

[0209] The grouped base data table is copied multiple times to obtain multiple emotion data tables that correspond one-to-one with multiple emotion types.

[0210] In an optional embodiment, the device further includes a batch tuning module (unidentified), used for:

[0211] Based on the sample emotion type, determine the target emotion data table from the plurality of emotion data tables;

[0212] From the target emotion data table, determine the target sign language group to which the sample word belongs;

[0213] At least some of the words in the target sign language group are adjusted to obtain optimized sign language data for each of the at least some words;

[0214] The target user data table is updated based on the optimized sign language data corresponding to each word in at least some of the words.

[0215] In an optional embodiment, the batch tuning module is further configured to:

[0216] Based on the sample action execution time, the action execution time of each word in at least a portion of the words is adjusted in batches.

[0217] In an optional embodiment, the user data table includes multiple emotional sign language data entries, at least some of which are configured with a freeze flag; the device further includes an exclusion module (unidentified), used for:

[0218] Identify a specific word in the target group and exclude that specific word from the adjustment scope;

[0219] The specific word is a word that is simultaneously located in the user data table and carries the frozen identifier.

[0220] In an optional embodiment, the learning module is used for:

[0221] Identify the sample sign language video to obtain multiple sign language expression keyframes;

[0222] Based on the multiple sign language expression keyframes, the execution time of the target user's sample action under the sample emotion type is obtained.

[0223] In an optional embodiment, the device further includes a push module (unidentified), used for:

[0224] Based on a preset pattern, push task requests to record sign language samples to the target user and / or other users;

[0225] The task request includes the words to be recorded and the emotion types to be recorded corresponding to the sign language to be recorded.

[0226] In an optional embodiment, the device further includes a reward module (unidentified) for:

[0227] Electronic rewards are provided to users who respond to the task request.

[0228] In an optional embodiment, the device further includes a reward (unidentified) for:

[0229] When the vocabulary optimization rate in the sign language library reaches a threshold ratio, a sign language reading challenge task is pushed to the target user;

[0230] The responses from the target users to the sign language reading challenge are evaluated; and

[0231] Based on the evaluation results, electronic rewards will be provided to the target users.

[0232] In an optional embodiment, the device further includes a reward (unidentified) for:

[0233] Provide the target user's sign language database to the download platform; and

[0234] Based on the number of times the target user's sign language library is downloaded, a corresponding electronic reward is provided to the target user.

[0235] Example 3

[0236] Figure 9 This diagram schematically illustrates the hardware architecture of a computer device 10000 suitable for implementing a sign language library optimization method according to Embodiment 3 of this application. The computer device 10000 can be a server 2 or part of a server 2, or it can be a terminal device. The computer device 10000 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. For example, it can be a rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), etc. Figure 9 As shown, the computer device 10000 includes, but is not limited to, at least the following: a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked to each other via a system bus. Wherein:

[0237] The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as program code for optimizing the sign language library. In addition, the memory 10010 can also be used to temporarily store various types of data that have been output or will be output.

[0238] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0239] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external user terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external user terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0240] It should be pointed out that, Figure 9 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0241] In this embodiment, the method for optimizing the sign language library stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (processor 10020 in this embodiment) to complete the embodiment of this application.

[0242] Example 4

[0243] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the sign language library optimization method in Embodiment 1.

[0244] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the sign language library optimization method in this embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0245] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0246] It should be noted that the above are merely preferred embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A method of tuning a sign language library, characterized by, The method comprises the following steps: providing a sign language library with built-in basic data tables, wherein the basic data tables comprise a plurality of basic sign language data, each basic sign language data comprising a word and a sign language action and action execution time corresponding to the word; providing a user data table in the sign language library, the user data table being used to store emotional sign language data with an emotional type; obtaining a sign language sample provided by a target user, the sign language sample comprising a sample word, a sample emotional type, and a sample sign language video; learning a sample sign language action and a sample action execution time from the sample sign language video; updating the user data table according to the sample word, the sample emotional type, the sample sign language action, and the sample action execution time; wherein, in providing a sign language library service, the user data table has a high priority, and the basic data table has a low priority; The method further comprises: dividing the plurality of basic sign language data into a plurality of sign language groups according to the action execution time in each basic sign language data in the basic data table; copying the grouped basic data table multiple times to obtain a plurality of emotional data tables corresponding to a plurality of emotional types one by one; determining a target emotional data table from the plurality of emotional data tables according to the sample emotional type; determining a target sign language group in which the sample word is located from the target emotional data table; adjusting at least part of the words in the target sign language group to obtain optimized sign language data of each of the at least part of the words; and updating the target user data table according to the optimized sign language data corresponding to each of the at least part of the words.

2. The method of tuning a sign language library according to claim 1, wherein, The adjusting at least part of the words in the target sign language group to obtain optimized sign language data of each of the at least part of the words comprises: batch adjusting the action execution time of each of the at least part of the words according to the sample action execution time.

3. The method of tuning a sign language library according to claim 1, wherein, The user data table comprises a plurality of emotional sign language data, and at least part of the plurality of emotional sign language data is configured with a frozen identifier; the method further comprises: determining a specific word in the target sign language group, and excluding the specific word from the adjustment range; wherein, the specific word is a word located in the user data table at the same time, and carries the frozen identifier.

4. The method of tuning a sign language library according to any one of claims 1 to 3, characterized in that, The learning a sample sign language action and a sample action execution time from the sample sign language video comprises: identifying the sample sign language video to obtain a plurality of sign language expression key frames; obtaining the sample action execution time of the target user under the sample emotional type according to the plurality of sign language expression key frames.

5. The method of tuning a sign language library according to any one of claims 1 to 3, characterized in that, Further comprising: pushing a task request of recording a sign language sample to the target user and / or other users in a preset manner; wherein, the task request comprises a to-be-recorded word and a to-be-recorded emotional type corresponding to a to-be-recorded sign language.

6. The method of tuning a sign language library according to claim 5, wherein, Further comprising: providing an electronic reward for the user responding to the task request.

7. The method of tuning a sign language corpus according to any one of claims 1 to 3, wherein, Further comprising: pushing a sign language reading challenge task to the target user when the word library optimization rate in the sign language library reaches a threshold rate; evaluating the reply according to the target user's reply to the sign language reading challenge task; and providing an electronic reward for the target user according to the evaluation result.

8. The method of tuning a sign language corpus according to any one of claims 1 to 3, characterized in that, Further comprising: providing a sign language library of the target user to a download platform; and providing a corresponding electronic reward for the target user according to the download times of the sign language library of the target user.

9. An apparatus for tuning a sign language library, characterized by comprise: a first providing module configured to provide a sign language library of built-in basic data tables, wherein the basic data tables comprise a plurality of basic sign language data, and each basic sign language data comprises a word and a sign language action and an action execution time mapped with the word; a second providing module configured to provide a user data table with an emotion type in the sign language library, wherein the user data table is used to store emotion sign language data with an emotion type; an obtaining module configured to obtain a sign language sample provided by a target user, wherein the sign language sample comprises a sample word, a sample emotion type, and a sample sign language video; a learning module configured to learn a sample sign language action and a sample action execution time according to the sample sign language video; an optimizing module configured to update the user data table according to the sample word, the sample emotion type, the sample sign language action, and the sample action execution time; wherein, when providing a sign language library service, the user data table has a high priority, and the basic data table has a low priority; a copying module configured to divide the plurality of basic sign language data into a plurality of sign language groups according to the action execution time in each basic sign language data in the basic data table, and copy the grouped basic data table a plurality of times to obtain a plurality of emotion data tables corresponding to a plurality of emotion types one by one; a batch optimization module configured to determine a target emotion data table from the plurality of emotion data tables according to the sample emotion type, determine a target sign language group in which the sample word is located from the target emotion data table, adjust at least part of the words in the target sign language group to obtain optimized sign language data of each of the at least part of the words, and update the target user data table according to the optimized sign language data corresponding to each of the at least part of the words.

10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor is configured to execute the computer program to implement the steps of the sign language library optimization method in any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executable by at least one processor to enable the at least one processor to execute the steps of the sign language library optimization method in any one of claims 1 to 8.

12. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executable by at least one processor to enable the at least one processor to execute the steps of the sign language library optimization method in any one of claims 1 to 8.

Images