A data processing method, a data processing device and a data processing device
Patent Information
- Application Number
- CN202110821632.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-20
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-07-20
AI Technical Summary
[0004]然而,在实际应用中,手语作为视觉语言,针对同一个手语词汇,不同表达者的呈现的手语动作的差异度极大;且由于通用手语普及时间晚,存在南北地域差异;再者,对于专用领域的复杂专业手语词汇,通常为各个普通词汇的排列组合,这种情况下手语翻译工具或手语合成工具的处理结果往往准确度较低
[0063] After acquiring the sign language action data to be processed, the embodiment of the present invention performs element analysis on the sign language action data based on the pre-constructed sign language annotation system to determine the sign language annotation sequence corresponding to the sign language action data. The sign language annotation sequence includes all annotation information of each basic sign language element corresponding to the sign language action data, and then perform operation processing on the sign language action data according to the sign language annotation sequence. The embodiment of the present invention performs element analysis on the sign language action data to be processed through a pre-built sign language annotation system to determine the basic sign language elements contained in the sign language action data, that is, to disassemble the sign language action data to be processed into the smallest feature units, which can be applied to various applications. The analysis and processing of sign language-like movement data is conducive to improving the analysis of sign language movement data. Analysis accuracy; and, by determining the annotation information of each basic sign language element in the sign language action data, the embodiment of the present invention converts the sign language action data into a sign language annotation sequence that can be automatically recognized by the machine, and then performs operations on the sign language action data based on the sign language annotation sequence. processing, thereby improving the processing efficiency and accuracy of sign language action data.
Smart Images

Figure CN113657101B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, device and a device for data processing. Background Art
[0002] Sign language is one of the ways for hearing-impaired or speech-deprived special users to communicate, but hearing users often do not understand sign language, which makes communication between special users and ordinary users very difficult.
[0003] At present, ordinary users and special users can communicate with each other with the help of sign language translation tools or sign language synthesis tools. Taking Chinese as an example, the sign language translation tool can recognize the text input by ordinary users and convert the text into the sign language movements corresponding to the sign language vocabulary; or the sign language synthesis tool can synthesize the corresponding sign language animation according to the text input by ordinary users; or the sign language translation tool can convert the sign language movements made by special users into corresponding text, and so on.
[0004] However, in actual applications, sign language is a visual language, and for the same sign language vocabulary, the sign language movements presented by different speakers vary greatly. In addition, due to the late popularization of universal sign language, there are regional differences between the north and the south. Furthermore, for complex professional sign language vocabulary in special fields, it is usually a combination of various ordinary vocabulary. In this case, the processing results of sign language translation tools or sign language synthesis tools are often less accurate. Summary of the invention
[0005] The embodiments of the present invention provide a data processing method, a device and a device for data processing, which can be applied to the analysis and processing of various types of sign language motion data, and improve the processing efficiency and accuracy of sign language motion data.
[0006] In order to solve the above problem, an embodiment of the present invention discloses a data processing method, which includes:
[0007] Obtaining sign language action data to be processed;
[0008] Performing element analysis on the sign language motion data based on a pre-constructed sign language annotation system to determine a sign language annotation sequence corresponding to the sign language motion data, wherein the sign language annotation sequence includes annotation information of each basic sign language element corresponding to the sign language motion data;
[0009] An operation process is performed on the sign language motion data according to the sign language annotation sequence.
[0010] Optionally, before performing element analysis on the sign language action data based on the pre-built sign language annotation system, the method further includes:
[0011] Based on the linguistic framework, the sign language action data in the database is disassembled and classified to obtain the basic sign language elements and the element types corresponding to each basic sign language element;
[0012] A sign language annotation system is constructed according to the basic sign language elements and the element types corresponding to each basic sign language element. The sign language annotation system includes annotation information corresponding to each element type of each basic sign language element.
[0013] Optionally, the basic sign language elements include at least one of left and right arm features, single and double hand shape features, directional movement features, knuckle bending angles, facial expression features, and constraint information.
[0014] Optionally, the step of disassembling and classifying the sign language action data in the database based on the linguistic framework to obtain basic sign language elements and element types corresponding to each basic sign language element includes:
[0015] Traversing each sign language action data in the database, performing action decomposition on each sign language action data, and determining the key parts corresponding to each sign language action data and the action features of the key parts;
[0016] Based on the linguistic framework, the key parts corresponding to each sign language action data in the database and the action features of the key parts are classified and processed to obtain at least 2 clusters, each of which corresponds to a basic sign language element;
[0017] The element type of the basic sign language element corresponding to each cluster is determined according to the action features included in the cluster.
[0018] Optionally, the action feature includes at least one of rotation data, displacement data, bending angle, key feature, and expression feature.
[0019] Optionally, performing element analysis on the sign language action data based on a pre-constructed sign language annotation system to determine a sign language annotation sequence corresponding to the sign language action data includes:
[0020] Performing element analysis on the sign language motion data to determine a first basic sign language element corresponding to the sign language motion data, a first element type of the first basic sign language element, and a first timestamp;
[0021] Determining first annotation information of the first basic sign language element and second annotation information of the first element type based on a pre-constructed sign language annotation system;
[0022] A sign language annotation sequence corresponding to the sign language action data is determined according to the first timestamp, the first annotation information, and the second annotation information.
[0023] Optionally, performing operation processing on the sign language action data according to the sign language annotation sequence includes:
[0024] Based on the sign language annotation sequence, a pre-built three-dimensional character model is driven to perform the sign language action corresponding to the sign language action data; and / or,
[0025] The sign language action data is subjected to sign language translation processing based on the sign language annotation sequence to obtain a target text sequence corresponding to the sign language action data.
[0026] Optionally, performing element analysis on the sign language action data based on a pre-constructed sign language annotation system to determine a sign language annotation sequence corresponding to the sign language action data includes:
[0027] Performing word segmentation processing on the text sequence to be translated to obtain a natural vocabulary sequence corresponding to the text sequence;
[0028] Determine the second basic sign language element corresponding to each natural word in the natural word sequence and the second element type corresponding to the second basic sign language element;
[0029] sorting the second basic sign language elements and the second element types to generate a sign language element sequence that complies with sign language grammar rules;
[0030] Determining a sign language annotation sequence corresponding to the sign language element sequence based on the sign language annotation system;
[0031] The performing operation processing on the sign language action data according to the sign language annotation sequence includes:
[0032] The text sequence is subjected to sign language translation processing according to the sign language annotation sequence to obtain sign language actions corresponding to the text sequence.
[0033] On the other hand, an embodiment of the present invention discloses a data processing device, the device comprising:
[0034] A sign language motion data acquisition module, used to acquire the sign language motion data to be processed;
[0035] A sign language annotation sequence determination module, configured to perform element analysis on the sign language motion data based on a pre-constructed sign language annotation system, and determine a sign language annotation sequence corresponding to the sign language motion data, wherein the sign language annotation sequence includes annotation information of each basic sign language element corresponding to the sign language motion data;
[0036] An operation processing execution module is used to perform operation processing on the sign language action data according to the sign language annotation sequence.
[0037] Optionally, the device further comprises:
[0038] A basic sign language element determination module is used to disassemble and classify the sign language action data in the database based on a linguistic framework to obtain basic sign language elements and the element type corresponding to each basic sign language element;
[0039] The sign language annotation system construction module is used to construct a sign language annotation system according to the basic sign language elements and the element types corresponding to each basic sign language element. The sign language annotation system includes annotation information corresponding to each element type of each basic sign language element.
[0040] Optionally, the basic sign language elements include at least one of left and right arm features, single and double hand shape features, directional movement features, knuckle bending angles, facial expression features, and constraint information.
[0041] Optionally, the basic sign language element determination module includes:
[0042] The motion data analysis submodule is used to traverse each sign language motion data in the database, decompose each sign language motion data, determine the key part corresponding to each sign language motion data, and the motion characteristics of the key part;
[0043] The action feature classification submodule is used to classify the key parts corresponding to each sign language action data in the database and the action features of the key parts based on the linguistic framework to obtain at least two clusters, each cluster corresponding to a basic sign language element;
[0044] The element type determination submodule is used to determine the element type of the basic sign language element corresponding to each cluster according to the action features contained in the cluster.
[0045] Optionally, the action feature includes at least one of rotation data, displacement data, bending angle, key feature, and expression feature.
[0046] Optionally, the sign language annotation sequence determination module includes:
[0047] A first element determination submodule, configured to perform element analysis on the sign language action data to determine a first basic sign language element corresponding to the sign language action data, a first element type of the first basic sign language element, and a first timestamp;
[0048] A marking information determination submodule, configured to determine first marking information of the first basic sign language element and second marking information of the first element type based on a pre-constructed sign language marking system;
[0049] The first annotation sequence determining submodule is used to determine the sign language annotation sequence corresponding to the sign language action data according to the first timestamp, the first annotation information and the second annotation information.
[0050] Optionally, the operation processing execution module includes:
[0051] A first operation processing submodule is used to drive a pre-built three-dimensional character model to perform a sign language action corresponding to the sign language action data based on the sign language annotation sequence; and / or,
[0052] The second operation processing submodule is used to perform sign language translation processing on the sign language action data based on the sign language annotation sequence to obtain a target text sequence corresponding to the sign language action data.
[0053] Optionally, the sign language annotation sequence determination module includes:
[0054] A word segmentation processing submodule is used to perform word segmentation processing on the text sequence to be translated to obtain a natural vocabulary sequence corresponding to the text sequence;
[0055] A second element determination submodule, used to determine the second basic sign language element corresponding to each natural word in the natural word sequence and the second element type corresponding to the second basic sign language element;
[0056] An element sequence generating submodule, used for sorting the second basic sign language elements and the second element types to generate a sign language element sequence that conforms to the sign language grammar rules;
[0057] A second annotation sequence determination submodule, configured to determine a sign language annotation sequence corresponding to the sign language element sequence based on the sign language annotation system;
[0058] The operation processing execution module includes:
[0059] The third operation processing submodule is used to perform sign language translation processing on the text sequence according to the sign language annotation sequence to obtain the sign language action corresponding to the text sequence.
[0060] On the other hand, an embodiment of the present invention discloses a device for data processing, which includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for executing one or more of the data processing methods described above.
[0061] On the other hand, an embodiment of the present invention discloses a machine-readable medium having instructions stored thereon, which, when executed by one or more processors, enables the device to perform one or more of the data processing methods described above.
[0062] The embodiments of the present invention include the following advantages:
[0063] After acquiring the sign language motion data to be processed, the embodiment of the present invention performs element analysis on the sign language motion data based on a pre-constructed sign language annotation system, determines the sign language annotation sequence corresponding to the sign language motion data, and the sign language annotation sequence contains the annotation information of each basic sign language element corresponding to the sign language motion data, and then performs operation processing on the sign language motion data according to the sign language annotation sequence. The embodiment of the present invention performs element analysis on the sign language motion data to be processed through a pre-constructed sign language annotation system, determines the basic sign language elements contained in the sign language motion data, that is, decomposes the sign language motion data to be processed into the smallest feature unit, which can be applied to the analysis and processing of various types of sign language motion data, and is conducive to improving the analysis accuracy of the sign language motion data; and the embodiment of the present invention converts the sign language motion data into a sign language annotation sequence that can be automatically recognized by a machine by determining the annotation information of each basic sign language element in the sign language motion data, and then performs operation processing on the sign language motion data based on the sign language annotation sequence, thereby improving the processing efficiency and accuracy of the sign language motion data. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0065] Figure 1 is a flow chart of steps of an embodiment of a data processing method of the present invention;
[0066] Figure 2 is a structural block diagram of an embodiment of a data processing device of the present invention;
[0067] Figure 3 is a block diagram of an apparatus 800 for data processing according to the present invention;
[0068] Figure 4 It is a schematic diagram of the structure of the server in some embodiments of the present invention. DETAILED DESCRIPTION
[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0070] Method Embodiment
[0071] Reference Figure 1 , shows a flow chart of steps of a data processing method embodiment of the present invention, the method may specifically include the following steps:
[0072] Step 101: Obtain sign language action data to be processed.
[0073] Step 102: Perform element analysis on the sign language motion data based on a pre-constructed sign language annotation system to determine a sign language annotation sequence corresponding to the sign language motion data, wherein the sign language annotation sequence includes annotation information of each basic sign language element corresponding to the sign language motion data.
[0074] Step 103: perform operation processing on the sign language action data according to the sign language annotation sequence.
[0075] The data processing method provided in the embodiment of the present invention can be applied to a terminal device or a server. The terminal device may include, but is not limited to, a dedicated sign language translation device, a sign language action synthesis device, a smart terminal, a computer, a personal digital assistant (PDA), a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, a vehicle-mounted device, a smart TV, a wearable device, etc.
[0076] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, cloud communications, network services, middleware services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.
[0077] It should be noted that the sign language action data obtained in the embodiments of the present invention may be a sign language video and / or a sign language image containing sign language actions. The sign language video and / or the sign language image may be a video and / or an image captured by the terminal device through a camera when the target object (including hearing-impaired, deaf-mute, hearing person) uses sign language to communicate with others or machines or transmit information to others, or may be a sign language video and / or a sign language image stored in the terminal device and / or the server, or may be a sign language video and / or a sign language image downloaded from a network terminal.
[0078] The sign language annotation system constructed in the embodiment of the present invention includes basic sign language elements corresponding to each sign language action, each basic sign language element includes at least one element type, the basic sign language element can be understood as the smallest characteristic unit of the sign language action, and any sign language action can be composed of one or more basic sign language elements. The sign language annotation system also includes the annotation information corresponding to the basic sign language elements and the element types of the basic sign language elements. Based on the sign language annotation system, the annotation information of each basic sign language element corresponding to the sign language action data to be processed can be determined, that is, the sign language annotation sequence corresponding to the sign language action data. Among them, the sign language annotation sequence can also include the timestamps corresponding to each basic sign language element of the sign language action data, so that when the operation processing is performed on the sign language action data, the processing order of each basic sign language element is determined according to the timestamps corresponding to each basic sign language element. For example, based on the annotation information and timestamps of each basic sign language element in the sign language annotation sequence, the digital human (that is, the three-dimensional character model constructed using 3D technology) is driven to perform the sign language action corresponding to the sign language action data.
[0079] The embodiment of the present invention uses a pre-constructed sign language annotation system to perform element analysis on the sign language motion data to be processed, and determines the basic sign language elements contained in the sign language motion data, that is, disassembles the sign language motion data to be processed into minimum feature units, which can be applicable to the analysis and processing of various types of sign language motion data, and is beneficial to improving the analysis accuracy of the sign language motion data; and, the embodiment of the present invention determines the annotation information of each basic sign language element in the sign language motion data, converts the sign language motion data into a sign language annotation sequence that can be automatically recognized by a machine, and then performs operation processing on the sign language motion data based on the sign language annotation sequence, which can improve the processing efficiency and accuracy of the sign language motion data.
[0080] In an optional embodiment of the present invention, before performing element analysis on the sign language action data based on a pre-built sign language annotation system in step 102, the method further includes:
[0081] Step S11, disassemble and classify the sign language action data in the database based on the linguistic framework to obtain basic sign language elements and the element type corresponding to each basic sign language element;
[0082] Step S12: construct a sign language annotation system according to the basic sign language elements and the element types corresponding to each basic sign language element, wherein the sign language annotation system includes annotation information corresponding to each element type of each basic sign language element.
[0083] The sign language action data in the database include sign language action data used for daily life interaction and special sign language action data in professional fields. The sign language action data can be descriptive text, pictures, sign language action videos, sign language animations, etc. for sign language actions. A sign language action database can be constructed by searching for various sign language materials, such as sign language teaching videos, online sign language action materials, sign language dictionaries, etc.
[0084] Based on the linguistic framework, all the sign language action data in the database are disassembled and classified, all the basic sign language elements involved in the database and the element types corresponding to each basic sign language element are determined, and the annotation information corresponding to each element type of each basic sign language element is determined, so as to construct a sign language annotation system. Among them, the annotation information is used to uniquely identify each element type of each basic sign language element, and the annotation information can be recognized by a machine, has broad representativeness and universality, and is suitable for most sign language application scenarios.
[0085] The basic sign language elements include at least one of left and right arm features, single and double hand shape features, directional movement features, knuckle bending angles, facial expression features, and constraint information. Different basic sign language elements represent sign language action features of different dimensions. Each basic sign language element corresponds to multiple element types. For example, in the present invention, which is an exemplary sign language annotation system, single and double hand shape features correspond to 67 different element types, directional movement features correspond to 66 different element types, constraint information corresponds to 57 different element types, facial expression features correspond to 8 different element types, and so on. By arranging and combining different types of basic sign language elements, different sign language actions can be obtained.
[0086] It should be noted that the sign language annotation system in the embodiment of the present invention can be continuously expanded and improved. The more detailed the sign language annotation system is and the more comprehensive the data it contains, the more conducive it is to improving the processing efficiency of sign language action data.
[0087] In addition, the sign language annotation system constructed based on the embodiment of the present invention can also be used to build and improve the sign language database, thereby reducing the construction cost of the sign language database, improving the data coverage of the sign language database, and providing a reliable linguistic theoretical basis for processing operations of sign language action data such as sign language synthesis and sign language translation.
[0088] In an optional embodiment of the present invention, the step S11 decomposes and classifies the sign language action data in the database based on the linguistic framework to obtain basic sign language elements and the element type corresponding to each basic sign language element, including:
[0089] Sub-step S111, traversing each sign language action data in the database, performing action decomposition on each sign language action data, determining the key part corresponding to each sign language action data, and the action characteristics of the key part;
[0090] Sub-step S112: classifying the key parts corresponding to each sign language action data in the database and the action features of the key parts based on a linguistic framework to obtain at least two clusters, each of which corresponds to a basic sign language element;
[0091] Sub-step S113: determining the element type of the basic sign language element corresponding to each cluster according to the action features contained in the cluster.
[0092] In the embodiment of the present invention, for each type of sign language action data in the database, the action is disassembled one by one to determine the key parts corresponding to each sign language action data and the action features of each key part. The key parts may include arms, palms, fingers, faces and other parts, and the action features may include at least one of rotation data, displacement data, bending angle, key features, and expression features.
[0093] By classifying all the key parts and their motion features, we can get multiple clusters, each of which corresponds to a basic sign language element. For example, when decomposing sign language movements, we can divide the sign language movement features into six dimensions, namely left and right arm features, single and double hand shape features, direction movement features, knuckle bending angle, facial expression features, and constraint information. Among them, the left and right arm features may include arm features such as the degree of arm bending in the sign language action, whether it is in an upright state or a drooping state; the single and double hand shape features may include the shape and posture features of each finger; the direction movement features may include features such as the palm direction, the movement of the arm and the palm; the degree of knuckle bending is used to indicate the bending condition of each knuckle in the finger, for example, the third knuckle of the left index finger is bent 45°, and so on; the facial expression features refer to specific facial expressions in the sign language action, such as smile, frustration, surprise, etc.; the constraint information may include the contact condition of key parts in the sign language action, specific time limit and other information. For example, in the sign language action of the sign language word "love", the constraint information is "the right hand palm is in contact with the top knuckle of the left thumb", or, in the sign language action of the sign language word "red", the constraint information is "the right index finger and middle finger touch the lips", and so on.
[0094] After determining the cluster corresponding to each basic sign language element, the element type of the basic sign language element corresponding to the cluster can be determined according to each action feature contained in each cluster. Specifically, each action feature in each cluster can be used as an element type. By arranging and combining the basic sign language elements and element types, each sign language action can be conveniently represented.
[0095] In an optional embodiment of the present invention, the step 102 of performing element analysis on the sign language action data based on a pre-built sign language annotation system to determine a sign language annotation sequence corresponding to the sign language action data includes:
[0096] Step S21, performing element analysis on the sign language action data to determine a first basic sign language element corresponding to the sign language action data, a first element type of the first basic sign language element, and a first timestamp;
[0097] Step S22: determining first annotation information of the first basic sign language element and second annotation information of the first element type based on a pre-constructed sign language annotation system;
[0098] Step S23: Determine a sign language annotation sequence corresponding to the sign language action data according to the first timestamp, the first annotation information, and the second annotation information.
[0099] In the embodiment of the present invention, after obtaining the sign language motion data to be processed, the sign language motion data can be firstly subjected to element analysis to determine the basic sign language elements and element types of the basic sign language elements contained in the sign language motion data, that is, to determine the first basic sign language element, the first element type of the first basic sign language element, and the timestamp of each first basic sign language element. By analyzing the first basic sign language element and the first element type in the sign language motion data, the minimum feature unit of the sign language motion data can be determined, which is convenient for subsequent analysis and processing of the sign language motion data.
[0100] The sign language annotation system contains annotation information corresponding to each basic sign language element and each element type. The sign language annotation system is equivalent to constructing annotation criteria for sign language action data, which can be applied to any scene and any sign language action data, and has wide applicability. The first annotation information of each first basic sign language element in the sign language action data and the second annotation information of the first element type can be searched in the sign language annotation system, and then a sign language annotation sequence of the sign language action data is generated according to the first annotation information and the second annotation information. The sign language annotation sequence corresponds to the sign language action data one by one, and the sign language action corresponding to the sign language action data can be determined based on the sign language annotation sequence. In addition, the sign language annotation sequence can be recognized by a machine, so that the sign language action data can be subjected to sign language translation processing, sign language synthesis processing, etc. based on the sign language annotation sequence of the sign language action data.
[0101] For example, for the sign language word "love", we first perform an element analysis on it to determine its corresponding basic sign language elements and the element types corresponding to each basic sign language element, and then determine the corresponding annotation information according to the sign language annotation system, as shown in Table 1.
[0102] Table 1
[0103]
[0104]
[0105] According to the annotation information corresponding to each basic sign language element and element type in the above table, the sign language annotation sequence of the sign language word "love" can be determined. It should be noted that the annotation information shown in Table 1 is only an exemplary description of the present invention, and does not constitute a limitation on the corresponding relationship between the annotation information of the sign language annotation system in the embodiment of the present invention and each basic sign language element and element type. The corresponding annotation criteria can be formulated according to actual needs, and the present invention does not make specific limitations on this.
[0106] In an optional embodiment of the present invention, the step 103 of performing operation processing on the sign language action data according to the sign language annotation sequence includes:
[0107] Step S31, driving a pre-built three-dimensional character model to perform a sign language action corresponding to the sign language action data based on the sign language annotation sequence; and / or,
[0108] Step S32: performing sign language translation processing on the sign language action data based on the sign language annotation sequence to obtain a target text sequence corresponding to the sign language action data.
[0109] After determining the sign language annotation sequence of the sign language action data to be processed, the three-dimensional character model can be further driven to perform the sign language action corresponding to the sign language action data based on the sign language annotation sequence, that is, the sign language action is synthesized through 3D technology; or, the sign language action data can be processed by sign language translation based on the sign language annotation sequence. Specifically, the sign language annotation sequence is input into a pre-trained sign language translation model to obtain a target text sequence corresponding to the sign language action data. The sign language translation model can be trained based on the sign language annotation system constructed by the present invention. The trained sign language translation model can accurately identify the sign language actions corresponding to each sign language annotation sequence, and then translate the identified sign language actions, thereby improving the accuracy of sign language translation.
[0110] In addition, in addition to translating the sign language action data into a target text sequence, the text sequence can also be analyzed and processed based on the sign language annotation system of the present invention to convert the text sequence into corresponding sign language actions.
[0111] In an optional embodiment of the present invention, the step 102 of performing element analysis on the sign language action data based on a pre-built sign language annotation system to determine a sign language annotation sequence corresponding to the sign language action data includes:
[0112] Step S41, performing word segmentation processing on the text sequence to be translated to obtain a natural vocabulary sequence corresponding to the text sequence;
[0113] Step S42, determining the second basic sign language elements corresponding to each natural word in the natural word sequence and the second element types corresponding to the second basic sign language elements;
[0114] Step S43, sorting the second basic sign language elements and the second element types to generate a sign language element sequence that complies with sign language grammar rules;
[0115] Step S44, determining a sign language annotation sequence corresponding to the sign language element sequence based on the sign language annotation system;
[0116] Step 103 of performing operation processing on the sign language action data according to the sign language annotation sequence includes:
[0117] Step S45: performing sign language translation processing on the text sequence according to the sign language annotation sequence to obtain sign language actions corresponding to the text sequence.
[0118] It should be noted that the embodiment of the present invention does not limit the language of the text sequence to be translated. For example, the language of the text sequence to be translated may be Chinese, English, Korean, etc. When converting the text sequence into sign language movements, texts in various languages may be converted into sign language movements of the corresponding countries, and texts in the source language may be converted into sign language movements of the corresponding countries of the target language.
[0119] The natural vocabulary specifically refers to the vocabulary used by users with sound hearing and language expression ability. Taking the language of the text sequence to be translated as Chinese as an example, the natural vocabulary can be the Chinese vocabulary included in the "Modern Chinese Dictionary" and "Xinhua Dictionary".
[0120] The natural vocabulary corresponds to the sign language vocabulary. According to the sign language vocabulary corresponding to the natural vocabulary, the basic sign language elements and element types corresponding to the natural vocabulary can be determined. For example, for the natural vocabulary "love", the basic sign language element information of the corresponding sign language vocabulary "love" is shown in Table 1. Since there is a sequence between the basic sign language elements and element types in the sign language action, the correct action can only be obtained based on the corresponding sequence. Therefore, after determining the basic sign language elements and element types corresponding to the natural vocabulary sequence, it is necessary to further sort the basic sign language elements and element types to generate a sign language element sequence. Then, based on the sign language annotation system, the sign language annotation sequence corresponding to the sign language element sequence can be determined. For example, for the natural vocabulary "love", based on Table 1, it can be determined that its corresponding sign language annotation sequence is "T0-T0_O60,IMRP0-Still_03-Straight_06-C(DR,T4L)-OR_L_right-OR_R_down-simile". The text sequence is processed by sign language translation based on the sign language annotation sequence to obtain the sign language action. The sign language action can be a sign language action picture or a demonstration animation of the sign language action. For example, after determining the sign language sequence corresponding to the text sequence through the sign language translation model, the sign language words are converted into a sign language annotation sequence to obtain the corresponding sign language action; or the three-dimensional character model can be driven to perform the corresponding sign language action directly based on the sign language action annotation sequence.
[0121] In addition, the embodiment of the present invention can also convert non-text content into corresponding sign language movements. The non-text content can be voice or picture. Specifically, the embodiment of the present invention can perform text recognition processing on the voice to be translated to obtain a text sequence corresponding to the voice, and then execute the aforementioned steps S41 to S45 on the obtained text sequence to obtain the sign language movement corresponding to the voice. Alternatively, the embodiment of the present invention can perform image recognition and text recognition on the picture to be translated to obtain a text sequence corresponding to the picture, and execute the aforementioned steps S41 to S45 on the obtained text sequence to obtain the sign language movement corresponding to the picture. It can be understood that the embodiment of the present invention does not specifically limit the specific content of the picture. For example, the picture may include text, graphics, expressions and other content.
[0122] In summary, the embodiment of the present invention performs element analysis on the sign language motion data to be processed through a pre-constructed sign language annotation system to determine the basic sign language elements contained in the sign language motion data, that is, to decompose the sign language motion data to be processed into standardized basic sign language motion units, which can be applicable to the analysis and processing of various types of sign language motion data, and is conducive to improving the analysis accuracy of the sign language motion data; and, the embodiment of the present invention determines the annotation information of each basic sign language element in the sign language motion data, converts the sign language motion data into a machine-trainable sign language annotation sequence, and then performs operation processing on the sign language motion data based on the sign language annotation sequence, thereby improving the processing efficiency and accuracy of the sign language motion data.
[0123] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0124] Device Embodiment
[0125] Reference Figure 2 , shows a structural block diagram of an embodiment of a data processing device of the present invention, the device may include:
[0126] A sign language motion data acquisition module 201 is used to acquire the sign language motion data to be processed;
[0127] A sign language annotation sequence determination module 202 is used to perform element analysis on the sign language motion data based on a pre-constructed sign language annotation system to determine a sign language annotation sequence corresponding to the sign language motion data, wherein the sign language annotation sequence includes annotation information of each basic sign language element corresponding to the sign language motion data;
[0128] The operation processing execution module 203 is used to perform operation processing on the sign language action data according to the sign language annotation sequence.
[0129] Optionally, the device further comprises:
[0130] A basic sign language element determination module is used to disassemble and classify the sign language action data in the database based on a linguistic framework to obtain basic sign language elements and the element type corresponding to each basic sign language element;
[0131] The sign language annotation system construction module is used to construct a sign language annotation system according to the basic sign language elements and the element types corresponding to each basic sign language element. The sign language annotation system includes annotation information corresponding to each element type of each basic sign language element.
[0132] Optionally, the basic sign language elements include at least one of left and right arm features, single and double hand shape features, directional movement features, knuckle bending angles, facial expression features, and constraint information.
[0133] Optionally, the basic sign language element determination module includes:
[0134] The motion data analysis submodule is used to traverse each sign language motion data in the database, decompose each sign language motion data, determine the key part corresponding to each sign language motion data, and the motion characteristics of the key part;
[0135] The action feature classification submodule is used to classify the key parts corresponding to each sign language action data in the database and the action features of the key parts based on the linguistic framework to obtain at least two clusters, each cluster corresponding to a basic sign language element;
[0136] The element type determination submodule is used to determine the element type of the basic sign language element corresponding to each cluster according to the action features contained in the cluster.
[0137] Optionally, the action feature includes at least one of rotation data, displacement data, bending angle, key feature, and expression feature.
[0138] Optionally, the sign language annotation sequence determination module includes:
[0139] A first element determination submodule, configured to perform element analysis on the sign language action data to determine a first basic sign language element corresponding to the sign language action data, a first element type of the first basic sign language element, and a first timestamp;
[0140] A marking information determination submodule, configured to determine first marking information of the first basic sign language element and second marking information of the first element type based on a pre-constructed sign language marking system;
[0141] The first annotation sequence determining submodule is used to determine the sign language annotation sequence corresponding to the sign language action data according to the first timestamp, the first annotation information and the second annotation information.
[0142] Optionally, the operation processing execution module includes:
[0143] A first operation processing submodule is used to drive a pre-built three-dimensional character model to perform a sign language action corresponding to the sign language action data based on the sign language annotation sequence; and / or,
[0144] The second operation processing submodule is used to perform sign language translation processing on the sign language action data based on the sign language annotation sequence to obtain a target text sequence corresponding to the sign language action data.
[0145] Optionally, the sign language annotation sequence determination module includes:
[0146] A word segmentation processing submodule is used to perform word segmentation processing on the text sequence to be translated to obtain a natural vocabulary sequence corresponding to the text sequence;
[0147] A second element determination submodule, used to determine the second basic sign language element corresponding to each natural word in the natural word sequence and the second element type corresponding to the second basic sign language element;
[0148] An element sequence generating submodule, used for sorting the second basic sign language elements and the second element types to generate a sign language element sequence that conforms to the sign language grammar rules;
[0149] A second annotation sequence determination submodule, configured to determine a sign language annotation sequence corresponding to the sign language element sequence based on the sign language annotation system;
[0150] The operation processing execution module includes:
[0151] The third operation processing submodule is used to perform sign language translation processing on the text sequence according to the sign language annotation sequence to obtain the sign language action corresponding to the text sequence.
[0152] In summary, the embodiment of the present invention performs element analysis on the sign language motion data to be processed through a pre-constructed sign language annotation system, determines the basic sign language elements contained in the sign language motion data, that is, decomposes the sign language motion data to be processed into standardized basic sign language motion units, which can be applied to the analysis and processing of various sign language motion data, and is beneficial to improving the analysis accuracy of the sign language motion data; and, the embodiment of the present invention determines the annotation information of each basic sign language element in the sign language motion data, converts the sign language motion data into a sign language annotation sequence that can be automatically recognized by a machine, and then performs operation processing on the sign language motion data based on the sign language annotation sequence, which can improve the processing efficiency and accuracy of the sign language motion data.
[0153] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0154] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0155] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0156] An embodiment of the present invention provides a device for data processing, the device comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations:
[0157] A11. Obtaining sign language action data to be processed;
[0158] A12. Performing element analysis on the sign language action data based on a pre-constructed sign language annotation system to determine a sign language annotation sequence corresponding to the sign language action data, wherein the sign language annotation sequence includes annotation information of each basic sign language element corresponding to the sign language action data;
[0159] A13. Perform operation processing on the sign language action data according to the sign language annotation sequence.
[0160] Optionally, before performing element analysis on the sign language action data based on the pre-built sign language annotation system, the method further includes:
[0161] Based on the linguistic framework, the sign language action data in the database is disassembled and classified to obtain the basic sign language elements and the element types corresponding to each basic sign language element;
[0162] A sign language annotation system is constructed according to the basic sign language elements and the element types corresponding to each basic sign language element. The sign language annotation system includes annotation information corresponding to each element type of each basic sign language element.
[0163] Optionally, the basic sign language elements include at least one of left and right arm features, single and double hand shape features, directional movement features, knuckle bending angles, facial expression features, and constraint information.
[0164] Optionally, the step of disassembling and classifying the sign language action data in the database based on the linguistic framework to obtain basic sign language elements and element types corresponding to each basic sign language element includes:
[0165] Traversing each sign language action data in the database, performing action decomposition on each sign language action data, and determining the key parts corresponding to each sign language action data and the action features of the key parts;
[0166] Based on the linguistic framework, the key parts corresponding to each sign language action data in the database and the action features of the key parts are classified and processed to obtain at least 2 clusters, each of which corresponds to a basic sign language element;
[0167] The element type of the basic sign language element corresponding to each cluster is determined according to the action features included in the cluster.
[0168] Optionally, the action feature includes at least one of rotation data, displacement data, bending angle, key feature, and expression feature.
[0169] Optionally, performing element analysis on the sign language action data based on a pre-constructed sign language annotation system to determine a sign language annotation sequence corresponding to the sign language action data includes:
[0170] Performing element analysis on the sign language motion data to determine a first basic sign language element corresponding to the sign language motion data, a first element type of the first basic sign language element, and a first timestamp;
[0171] Determining first annotation information of the first basic sign language element and second annotation information of the first element type based on a pre-constructed sign language annotation system;
[0172] A sign language annotation sequence corresponding to the sign language action data is determined according to the first timestamp, the first annotation information, and the second annotation information.
[0173] Optionally, performing operation processing on the sign language action data according to the sign language annotation sequence includes:
[0174] Based on the sign language annotation sequence, a pre-built three-dimensional character model is driven to perform the sign language action corresponding to the sign language action data; and / or,
[0175] The sign language action data is subjected to sign language translation processing based on the sign language annotation sequence to obtain a target text sequence corresponding to the sign language action data.
[0176] Optionally, performing element analysis on the sign language action data based on a pre-constructed sign language annotation system to determine a sign language annotation sequence corresponding to the sign language action data includes:
[0177] Performing word segmentation processing on the text sequence to be translated to obtain a natural vocabulary sequence corresponding to the text sequence;
[0178] Determine the second basic sign language element corresponding to each natural word in the natural word sequence and the second element type corresponding to the second basic sign language element;
[0179] sorting the second basic sign language elements and the second element types to generate a sign language element sequence that complies with sign language grammar rules;
[0180] Determining a sign language annotation sequence corresponding to the sign language element sequence based on the sign language annotation system;
[0181] The performing operation processing on the sign language action data according to the sign language annotation sequence includes:
[0182] The text sequence is subjected to sign language translation processing according to the sign language annotation sequence to obtain sign language actions corresponding to the text sequence.
[0183] Figure 3 8 is a block diagram of a data processing apparatus 800 according to an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0184] Reference Figure 3 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .
[0185] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
[0186] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0187] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.
[0188] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0189] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice information processing mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.
[0190] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
[0191] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, the sensor assembly 814 can also detect the position change of the voice processing device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0192] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency information processing (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0193] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.
[0194] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0195] Figure 41 is a schematic diagram of the structure of the server in some embodiments of the present invention. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1922 (for example, one or more processors) and memory 1932, and one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1922 can be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.
[0196] The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, one or more keyboards 1956, and / or, one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
[0197] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (server or terminal), enables the device to perform Figure 1 The data processing method shown.
[0198] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (server or terminal), enables the device to execute a data processing method, the method comprising: obtaining sign language motion data to be processed; performing element analysis on the sign language motion data based on a pre-constructed sign language annotation system to determine a sign language annotation sequence corresponding to the sign language motion data, the sign language annotation sequence containing annotation information of each basic sign language element corresponding to the sign language motion data; and performing operation processing on the sign language motion data according to the sign language annotation sequence.
[0199] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed by the present invention. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present invention is indicated by the following claims.
[0200] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
[0201] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
[0202] The data processing method, the data processing device and the device for data processing provided by the present invention are introduced in detail above. The principle and implementation mode of the present invention are explained by using specific examples in this article. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation mode and the scope of application. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A data processing method, characterized in that: The method comprises: Obtaining sign language action data to be processed; The sign language action data to be processed is a sign language video and / or a sign language image containing sign language actions; Performing element analysis on the sign language motion data based on a pre-constructed sign language annotation system to determine a sign language annotation sequence corresponding to the sign language motion data, wherein the sign language annotation sequence includes annotation information of each basic sign language element corresponding to the sign language motion data; Performing operation processing on the sign language action data according to the sign language annotation sequence; Before performing element analysis on the sign language action data based on the pre-built sign language annotation system, the method further includes: Based on the linguistic framework, all the sign language action data in the database are disassembled and classified to obtain all the basic sign language elements involved in the database and the element type corresponding to each basic sign language element; The basic sign language elements include at least one of left and right arm features, single and double hand shape features, directional movement features, knuckle bending angles, facial expression features, and constraint information; The basic sign language element is the smallest feature unit obtained by decomposing the sign language action data to be processed; The sign language annotation sequence can be automatically recognized by a machine.
2. The method according to claim 1, characterized in that: Before performing element analysis on the sign language action data based on the pre-built sign language annotation system, the method further includes: A sign language annotation system is constructed according to the basic sign language elements and the element types corresponding to each basic sign language element. The sign language annotation system includes annotation information corresponding to each element type of each basic sign language element.
3. The method according to claim 2, characterized in that The linguistic framework is used to disassemble and classify the sign language action data in the database to obtain basic sign language elements and the element types corresponding to each basic sign language element, including: Traversing each sign language action data in the database, performing action decomposition on each sign language action data, and determining the key parts corresponding to each sign language action data and the action features of the key parts; Based on a linguistic framework, the key parts corresponding to each sign language action data in the database and the action features of the key parts are classified and processed to obtain at least two clusters, each of which corresponds to a basic sign language element; The element type of the basic sign language element corresponding to each cluster is determined according to the action features included in the cluster.
4. The method according to claim 3, characterized in that The action feature includes at least one of rotation data, displacement data, bending angle, key features, and expression features.
5. The method according to claim 1, characterized in that The performing element analysis on the sign language action data based on the pre-constructed sign language annotation system to determine the sign language annotation sequence corresponding to the sign language action data includes: Performing element analysis on the sign language motion data to determine a first basic sign language element corresponding to the sign language motion data, a first element type of the first basic sign language element, and a first timestamp; Determining first annotation information of the first basic sign language element and second annotation information of the first element type based on a pre-constructed sign language annotation system; A sign language annotation sequence corresponding to the sign language action data is determined according to the first timestamp, the first annotation information, and the second annotation information.
6. The method according to claim 1, characterized in that The performing operation processing on the sign language action data according to the sign language annotation sequence includes: Based on the sign language annotation sequence, a pre-built three-dimensional character model is driven to perform the sign language action corresponding to the sign language action data; and / or, The sign language action data is subjected to sign language translation processing based on the sign language annotation sequence to obtain a target text sequence corresponding to the sign language action data.
7. The method according to claim 1, characterized in that The performing element analysis on the sign language action data based on the pre-constructed sign language annotation system to determine the sign language annotation sequence corresponding to the sign language action data includes: Performing word segmentation processing on the text sequence to be translated to obtain a natural vocabulary sequence corresponding to the text sequence; Determine the second basic sign language element corresponding to each natural word in the natural word sequence and the second element type corresponding to the second basic sign language element; sorting the second basic sign language elements and the second element types to generate a sign language element sequence that complies with sign language grammar rules; Determining a sign language annotation sequence corresponding to the sign language element sequence based on the sign language annotation system; The performing operation processing on the sign language action data according to the sign language annotation sequence includes: The text sequence is subjected to sign language translation processing according to the sign language annotation sequence to obtain sign language actions corresponding to the text sequence.
8. A data processing device, characterized in that: The device comprises: A sign language motion data acquisition module, used to acquire the sign language motion data to be processed; The sign language action data to be processed is a sign language video and / or a sign language image containing sign language actions; A sign language annotation sequence determination module, configured to perform element analysis on the sign language motion data based on a pre-constructed sign language annotation system, and determine a sign language annotation sequence corresponding to the sign language motion data, wherein the sign language annotation sequence includes annotation information of each basic sign language element corresponding to the sign language motion data; An operation processing execution module, used for performing operation processing on the sign language action data according to the sign language annotation sequence; A basic sign language element determination module is used to disassemble and classify all sign language action data in the database based on a linguistic framework to obtain all basic sign language elements involved in the database and the element type corresponding to each basic sign language element; The basic sign language elements include at least one of left and right arm features, single and double hand shape features, directional movement features, knuckle bending angles, facial expression features, and constraint information; The basic sign language element is the smallest feature unit obtained by decomposing the sign language action data to be processed; The sign language annotation sequence can be automatically recognized by a machine.
9. The device according to claim 8, characterized in that The device also includes: The sign language annotation system construction module is used to construct a sign language annotation system according to the basic sign language elements and the element types corresponding to each basic sign language element. The sign language annotation system includes annotation information corresponding to each element type of each basic sign language element.
10. The device according to claim 9, characterized in that The basic sign language element determination module includes: The motion data analysis submodule is used to traverse each sign language motion data in the database, decompose each sign language motion data, determine the key part corresponding to each sign language motion data, and the motion characteristics of the key part; The action feature classification submodule is used to classify the key parts corresponding to each sign language action data in the database and the action features of the key parts based on the linguistic framework to obtain at least two clusters, each cluster corresponding to a basic sign language element; The element type determination submodule is used to determine the element type of the basic sign language element corresponding to each cluster according to the action features contained in the cluster.
11. The device according to claim 10, characterized in that The action feature includes at least one of rotation data, displacement data, bending angle, key features, and expression features.
12. A device for data processing, characterized in that: The device includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for executing the data processing method as described in any one of claims 1 to 7.
13. A machine-readable medium having instructions stored thereon, which, when executed by one or more processors, causes an apparatus to perform the data processing method according to any one of claims 1 to 7.
