Sign language gesture recognition method and device and AR head-mounted display equipment

The method and device address communication barriers by recognizing sign language gestures from different perspectives, enhancing understanding and accuracy through paired video decomposition and translation, facilitating effective interaction between special and ordinary groups.

CN119068540APending Publication Date: 2024-12-03BEIJING XIHONGXIAN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311382347.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-10-24
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

There are significant communication barriers and low accuracy in sign language gesture recognition between special groups and ordinary people, hindering effective communication.

Method used

A method and device that utilize paired sign language gesture videos from different perspectives, decomposed into image frames, generate skeleton point animations, match semantic information using a pre-built resource feature library, and display high-confidence information in three-dimensional space, while also translating voice information into sign language gestures.

Benefits of technology

Enhances communication efficiency by allowing ordinary people to accurately understand sign language gestures and special groups to comprehend voice information, improving recognition accuracy through multi-body movement consideration.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068540A_ABST
    Figure CN119068540A_ABST
Patent Text Reader

Abstract

The invention provides a sign language gesture recognition method and device and AR head-mounted display equipment, and relates to the technical field of sign language gesture recognition and range expansion. According to the sign language gesture recognition method, sign language gesture recognition is carried out by combining sign language gesture videos of a general person view angle and a special crowd view angle, so that general persons can quickly and accurately understand semantic information corresponding to sign language gesture actions used by special crowds; and the voice information is translated into a standard sign language gesture image sequence, so that special crowds can quickly and accurately understand the voice information expressed by general people. Besides, in consideration that sign language gesture actions are related to both hands and actually need other human body parts to jointly participate in representation, the first sign language gesture video and the second sign language gesture video are set to at least store complete actions of both hands of the special crowd and store complete actions of one or more combinations of the special crowd. The recognition accuracy of sign language gestures is further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of sign language gesture recognition and extended range technology, and in particular to a sign language gesture recognition method, device and AR head mounted display device. Background Art

[0002] Special groups have always been the focus of social attention. The published "National Common Sign Language Vocabulary" is used as a general textbook to guide the learning of special groups and school teaching. In this way, special groups who have been specially trained can express their meanings relatively accurately. However, when these special groups need to communicate with ordinary people (generally referring to people who cannot understand sign language and have normal hearing and language), there are still considerable communication barriers between the two sides. In view of this, it is necessary to provide a new sign language gesture recognition method. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a sign language gesture recognition method, device and AR head-mounted display device, which can solve the technical problems of communication barriers between the two parties and low accuracy of sign language gesture recognition.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0005] A sign language gesture recognition method, comprising:

[0006] Receiving paired first and second sign language gesture videos in real time, and decomposing the first and second sign language gesture videos into first and second sign language gesture image frame sequences respectively; wherein the first sign language gesture video is derived from the perspective of an average person, and the second sign language gesture video is derived from the perspective of a special group of people;

[0007] Generating first and second skeleton point graph animations correspondingly according to the first and second sign language gesture image frame sequences;

[0008] According to the first and second skeleton point diagram animations, correspondingly matching the first and second semantic information based on a pre-built sign language gesture resource feature library;

[0009] Comparing the confidence values ​​of the first and second semantic information;

[0010] An extended range method is used to present semantic information with high confidence values ​​to ordinary people in three-dimensional space.

[0011] Preferably, the first and second sign language gesture videos are paired based on ultra-wideband wireless communication.

[0012] Preferably, the first sign language gesture video at least stores the complete movements of the hands of the special population from the perspective of ordinary people, and stores the complete movements of one or more combinations of the forearm, upper arm, shoulder, mouth, nose, eye, ear, and head of the special population;

[0013] The second sign language gesture video at least saves the complete movements of the hands of the special people themselves from the perspective of the special people, and saves the complete movements of one or more combinations of the special people's forearms, upper arms, shoulders, mouths, noses, eyes, ears, and heads.

[0014] Preferably, the sign language gesture recognition method further comprises:

[0015] Receive voice messages in real time and recognize them as text messages;

[0016] Extracting keywords from the text information to obtain third semantic information;

[0017] According to the third semantic information, matching a corresponding standard sign language gesture image frame sequence based on the sign language gesture resource feature library;

[0018] An extended range method is used to display the standard sign language gesture image frame sequence to special groups of people in three-dimensional space.

[0019] Preferably, the process of constructing the sign language gesture resource feature library includes:

[0020] S100, associating current semantic information with a specified standard sign language gesture image frame sequence;

[0021] S200, for each standard sign language gesture image frame sequence, obtaining a preset number of pairs of real sign language gesture image frame sequences from the perspectives of ordinary people and special groups;

[0022] S300, for each pair of the real sign language gesture image frame sequence, correspondingly generate a skeleton point graph animation of different viewing angles; wherein the skeleton point graph animation is composed of a plurality of skeleton point graphs;

[0023] S400, repeatedly executing S100 to S300 until all semantic information in the common vocabulary of the national general sign language is traversed, and the construction of the sign language gesture resource feature library is completed.

[0024] A sign language gesture recognition device, comprising:

[0025] A decomposition module is used to receive the paired first and second sign language gesture videos in real time, and decompose the first and second sign language gesture videos into first and second sign language gesture image frame sequences respectively; wherein the first sign language gesture video is derived from the perspective of an average person, and the second sign language gesture video is derived from the perspective of a special group of people;

[0026] A generating module, configured to generate first and second skeleton point graph animations correspondingly according to the first and second sign language gesture image frame sequences;

[0027] A first matching module is used to match the first and second semantic information based on a pre-built sign language gesture resource feature library according to the first and second skeleton point diagram animations;

[0028] A comparison module, used for comparing the confidence values ​​of the first and second semantic information;

[0029] The first display module is used to display semantic information with a large confidence value to ordinary people in three-dimensional space by adopting an extended range method.

[0030] Preferably, the first and second sign language gesture videos are paired based on ultra-wideband wireless communication.

[0031] Preferably, the first sign language gesture video at least stores the complete movements of the hands of the special population from the perspective of ordinary people, and stores the complete movements of one or more combinations of the forearm, upper arm, shoulder, mouth, nose, eye, ear, and head of the special population;

[0032] The second sign language gesture video at least saves the complete movements of the hands of the special people themselves from the perspective of the special people, and saves the complete movements of one or more combinations of the special people's forearms, upper arms, shoulders, mouths, noses, eyes, ears, and heads.

[0033] Preferably, the sign language gesture recognition device further comprises:

[0034] A recognition module is used to receive voice information in real time and recognize it as text information;

[0035] An extraction module, used for extracting keywords from the text information to obtain third semantic information;

[0036] A second matching module is used to match the corresponding standard sign language gesture image frame sequence based on the sign language gesture resource feature library according to the third semantic information;

[0037] The second display module is used to display the standard sign language gesture image frame sequence to special groups of people in three-dimensional space by adopting an extended range method.

[0038] Preferably, the process of constructing the sign language gesture resource feature library includes:

[0039] S100, associating current semantic information with a specified standard sign language gesture image frame sequence;

[0040] S200, for each specified standard sign language gesture image frame sequence, obtaining a preset number of pairs of real sign language gesture image frame sequences from the perspectives of ordinary people and special groups;

[0041] S300, for each pair of the real sign language gesture image frame sequence, correspondingly generate a skeleton point graph animation of different viewing angles; wherein the skeleton point graph animation is composed of a plurality of skeleton point graphs;

[0042] S400, repeatedly executing S100 to S300 until all semantic information in the common vocabulary of the national general sign language is traversed, and the construction of the sign language gesture resource feature library is completed.

[0043] An AR head mounted display device comprises the sign language gesture recognition device as described above.

[0044] The present invention provides a sign language gesture recognition method, device and AR head mounted display device. Compared with the prior art, the present invention has the following beneficial effects:

[0045] In the present invention, paired first and second sign language gesture videos are received in real time, and the first and second sign language gesture videos are correspondingly decomposed into first and second sign language gesture image frame sequences; wherein the first sign language gesture video is derived from the perspective of ordinary people, and the second sign language gesture video is derived from the perspective of special groups; first and second skeleton dot graph animations are correspondingly generated according to the first and second sign language gesture image frame sequences; according to the first and second skeleton dot graph animations, the first and second semantic information are correspondingly matched based on a pre-built sign language gesture resource feature library; the confidence values ​​of the first and second semantic information are compared; and the extended range method is used to display the semantic information with a large confidence value to ordinary people in three-dimensional space. By performing sign language gesture recognition from different perspectives, it is convenient for ordinary people to quickly and accurately understand the semantic information corresponding to the sign language gesture actions used by special groups. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A block diagram of a sign language gesture recognition method provided by an embodiment of the present invention;

[0048] Figure 2 A block diagram of another sign language gesture recognition method provided by an embodiment of the present invention;

[0049] Figure 3 An example of a skeleton point map provided by an embodiment of the present invention;

[0050] Figure 4 A structural diagram of a sign language gesture recognition device provided by an embodiment of the present invention;

[0051] Figure 5 A structural diagram of another sign language gesture recognition device provided by an embodiment of the present invention;

[0052] Figure 6 A schematic diagram of the working principle of an AR head mounted display device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The embodiments of the present application solve the technical problem of low accuracy in sign language gesture recognition by providing a sign language gesture recognition method, device and AR head-mounted display device, thereby greatly improving the communication efficiency between special groups and the general public.

[0055] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0056] From the perspective of an average person: receive the paired first and second sign language gesture videos in real time, and decompose the first and second sign language gesture videos into first and second sign language gesture image frame sequences; wherein the first sign language gesture video is derived from the perspective of an average person, and the second sign language gesture video is derived from the perspective of a special group of people; according to the first and second sign language gesture image frame sequences, generate first and second skeleton dot graph animations; according to the first and second skeleton dot graph animations, match the first and second semantic information based on a pre-built sign language gesture resource feature library; compare the confidence values ​​of the first and second semantic information; and use an extended range method to display semantic information with a large confidence value to an average person in three-dimensional space. Performing sign language gesture recognition from different perspectives makes it easier for average people to quickly and accurately understand the semantic information corresponding to the sign language gesture actions used by special groups of people.

[0057] From the perspective of special groups: receive voice information in real time and identify it as text information; extract keywords from the text information to obtain third semantic information; match the corresponding standard sign language gesture image frame sequence based on the sign language gesture resource feature library according to the third semantic information; use the extended range method to display the standard sign language gesture image frame sequence to special groups in three-dimensional space. Through the above translation process, special groups can quickly and accurately understand the voice information expressed by ordinary people.

[0058] In addition, considering that sign language gestures are not only related to the hands, but also require other parts of the human body to participate in the expression. Therefore, the embodiment of the present application also sets the first and second sign language gesture videos to at least save the complete movements of the hands of the special group of people, and also save the complete movements of one or more combinations of the forearm, upper arm, shoulder, mouth, nose, eye, ear, and head of the special group of people, so as to further improve the recognition accuracy of sign language gestures.

[0059] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0060] First of all, it should be noted that the "special population" in the embodiments of the present application generally refers to people who have hearing and language impairments and have learned the sign language in the "National Common Sign Language Vocabulary" and can express sign language normally; "ordinary people" generally refers to people who cannot understand sign language and have normal hearing and language.

[0061] The "general public perspective" and "special group perspective" in the embodiments of the present application are different observation angles defined based on the sign language gestures used by special groups of people, among which the "general public perspective" is a relatively flexible angle (the perspective from which general people observe the sign language gestures used by special groups of people), and the "special group perspective" is a relatively fixed angle (the perspective from which special groups observe the sign language gestures used by themselves).

[0062] Embodiment 1:

[0063] like Figure 1 As shown, an embodiment of the present invention provides a sign language gesture recognition method (from a general person's perspective), comprising:

[0064] S1. Receive paired first and second sign language gesture videos in real time, and decompose the first and second sign language gesture videos into first and second sign language gesture image frame sequences respectively; wherein the first sign language gesture video is derived from the perspective of an average person, and the second sign language gesture video is derived from the perspective of a special group of people;

[0065] S2. Generate first and second skeleton point diagram animations correspondingly according to the first and second sign language gesture image frame sequences;

[0066] S3, according to the first and second skeleton point diagram animations, correspondingly matching the first and second semantic information based on a pre-built sign language gesture resource feature library;

[0067] S4, comparing the confidence values ​​of the first and second semantic information;

[0068] S5. Use the extended range method to display semantic information with large confidence values ​​to ordinary people in three-dimensional space.

[0069] The embodiments of the present invention perform sign language gesture recognition through different viewing angles, so that ordinary people can quickly and accurately understand the semantic information corresponding to the sign language gesture movements used by special groups of people.

[0070] In an optional implementation, the first and second sign language gesture videos are paired based on ultra-wideband wireless communication in S1.

[0071] Since the first and second sign language gesture videos come from different perspectives, the first sign language gesture video can be set to be collected by an AR head-mounted display device worn by ordinary people, while the second sign language gesture video can be collected by an AR head-mounted display device worn by special people. Two AR head-mounted display devices are set to transmit relative position relationship and shared perspective based on ultra-wideband (Ultra Wideband Band, UWB) wireless communication, determine the communication object and then pair the first and second sign language gesture videos. It should be noted that, assuming that two AR head-mounted display devices establish face-to-face communication based on UWB, even if the positions of the communicating parties change to non-face-to-face, it does not affect the pairing of the collected videos, so as to finally complete the sign language gesture recognition work.

[0072] Considering that sign language gestures are not only related to the hands, but also require other parts of the human body to participate in the expression, it is necessary to consider the impact of this factor on the sign language gesture recognition process. Therefore, in an optional implementation, the following settings are made:

[0073] The first sign language gesture video at least preserves the complete movements of the hands of the special population from the perspective of ordinary people, and preserves the complete movements of one or more combinations of the forearm, upper arm, shoulder, mouth, nose, eye, ear, and head of the special population.

[0074] The second sign language gesture video at least saves the complete movements of the hands of the special people themselves from the perspective of the special people, and saves the complete movements of one or more combinations of the special people's forearms, upper arms, shoulders, mouths, noses, eyes, ears, and heads.

[0075] In order to achieve two-way barrier-free communication between special groups and ordinary people, in an optional implementation method, Figure 2 As shown, the sign language gesture recognition method (special population perspective) also includes:

[0076] S6. Receive voice information in real time and recognize it as text information;

[0077] S7, extracting keywords from the text information to obtain third semantic information;

[0078] S8. Matching a corresponding standard sign language gesture image frame sequence based on the sign language gesture resource feature library according to the third semantic information;

[0079] S9. Use an extended range method to display the standard sign language gesture image frame sequence to special groups of people in three-dimensional space.

[0080] The embodiment of the present invention helps special groups to quickly and accurately understand the voice information expressed by ordinary people by translating the voice information into a standard sign language gesture image sequence.

[0081] In an optional implementation, the process of constructing the sign language gesture resource feature library includes:

[0082] S100, associating current semantic information with a specified standard sign language gesture image frame sequence.

[0083] Supplementary explanation: The semantic information here can be text or audio. The embodiment of the present invention does not impose an absolute restriction on this. It only needs to display the matched text content directly in the display module of the AR head-mounted display device, or play the matched audio content directly through the speaker of the AR head-mounted display device in specific applications.

[0084] S200. For each standard sign language gesture image frame sequence, obtain a preset number of pairs of real sign language gesture image frame sequences from the perspectives of ordinary people and special groups of people.

[0085] S300, for each pair of the real sign language gesture image frame sequence, correspondingly generate skeleton point graph animations of different viewing angles; wherein the skeleton point graph animation is composed of a plurality of skeleton point graphs ( Figure 3 An example of a skeleton dot plot is given).

[0086] S400, repeatedly executing S100 to S300 until all semantic information in the common vocabulary of the national general sign language is traversed, and the construction of the sign language gesture resource feature library is completed.

[0087] It is not difficult to understand that each of the real sign language gesture image frame sequence pairs obtained in the above step S200 can also be obtained by processing sign language gesture videos collected by AR head-mounted display devices (or other collection devices) worn by ordinary people and special groups of people.

[0088] Embodiment 2:

[0089] like Figure 1 As shown, an embodiment of the present invention provides a sign language gesture recognition device (from a general person's perspective), comprising:

[0090] A decomposition module is used to receive the paired first and second sign language gesture videos in real time, and decompose the first and second sign language gesture videos into first and second sign language gesture image frame sequences respectively; wherein the first sign language gesture video is derived from the perspective of an average person, and the second sign language gesture video is derived from the perspective of a special group of people;

[0091] A generating module, configured to generate first and second skeleton point graph animations correspondingly according to the first and second sign language gesture image frame sequences;

[0092] A first matching module is used to match the first and second semantic information based on a pre-built sign language gesture resource feature library according to the first and second skeleton point diagram animations;

[0093] A comparison module, used for comparing the confidence values ​​of the first and second semantic information;

[0094] The first display module is used to display semantic information with a large confidence value to ordinary people in three-dimensional space by adopting an extended range method.

[0095] The embodiments of the present invention perform sign language gesture recognition through different viewing angles, so that ordinary people can quickly and accurately understand the semantic information corresponding to the sign language gesture movements used by special groups of people.

[0096] In an optional implementation, the decomposition module pairs the first and second sign language gesture videos based on ultra-wideband wireless communication.

[0097] Since the first and second sign language gesture videos come from different perspectives, the first sign language gesture video can be set to be collected by an AR head-mounted display device worn by ordinary people, while the second sign language gesture video can be collected by an AR head-mounted display device worn by special people. Two AR head-mounted display devices are set to transmit relative position relationship and shared perspective based on ultra-wideband (Ultra Wideband Band, UWB) wireless communication, determine the communication object and then pair the first and second sign language gesture videos. It should be noted that, assuming that two AR head-mounted display devices establish face-to-face communication based on UWB, even if the positions of the communicating parties change to non-face-to-face, it does not affect the pairing of the collected videos, so as to finally complete the sign language gesture recognition work.

[0098] Considering that sign language gestures are not only related to the hands, but also require other parts of the human body to participate in the expression, it is necessary to consider the impact of this factor on the sign language gesture recognition process. Therefore, in an optional implementation, the following settings are made:

[0099] The first sign language gesture video at least preserves the complete movements of the hands of the special population from the perspective of ordinary people, and preserves the complete movements of one or more combinations of the forearm, upper arm, shoulder, mouth, nose, eye, ear, and head of the special population.

[0100] The second sign language gesture video at least saves the complete movements of the hands of the special people themselves from the perspective of the special people, and saves the complete movements of one or more combinations of the special people's forearms, upper arms, shoulders, mouths, noses, eyes, ears, and heads.

[0101] In order to achieve two-way barrier-free communication between special groups and ordinary people, in an optional implementation method, Figure 5 As shown, the sign language gesture recognition device (special population perspective) also includes:

[0102] A recognition module is used to receive voice information in real time and recognize it as text information;

[0103] An extraction module, used for extracting keywords from the text information to obtain third semantic information;

[0104] A second matching module is used to match the corresponding standard sign language gesture image frame sequence based on the sign language gesture resource feature library according to the third semantic information;

[0105] The second display module is used to display the standard sign language gesture image frame sequence to special groups of people in three-dimensional space by adopting an extended range method.

[0106] The embodiment of the present invention helps special groups to quickly and accurately understand the voice information expressed by ordinary people by translating the voice information into a standard sign language gesture image sequence.

[0107] In an optional implementation, the process of constructing the sign language gesture resource feature library includes:

[0108] S100, associating current semantic information with a specified standard sign language gesture image frame sequence.

[0109] Supplementary explanation: The semantic information here can be text or audio. The embodiment of the present invention does not impose an absolute restriction on this. It only needs to display the matched text content directly in the display module of the AR head-mounted display device, or play the matched audio content directly through the speaker of the AR head-mounted display device in specific applications.

[0110] S200. For each standard sign language gesture image frame sequence, obtain a preset number of pairs of real sign language gesture image frame sequences from the perspectives of ordinary people and special groups of people.

[0111] S300, for each pair of the real sign language gesture image frame sequence, correspondingly generate skeleton point graph animations of different viewing angles; wherein the skeleton point graph animation is composed of a plurality of skeleton point graphs ( Figure 3 An example of a skeleton dot plot is given).

[0112] S400, repeatedly executing S100 to S300 until all semantic information in the common vocabulary of the national general sign language is traversed, and the construction of the sign language gesture resource feature library is completed.

[0113] It is not difficult to understand that each of the real sign language gesture image frame sequence pairs obtained in the above step S200 can also be obtained by processing sign language gesture videos collected by AR head-mounted display devices (or other collection devices) worn by ordinary people and special groups of people.

[0114] Embodiment 3:

[0115] An embodiment of the present invention provides an AR head-mounted display device, including the sign language gesture recognition device as described in Example 2.

[0116] Among them, the working principle of the AR head mounted display device is as follows: Figure 6 As shown, the following general description is given as an example:

[0117] (1) The special group of people sends a sign language composed of several hand gestures to the general public. At this time, the AR head-mounted display device worn by the general public will actively collect a first sign language gesture video through its camera, and receive a second sign language gesture video collected by the camera of the AR head-mounted display device worn by the paired special group of people. The first and second sign language gesture image frame sequences are correspondingly obtained through image recognition technology, and the first and second skeleton point diagram animations are generated. The first and second semantic information are correspondingly matched based on a pre-built sign language gesture resource feature library, and the confidence values ​​of the first and second semantic information are compared. The semantic information with a larger confidence value is displayed in the display module of the AR head-mounted display device worn by the general public.

[0118] (2) An ordinary person sends a voice message to a special group of people. At this time, the AR head-mounted display device worn by the special group of people will recognize it as a text message through the microphone pickup function, and then perform keyword extraction to obtain third semantic information. The third semantic information is matched in the sign language gesture resource feature library. After matching, a standard sign language gesture image frame sequence is obtained, which is played in the form of a 3D virtual character in the display module of the AR head-mounted device worn by the special group of people.

[0119] It is understandable that since the AR head-mounted display device provided by an embodiment of the present invention includes the above-mentioned sign language gesture recognition device, the explanations, examples and beneficial effects of other related contents can refer to the corresponding parts in the sign language gesture recognition device and will not be repeated here.

[0120] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0121] 1. The embodiments of the present invention perform sign language gesture recognition from different perspectives, so that ordinary people can quickly and accurately understand the semantic information corresponding to the sign language gesture movements used by special groups; and by translating voice information into a standard sign language gesture image sequence, special groups can quickly and accurately understand the voice information expressed by ordinary people.

[0122] 2. In addition, considering that sign language gestures are not only related to the hands, but also require the participation of other parts of the human body, it is necessary to consider the impact of this factor on the sign language gesture recognition process. Therefore, it is also set that the first and second sign language gesture videos at least save the complete movements of the hands of special people, and also save the complete movements of one or more combinations of the forearms, upper arms, shoulders, mouths, noses, eyes, ears, and heads of special people, so as to further improve the recognition accuracy of sign language gestures.

[0123] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0124] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sign language gesture recognition method, characterized in that: include: Receiving paired first and second sign language gesture videos in real time, and decomposing the first and second sign language gesture videos into first and second sign language gesture image frame sequences respectively; wherein the first sign language gesture video is derived from the perspective of an average person, and the second sign language gesture video is derived from the perspective of a special group of people; Generating first and second skeleton point graph animations correspondingly according to the first and second sign language gesture image frame sequences; According to the first and second skeleton dot graph animations, correspondingly matching the first and second semantic information based on a pre-built sign language gesture resource feature library; Comparing the confidence values ​​of the first and second semantic information; An extended range method is used to present semantic information with high confidence values ​​to ordinary people in three-dimensional space.

2. The sign language gesture recognition method according to claim 1, characterized in that: The first and second sign language gesture videos are paired based on ultra-wideband wireless communication.

3. The sign language gesture recognition method according to claim 1, characterized in that: The first sign language gesture video at least stores complete movements of the hands of the special population from the perspective of ordinary people, and stores complete movements of one or more combinations of the forearm, upper arm, shoulder, mouth, nose, eye, ear, and head of the special population; The second sign language gesture video at least saves the complete movements of the hands of the special people themselves from the perspective of the special people, and saves the complete movements of one or more combinations of the special people's forearms, upper arms, shoulders, mouths, noses, eyes, ears, and heads.

4. The sign language gesture recognition method according to any one of claims 1 to 3, characterized in that: Also includes: Receive voice messages in real time and recognize them as text messages; Extracting keywords from the text information to obtain third semantic information; According to the third semantic information, matching a corresponding standard sign language gesture image frame sequence based on the sign language gesture resource feature library; An extended range method is used to display the standard sign language gesture image frame sequence to special groups of people in three-dimensional space.

5. The sign language gesture recognition method according to claim 4, characterized in that: The construction process of the sign language gesture resource feature library includes: S100, associating current semantic information with a specified standard sign language gesture image frame sequence; S200, for each standard sign language gesture image frame sequence, obtaining a preset number of pairs of real sign language gesture image frame sequences from the perspectives of ordinary people and special groups; S300, for each pair of the real sign language gesture image frame sequence, correspondingly generate a skeleton point graph animation of different viewing angles; wherein the skeleton point graph animation is composed of a plurality of skeleton point graphs; S400, repeatedly executing S100 to S300 until all semantic information in the common vocabulary of the national general sign language is traversed, and the construction of the sign language gesture resource feature library is completed.

6. A sign language gesture recognition device, characterized in that: include: A decomposition module is used to receive the paired first and second sign language gesture videos in real time, and decompose the first and second sign language gesture videos into first and second sign language gesture image frame sequences respectively; wherein the first sign language gesture video is derived from the perspective of an average person, and the second sign language gesture video is derived from the perspective of a special group of people; A generating module, configured to generate first and second skeleton point graph animations correspondingly according to the first and second sign language gesture image frame sequences; A first matching module is used to match the first and second semantic information based on a pre-built sign language gesture resource feature library according to the first and second skeleton point diagram animations; A comparison module, used for comparing the confidence values ​​of the first and second semantic information; The first display module is used to display semantic information with a large confidence value to ordinary people in three-dimensional space by adopting an extended range method.

7. The sign language gesture recognition device according to claim 6, characterized in that: The first and second sign language gesture videos are paired based on ultra-wideband wireless communication.

8. The sign language gesture recognition device according to claim 6, characterized in that: The first sign language gesture video at least stores complete movements of the hands of the special population from the perspective of ordinary people, and stores complete movements of one or more combinations of the forearm, upper arm, shoulder, mouth, nose, eye, ear, and head of the special population; The second sign language gesture video at least saves the complete movements of the hands of the special people themselves from the perspective of the special people, and saves the complete movements of one or more combinations of the special people's forearms, upper arms, shoulders, mouths, noses, eyes, ears, and heads.

9. The sign language gesture recognition device according to any one of claims 6 to 8, characterized in that: Also includes: A recognition module is used to receive voice information in real time and recognize it as text information; An extraction module, used for extracting keywords from the text information to obtain third semantic information; A second matching module is used to match the corresponding standard sign language gesture image frame sequence based on the sign language gesture resource feature library according to the third semantic information; The second display module is used to display the standard sign language gesture image frame sequence to special groups of people in three-dimensional space by adopting an extended range method.

10. The sign language gesture recognition device according to claim 9, characterized in that: The construction process of the sign language gesture resource feature library includes: S100, associating current semantic information with a specified standard sign language gesture image frame sequence; S200, for each specified standard sign language gesture image frame sequence, obtaining a preset number of pairs of real sign language gesture image frame sequences from the perspectives of ordinary people and special groups; S300, for each pair of the real sign language gesture image frame sequence, correspondingly generate a skeleton point graph animation of different viewing angles; wherein the skeleton point graph animation is composed of a plurality of skeleton point graphs; S400, repeatedly executing S100 to S300 until all semantic information in the common vocabulary of the national general sign language is traversed, and the construction of the sign language gesture resource feature library is completed.

11. An AR head mounted display device, characterized in that: It comprises a sign language gesture recognition device as described in any one of claims 6 to 10.