Intelligent sign language ability identity identification system based on multi-modal data analysis
By building a sign language animation resource library and using AI to synthesize sign language animations, combined with disordered processing and countdown verification, the problem of existing technologies being unable to identify users' sign language abilities has been solved, achieving efficient and secure user screening and resource optimization.
Patent Information
- Application Number
- CN202511148612.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-18
- Publication Date
- 2025-12-02
AI Technical Summary
Existing authentication mechanisms cannot effectively identify whether users have basic sign language skills, resulting in the inefficient use of service resources, reduced efficiency and security of sign language translation services, and increased risk of harassment to hearing-impaired individuals.
The intelligent sign language ability identification system, which employs multimodal data analysis, constructs a sign language animation resource library, uses AI to synthesize sign language animations, randomly extracts and shuffles sign language animations and words, and combines a 60-second countdown and coordinate matching to verify the user's sign language ability.
It enables precise screening of non-target users, reduces server hardware and cloud leasing costs, improves service accuracy and security, and effectively prevents machine hacking and harassment.
Smart Images
Figure CN121051731A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent recognition technology, and more specifically to an intelligent sign language ability identification system based on multimodal data analysis. Background Technology
[0002] After the internet service was made publicly available, there were many problems of harassment and misuse by non-target users. There is an urgent need for a mechanism to identify whether users have basic sign language skills, so as to effectively prevent service abuse and ensure that resources are used to serve people with disabilities who really need help.
[0003] In the current state of industry technology development, there are almost no dedicated verifiers on the market that can identify whether a user possesses sign language skills. While some identity verification products based on image recognition technology exist, these products primarily detect specific gestures to determine if a user is a bot (e.g., through CAPTCHA mechanisms), and cannot identify whether a user possesses basic sign language communication abilities. In other words, these existing technologies focus only on anti-bot mechanisms, not on verifying sign language capabilities. Therefore, the entire industry remains technologically unavailable on the crucial issue of how to verify whether a user possesses basic sign language skills, lacking an effective solution to screen out users with sign language abilities.
[0004] Existing technologies have significant flaws and shortcomings. Specifically, existing authenticators typically focus on Anomaly Detection (ADT), a mechanism for identifying abnormal user behavior (such as frequent login attempts or automated script attacks), but their core functionality does not involve determining whether a user possesses basic sign language skills. Therefore, in practical applications, when service providers need to filter out users who lack sign language communication skills (such as malicious harassers or non-target groups), existing authenticators cannot meet this requirement. This limitation leads to the inefficient use of service resources, not only reducing the efficiency and security of sign language translation services but also exacerbating the risk of harassment against hearing-impaired individuals, failing to provide an efficient and secure service environment for disabled users. Summary of the Invention
[0005] In view of this, the present invention provides an intelligent sign language ability identification system based on multimodal data analysis, which aims to solve the above-mentioned technical problems.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] The intelligent sign language ability identification system based on multimodal data analysis includes the following steps: S10: Construct a sign language animation resource library by decomposing the sign language expression system in the National Common Sign Language Dictionary into 500 basic gesture units and 1200 composite action templates, and synthesizing sign language animations by combining artificial intelligence technology;
[0008] S20: Randomly select 4 sign language animations and corresponding text words from the resource library, and perform image processing on the text words to scramble and distort interference.
[0009] S30: Push an interactive webpage containing the four sign language animations and the image-processed words, start a 60-second countdown, and ask the user to click on the matching word images in the order of the animations;
[0010] S40: Perform multimodal analysis based on user operation time difference and coordinate position to verify user sign language ability.
[0011] Furthermore, the basic gesture unit in S10 includes: 120 hand shape parameters, defining the finger shape and palm direction; 80 direction parameters, defining the direction of the palm movement trajectory; and 300 position parameters, defining the positioning coordinates of the palm in space.
[0012] Furthermore, the synthesis step of the digital human sign language animation in S10 includes: inputting target vocabulary and parsing its expression standard in the "National Common Sign Language Dictionary"; calling the corresponding compound action template and extracting basic gesture unit parameters; generating a continuous action sequence through AI-driven digital human model and outputting the animation file within 1 second.
[0013] Furthermore, the random sampling operation in S20 is designed to prevent users from summarizing sampling patterns, and the disordering of text words is used to verify the user's ability to match text words with sign language animations.
[0014] Furthermore, the distortion and interference image processing performed on the disordered text words in S20 is used to prevent technical means from capturing word information and to avoid brute-force attacks.
[0015] Furthermore, the interactive webpage layout in S30 is set as follows: four sign language animations are arranged at the top of the webpage in the order of extraction; a dotted image with image processing is set directly below each animation, and the dotted image contains distorted and interfering word image information.
[0016] Furthermore, in S30, the coordinate position of the selected image is bound to a preset range of the vocabulary image, and the user clicks on the coordinate position for the background to match specific words.
[0017] Furthermore, the timeout determination logic in S40 is as follows: if the time difference between user operations is greater than 60 seconds, a forced refresh is triggered and the verification content is regenerated.
[0018] Furthermore, the coordinate matching verification logic in S40 includes: comparing the user's selection order with the animation extraction order recorded in S20; verifying whether all selected coordinates fall within the preset word image range; and determining that the verification is successful only when the order is consistent and all coordinates match.
[0019] Compared with existing technologies, this invention first employs parametric AI synthesis technology (S10) to decompose the "National Common Sign Language Dictionary" into 500 basic gesture units (including 120 hand shapes / 80 directions / 300 position parameters) and 1200 composite action templates. Through AI-driven digital human models, it automatically generates compliant sign language animations within one second, solving the problem of resource libraries relying on manual filming, reducing production costs by more than 60%, and supporting real-time updates. Second, it designs a random verification mechanism (S20-S30), randomly selecting four sign language animations and their corresponding words. After scrambling and distorting the images, these are pushed to an interactive webpage. Combined with a 60-second time limit and pre-stored coordinate mapping rules (users must select the correct word coordinates according to the animation order), this effectively verifies the user's sign language cognitive ability and prevents machine cracking. Finally, through multimodal analysis (S40), it verifies timeliness (≤60-second operation time difference) and accuracy (selection order + coordinate dual matching), accurately blocking more than 80% of non-target users (such as harassers or those without sign language ability). The positive effects of this include: (1) Optimized resource costs: server hardware investment and cloud leasing costs are reduced by 50%; (2) Precise service: hearing-impaired people receive exclusive and efficient services; (3) Enhanced security: the risk of system brute-force attacks is significantly reduced, forming a technical closed loop of "low-cost construction - precise screening - efficient resource utilization". Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0021] Figure 1 This is a flowchart of the implementation process of an intelligent sign language ability identification system service based on multimodal data analysis. Detailed Implementation
[0022] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0023] Example
[0024] See appendix Figure 1 To make the above-mentioned objectives of the present invention more apparent and understandable, and to demonstrate the features and advantages of the inventive method, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0025] The intelligent sign language ability identification system based on multimodal data analysis includes the following steps: S10: Constructing a sign language animation resource library by decomposing the sign language expression system in the "National Common Sign Language Dictionary" into 500 basic gesture units and 1200 composite action templates, and synthesizing sign language animations using artificial intelligence technology; I. Division and parameter definition of basic gesture units.
[0026] The basic gesture unit contains 500 basic gestures, specifically divided into:
[0027] 1. Hand shape parameters (120): such as "palm shape" (fingers together and straight, palm facing forward, corresponding to the starting hand shape of "hello" in the National Common Sign Language Dictionary), "fist shape" (fingers bent and fist clenched, thumb touching the second joint of the index finger, corresponding to the core hand shape of "thank you"), etc.
[0028] 2. Directional parameters (80): such as "forward" (the palm extends from the chest to the front, corresponding to the arm movement direction of "goodbye"), "up" (the palm rises from the waist to the top of the head, corresponding to the gesture direction of "victory"), etc.
[0029] 3. Position parameters (300): such as "chest" (palm is located above the middle of the sternum, corresponding to the starting position of "hello"), "cheek" (palm is close to the cheek on the same side, corresponding to the hand gesture position of "mother"), etc.
[0030] The above parameters constitute the basic elements of sign language expression, and can be digitally modeled to form a standard parameter library, serving as the underlying data support for animation synthesis.
[0031] II. Construction of Composite Action Templates and Linkage Rules
[0032] The composite action template contains 1200 high-frequency word phrase linkage rules, which are constructed based on the parameter combination logic of basic gesture units, for example:
[0033] 1. For the word "Hello" in the "National Common Sign Language Dictionary", its composite action template is defined as: based on the "palm shape" (hand shape parameter), making an arc swing from the "front of the chest" (position parameter) to "45° forward" (direction parameter), keeping the wrist naturally relaxed during the linkage process, and the action lasts for 0.8 seconds;
[0034] 2. For the word "Thank you", its composite action template is defined as: starting with the hand shape of "fist shape" (hand shape parameter), making an upward movement from the "abdomen" (position parameter) to the "front of the chest" (position parameter), while the palm unfolds into the "palm shape", with the elbow slightly bent during the linkage process, and the action lasts for 1.2 seconds.
[0035] III. Functions of the Composite Action Template
[0036] The function of the composite action template is to standardize the action sequence logic of high-frequency phrases, ensuring that the sign language animations generated by parameter combinations conform to the standard expression norms of the "National Common Sign Language Dictionary".
[0037] The composite action template is the core link connecting basic gesture units and specific vocabulary. Based on the action rules of high-frequency phrases in the "National Common Sign Language Dictionary", it presets the linkage rules of basic gesture units, ensuring that the action sequences generated by parameter combinations conform to sign language expression habits. For example, the composite action template of the word "Welcome" stipulates the linkage logic of the "two palm shapes" making a cross swing from "both sides of the body" to the "midline of the front of the chest", avoiding action distortion caused by random parameter combinations.
[0038] IV. Formation Steps of the Digital Human Sign Language Display Animation (Taking "Hello" and "Thank you" as Examples)
[0039] Step 1: Input the target sign language vocabulary (such as "Hello");
[0040] Step 2: The system analyzes the standard expression of this vocabulary in the "National Common Sign Language Dictionary" and calls the corresponding composite action template (the linkage rule of "Hello");
[0041] Step 3: Based on the composite action template, extract the required parameters from the basic gesture unit library: "palm shape" (hand shape), "front of the chest" (starting position), "45° forward" (direction);
[0042] Step 4: The artificial intelligence synthesis module drives the digital human model to generate continuous actions according to the parameter sequence and linkage rule: the digital human's hands are in the palm shape in front of the chest and make an arc swing along the 45° forward direction to complete the sign language demonstration of "Hello"; [[ID=X]] [[ID=Y]]
[0043] Step 5: The system automatically renders and outputs the animation file, and the whole process requires no manual participation and can be completed within one second.
[0044] Similarly, when generating the sign language animation for "thank you", the system calls the corresponding composite action template, extracts parameters such as "fist shape", "abdomen", and "chest", and uses AI to synthesize a digital human's action sequence from clenching a fist to raising it to the chest and then opening the palm.
[0045] V. Relationships between elements
[0046] 1. There is a one-to-one correspondence between sign language vocabulary (from the National Common Sign Language Dictionary) and compound action templates: each vocabulary corresponds to a unique compound action template (e.g., "goodbye" corresponds to the "waving" template);
[0047] 2. Composite action templates and basic gesture units are contained within each other: each template is composed of 2-5 basic gesture units combined according to linkage rules (e.g., the template for "sorry" includes units such as "palm shape", "forehead position", and "moving downward");
[0048] 3. The digital human sign language display animation is the final presentation form of basic gesture units driven by composite action templates and synthesized by AI: it is generated through three stages: using vocabulary as input, template calling, parameter extraction, and AI synthesis.
[0049] The above solution breaks through the limitations of traditional shooting modes, and can quickly generate sign language animations that meet national standards without human intervention. It significantly reduces the production cost of the resource library (more than 60% lower than the traditional method) and the cycle (the production cycle of a single animation is shortened from 2 hours to less than one second). Moreover, when the vocabulary standards change, only the corresponding parameters or templates need to be adjusted to achieve updates, which significantly improves flexibility.
[0050] S20: Randomly select 4 sign language animations and corresponding text words from the resource library, shuffle the text words and perform image processing with distortion interference; randomly select 4 sign language animations and 4 text words corresponding to the 4 sign language animations from the sign language resources obtained in step S10.
[0051] The purpose of randomly selecting sign language animations is to prevent users from using the sampling patterns to deceive others.
[0052] The four text words are shuffled to verify whether users have sign language recognition ability and the ability to match text words with corresponding sign language animations. Through the above verification, efficient screening of the target audience can be achieved, and harassing users can be effectively excluded.
[0053] Furthermore, the disordered text words are subjected to distorted and interfered visual processing to prevent users from using technical means to capture word information and to avoid brute-force attacks that are not performed manually.
[0054] S30: Push an interactive webpage containing the four sign language animations and the image-processed words, start a 60-second countdown, and require the user to select the matching word images in the order of the animations; based on the four sign language animations generated in step S20 and the corresponding text word images that have been scrambled and distorted, construct a sign language verification interactive webpage and push the webpage to the front-end user interface. The layout of the sign language verification interactive webpage is set as follows: according to the random sampling order recorded in step S20, the four sign language animations are arranged sequentially in the upper area of the webpage; directly below each sign language animation, a corresponding image-processed "selection image" (i.e., the text word image that has been scrambled and distorted in step S20, each selection image containing one word image information to be matched) is set.
[0055] After the webpage is pushed out, the backend system automatically records the current push time and triggers a 60-second countdown. Users need to watch the sign language animation at the top of the webpage and, from left to right, click on the "Select Image" button below each animation to choose the word image that matches the meaning of that sign language animation.
[0056] The technical purpose of setting a time limit is to set a 60-second countdown timer, and its main functions are as follows:
[0057] Improve verification efficiency: Ensure users complete the operation within a reasonable time by using time constraints, avoiding excessive system resource consumption caused by long verification processes;
[0058] Anti-cheating mechanism: Prevent users from using external tools to look up the meaning of sign language or seeking assistance from others (such as consulting sign language users in real time), and ensure that the verification results reflect the user's true sign language cognition ability;
[0059] Mitigating system risks: Without setting a time limit, malicious users may occupy the verification interface for an extended period, using technical means to analyze the correspondence between sign language animations and vocabulary, increasing the risk of brute-force attacks, while also prolonging the waiting period for legitimate users and reducing the service experience.
[0060] The technical role of coordinate position
[0061] The "selected image" is a word display carrier that has undergone image processing (corresponding to the processing result of "word distortion and interference image processing" in step S20), and it contains multiple randomly arranged word image regions. Because the word images have undergone distortion and interference processing, the content cannot be directly extracted through text recognition. Therefore, the coordinates of the user's click become the core basis for the backend to determine the user's selection result:
[0062] Each word image occupies a unique coordinate range in the selected image, and the background system pre-stores the coordinate mapping relationship of each word image;
[0063] The coordinates generated by the user's click are analyzed by the backend and matched to specific words. Combined with the click order information, the user's selection result is compared and verified with the correct answer.
[0064] After the user completes all the clicks, the front-end system automatically packages the click order (corresponding to the arrangement of the four sign language animations) and the coordinates of each click and sends them to the back-end server, where the verification logic is executed. If the user does not complete the operation within 60 seconds, the system will trigger a forced refresh mechanism, re-executing step S20 to generate new sign language animations and word combinations, ensuring the security and uniqueness of the verification process.
[0065] S40: Perform multimodal analysis based on user operation time difference and coordinate position to verify user sign language ability.
[0066] Multimodal analysis and capability identification: After receiving the user-uploaded selection sequence data and corresponding coordinate location information, the backend system executes the following verification process. First, it extracts the push time recorded in step S30 and calculates the difference between the push time and the current receiving time. If the time difference is greater than 60 seconds, it is determined to be a timeout, and the system triggers a forced refresh mechanism, returning to step S20 to re-execute the sign language animation extraction and vocabulary processing process and generate new push content; if the time difference is less than or equal to 60 seconds, it enters the coordinate matching verification stage.
[0067] In the coordinate matching verification, the system compares the order of clicks uploaded by the user with the order of sign language animation extraction recorded in step S20, and simultaneously verifies whether the coordinates of each click fall within the preset coordinate range of the corresponding word image on the webpage. When the click order matches the extraction order and all coordinates are within the preset range, the verification is deemed successful, confirming that the user possesses the corresponding sign language recognition and matching ability; otherwise, the verification is deemed unsuccessful, and the system can implement corresponding strategies based on the number of failures (such as pushing verification content again or restricting access).
[0068] This step utilizes a multimodal analysis approach that combines time threshold control with coordinate sequence verification to comprehensively identify the timeliness and accuracy of user operations, further enhancing the rigor of the verification mechanism and effectively resisting evasion behaviors by non-target users.
[0069] This invention can effectively block and filter out misuse, harassment, cyberattacks, and bots, especially those from users without sign language skills, up to 80%. While solving security issues, it allows more resources to be used to provide efficient services to hearing and speech-impaired individuals who genuinely need assistance. This invention can effectively save server resources such as CPU, memory, hard drive, and bandwidth, reducing server configuration and thus significantly lowering hardware investment and cloud server rental costs for sign language service systems by 50%.
[0070] Although embodiments of this application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the appended claims and their equivalents.
Claims
1. An intelligent sign language ability identification system based on multimodal data analysis, characterized in that, Includes the following steps: S10. Construct a sign language animation resource library by decomposing the sign language expression system in the National Common Sign Language Dictionary into 500 basic gesture units and 1200 composite action templates, and synthesizing sign language animations by combining artificial intelligence technology. S20. Randomly select 4 sign language animations and corresponding text words from the resource library, and perform image processing on the text words to scramble and distort interference. S30. Push an interactive webpage containing the four sign language animations and the image-processed words, start a 60-second countdown, and ask the user to click on the matching word images in the order of the animations; S40. Perform multimodal analysis based on user operation time difference and coordinate position to verify user sign language ability.
2. The intelligent sign language ability identification system based on multimodal data analysis according to claim 1, characterized in that, The basic gesture unit in S10 includes: 120 hand shape parameters, defining the finger shape and palm direction; 80 direction parameters, defining the direction of the palm movement trajectory; and 300 position parameters, defining the positioning coordinates of the palm in space.
3. The intelligent sign language ability identification system based on multimodal data analysis according to claim 1, characterized in that, The synthesis steps of the digital human sign language animation in S10 include: inputting target words and parsing their expression standards in the "National Common Sign Language Dictionary"; calling the corresponding compound action template and extracting basic gesture unit parameters; generating a continuous action sequence through AI-driven digital human model and outputting the animation file within 1 second.
4. The intelligent sign language ability identification system based on multimodal data analysis according to claim 1, characterized in that, The random sampling operation in S20 is designed to prevent users from summarizing sampling patterns, and the disordered processing of text words is used to verify the user's ability to match text words with sign language animations.
5. The intelligent sign language ability identification system based on multimodal data analysis according to claim 1, characterized in that, The distortion and interference image processing implemented on disordered text words in S20 is used to prevent technical means from capturing word information and to avoid brute-force attacks.
6. The intelligent sign language ability identification system based on multimodal data analysis according to claim 1, characterized in that, The interactive webpage layout in S30 is set as follows: four sign language animations are arranged at the top of the webpage in the order of extraction; a dotted image with image processing is set directly below each animation, and the dotted image contains distorted and interfering word image information.
7. The intelligent sign language ability identification system based on multimodal data analysis according to claim 1, characterized in that, In S30, the coordinate position of the selected image is bound to a preset range of the vocabulary image, and the user clicks on the coordinate position for the background to match specific words.
8. The intelligent sign language ability identification system based on multimodal data analysis according to claim 1, characterized in that, The timeout determination logic in S40 is as follows: if the time difference between user operations is greater than 60 seconds, a forced refresh is triggered and the verification content is regenerated.
9. The intelligent sign language ability identification system based on multimodal data analysis according to claim 1, characterized in that, The coordinate matching verification logic in S40 includes: comparing the user's selection order with the animation extraction order recorded in S20; verifying whether all selected coordinates fall within the preset word image range; and determining that the verification is successful only when the order is consistent and all coordinates match.