Smart home control methods, devices, terminal equipment, and storage media based on continuous sign language recognition

By combining data collection with continuous sign language recognition models and motion and recognition features, the problems of inaccurate sign language command recognition and unnatural interaction in smart home systems have been solved. This has enabled reliable control for hearing-impaired users and improved platform compatibility, as well as enhanced the accuracy and naturalness of sign language control.

CN123123054APending Publication Date: 2026-11-17GUANGDONG LAB OF ARTIFICIAL INTELLIGENCE & DIGITAL ECONOMY (SZ)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611613753.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-19
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing smart home control systems struggle to effectively recognize and process continuous sign language commands from hearing-impaired users, resulting in issues such as misoperation and unnatural interaction, as well as poor compatibility across different platforms.

Method used

The system collects continuous sign language data from users using a data glove, identifies and verifies the data using a continuous sign language recognition model, combines hand movements and recognition state features to determine the end of the sign language command, and uses the device knowledge base and grammar templates to determine the completeness of the command and select an appropriate control strategy to execute the command.

Benefits of technology

It improves the accuracy and stability of hearing-impaired users controlling smart home devices using sign language, reduces the probability of false triggers, and enhances the naturalness of interaction and system compatibility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123054A_ABST
    Figure CN123123054A_ABST
Patent Text Reader

Abstract

This application relates to the fields of artificial intelligence human-computer interaction and smart home IoT technology, and provides a smart home control method, device, terminal equipment, and storage medium based on continuous sign language recognition. The method includes: collecting continuous sign language action data from a user through a data glove; inputting the continuous sign language action data into a trained continuous sign language recognition model to obtain a sign language recognition result; determining whether the continuous sign language input has ended based on the continuous sign language action data and the sign language recognition result; determining whether the sign language recognition result constitutes an executable control command after determining that the continuous sign language input has ended; and controlling the target smart home device based on the sign language recognition result after determining that the sign language recognition result constitutes an executable control command. This method enables hearing-impaired users to reliably control smart homes based on sign language, reduces the probability of false sign language triggering, and improves the stability of sign language interaction control.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the fields of artificial intelligence human-computer interaction and smart home Internet of Things technology, and in particular relates to a smart home control method, device, terminal equipment and storage medium based on continuous sign language recognition. Background Technology

[0002] With the popularization of smart homes, voice interaction has become the mainstream control method. However, users with hearing or speech impairments find it difficult to use this method. Most existing gesture controls only support a few fixed gestures and cannot express complex semantics. Even continuous sign language recognition systems that can output text still have many shortcomings when directly used for smart home control. On the one hand, existing technologies have problems such as unclear boundaries of continuous sign language movements, which easily break complete control commands into multiple segments, triggering device control by recognizing partial words, causing erroneous operations, lacking a command integrity judgment mechanism, and being unable to distinguish between incomplete input and executable commands. On the other hand, they cannot rely on the previous controlled object and scene to carry out continuous interaction, resulting in poor interaction naturalness. At the same time, the interfaces and control protocols of different smart home platforms are not unified, resulting in poor system compatibility. Summary of the Invention

[0003] This application provides a smart home control method, device, terminal equipment, and storage medium based on continuous sign language recognition, which enables hearing-impaired users to reliably control smart homes based on sign language, reduces the probability of false sign language triggering, and improves the stability and usability of sign language interactive control.

[0004] In a first aspect, embodiments of this application provide a smart home control method based on continuous sign language recognition, including: The system collects continuous sign language data from users using a data glove; this data includes finger flexion data, palm posture data, and hand movement data. Continuous sign language action data is input into a trained continuous sign language recognition model to obtain sign language recognition results. The sign language recognition results include at least one candidate word sequence, the confidence score of each candidate word sequence, and intermediate recognition results. The intermediate recognition results include at least one of the following: the time step word category probability distribution corresponding to each sliding acquisition window, candidate word sequence, candidate word sequence confidence score, blank label probability, edit distance between candidate word sequences, and action boundary prediction value. Determine whether the continuous sign language input has ended based on continuous sign language movement data and sign language recognition results; After determining the end of continuous sign language input, determine whether the sign language recognition result constitutes an executable control command according to preset rules; After determining that the sign language recognition result constitutes an executable control command, the target smart home device is controlled according to the sign language recognition result.

[0005] In this embodiment, raw sign language physical data such as finger bending data, palm posture data, and hand movement data collected by data gloves, as well as intermediate recognition results output by the model, such as time step word category probability distribution, candidate word sequence, candidate word sequence confidence, blank label probability, edit distance between candidate word sequences, and action boundary prediction value, can effectively distinguish between brief gesture pauses and the actual end of a command during sign language interaction, avoiding misidentification of intermediate gesture pauses as command termination. On the other hand, after the sign language command determination is completed, executable control command verification is carried out, and the legality of the controlled device, control action, and grammatical structure of the command is screened to filter out invalid instructions that are incomplete, mismatched, or do not conform to grammatical norms. Then, multiple candidate word sequences and confidence are used to provide alternative recognition results to make up for the defects of errors in a single recognition result. The combined effect of multiple mechanisms reduces the issuance of erroneous commands, lowers the probability of sign language mis-triggering, and thus significantly improves the interaction stability and actual usability of hearing-impaired users when using sign language to control smart homes.

[0006] In one possible implementation of the first aspect, determining whether the continuous sign language input has ended based on continuous sign language action data and sign language recognition results includes: Hand movement state features are extracted from continuous sign language action data; among them, hand movement state features represent hand movement information and posture change information; Sign language recognition state features are extracted based on the sign language recognition results; among them, the sign language recognition state features represent the convergence and fluctuation information of the sign language recognition results; The changes in finger flexion, palm angular velocity, and acceleration are extracted from the characteristics of hand movement, and these changes are then normalized. Based on the characteristics of hand movement, sign language recognition, and normalized changes in finger bending, palm angular velocity, and acceleration, it is determined whether continuous sign language input has ended.

[0007] In this embodiment, by combining hand movement posture change information and convergence fluctuation information of sign language recognition results, and by introducing normalized multi-dimensional hand movement change quantities to comprehensively determine whether the sign language command has ended, the probability of misjudgment caused by single-dimensional judgment can be reduced, and the accuracy and reliability of detecting the end time of continuous sign language input can be improved.

[0008] In one possible implementation of the first aspect, determining whether continuous sign language input has ended based on hand movement state characteristics, sign language recognition state characteristics, and normalized changes in finger flexion, palm angular velocity, and acceleration includes: The hand motion state features and sign language recognition state features are normalized, and the normalized hand motion state features and sign language recognition state features are then weighted and fused according to preset weights to obtain the action boundary probability. The action boundary probability is compared with the first preset threshold, and the normalized changes in finger bending, palm angular velocity, and acceleration are compared with the second, third, and fourth preset thresholds, respectively. If the action boundary probability is greater than the first preset threshold, and the normalized change in finger bending is not greater than the second preset threshold, the normalized change in palm angular velocity is not greater than the third preset threshold, and the normalized change in acceleration is not greater than the fourth preset threshold, then the current acquisition window is determined to meet the preset criteria, and the number of sliding acquisition windows that continuously meet the preset criteria is incremented by one; if the current sliding acquisition window does not meet the preset criteria, then the number of sliding acquisition windows that continuously meet the preset criteria is reset to zero. When K consecutive sliding collection windows meet the preset criteria, the continuous sign language input is determined to have ended, where K is a preset positive integer.

[0009] In this embodiment, the action boundary probability is obtained by weighted fusion of multiple features, and combined with multi-dimensional normalized motion indicators and a joint judgment mechanism of K consecutive sliding windows, misjudgment caused by instantaneous noise of a single feature is avoided, effectively improving the accuracy and anti-interference ability of continuous sign language input end detection.

[0010] In one possible implementation of the first aspect, determining whether the sign language recognition result constitutes an executable control command includes: Extract the target smart home device, target control action, and target grammatical structure corresponding to each candidate word sequence; Multiple candidate word sequences and their corresponding confidence scores are obtained from the sign language recognition results, and the candidate word sequences are sorted from high to low confidence scores. Based on the sorting results, the candidate word sequences are sequentially subjected to device identification, control action matching, and grammatical structure verification to determine whether the sign language recognition results constitute executable control commands.

[0011] In this embodiment, the candidate word sequences are sorted in descending order of confidence and then device identification, control action matching, and grammatical structure verification are performed sequentially. The identification results with high confidence are processed first, realizing multi-dimensional verification and screening, and improving the accuracy and processing efficiency of executable control command determination.

[0012] In one possible implementation of the first aspect, the step of extracting the target smart home device corresponding to each candidate word sequence includes: If the controlled device is present in the candidate word sequence, then the controlled device is identified as the target smart home device; If the controlled device is not found in the candidate word sequence, the historical control context information corresponding to the control command that was successfully executed in the previous round is obtained; If the historical control context information is within a preset valid time range, and the current control action corresponding to the candidate word sequence matches the device corresponding to the historical control context information, the target smart home device is determined based on the historical control context information.

[0013] In this embodiment, the controlled device is extracted directly from the candidate word sequence, or the target smart home device can be completed using the historical control context within the effective time when the candidate word lacks device vocabulary. This supports simplified sign language input that omits the device, improves the ease of use of sign language control, and ensures the reliability of device determination.

[0014] In one possible implementation of the first aspect, the candidate word sequence is sequentially subjected to device identification, control action matching, and grammatical structure verification according to the sorting results to determine whether the sign language recognition result constitutes an executable control command, including: For the candidate word sequence of the identified target smart home device, the target control entry consisting of the target smart home device and the target control action corresponding to the candidate word sequence is matched with a preset device knowledge base according to the sorting result; wherein, the device knowledge base contains device attribute information and supported control actions of various smart home devices; If the device knowledge base records information that the target smart home device supports the target control action, then the target smart home device and the target control action are determined to match. After determining the target smart home device and the target control action, the target grammatical structure corresponding to the candidate word sequence is compared with the preset control command grammatical template; If the target grammatical structure conforms to the structural rules corresponding to the control command grammatical template, then the sign language recognition result is determined to constitute an executable control command. If all candidate word sequences fail the verification, the sign language recognition result is determined not to constitute an executable control command.

[0015] In this embodiment of the application, by matching the target control entries composed of device-action with the device knowledge base and combining them with grammatical template verification, the candidate word sequence is filtered layer by layer, which can effectively filter out illegal combinations of device actions and sign language recognition results with abnormal sentence structure, thereby improving the reliability of the determination of executable control commands.

[0016] In one possible implementation of the first aspect, controlling a target smart home device based on sign language recognition results includes: The target candidate word sequence in the sign language recognition results is parsed to obtain structured control information. The target candidate word sequence is the first candidate word sequence that passes the executable control command verification after being sorted from high to low confidence. The structured control information includes control intent, target smart home device, location, and control parameters. The control strategy for the target smart home device is determined according to a preset device capability table; wherein, the control strategy includes a first control strategy adopted when the target smart home device has an open interface, or a second control strategy adopted when the target smart home device does not have an open interface; the second control strategy includes at least one of voice command-based control, gateway control, and infrared forwarding control; the target smart home device is controlled according to structured control information and the control strategy.

[0017] In this embodiment, the first verified target candidate word sequence is parsed to obtain structured control information, and the corresponding control strategy is adaptively selected according to the device capability table. This can adapt to different smart home devices with and without open interfaces, broaden the device compatibility range of sign language control, and ensure the effective execution of device control.

[0018] Secondly, embodiments of this application provide a smart home control device based on continuous sign language recognition, comprising: The data acquisition module is used to collect continuous sign language data from users through a data glove; the continuous sign language data includes user finger bending data, palm posture data, and hand movement data. The recognition result acquisition module is used to input continuous sign language action data into the trained continuous sign language recognition model to obtain sign language recognition results. The sign language recognition results include at least one candidate word sequence, the confidence score of each candidate word sequence, and intermediate recognition results. The intermediate recognition results include at least one of the following: the time step word category probability distribution corresponding to each sliding acquisition window, candidate word sequence, candidate word sequence confidence score, blank label probability, edit distance between candidate word sequences, and action boundary prediction value. The first data determination module is used to determine whether the continuous sign language input has ended based on the continuous sign language action data and the sign language recognition result. The second data determination module is used to determine whether the sign language recognition result constitutes an executable control command according to preset rules after the continuous sign language input ends. The intelligent control module is used to control the target smart home device based on the sign language recognition result after determining that the sign language recognition result constitutes an executable control command.

[0019] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a smart home control method based on continuous sign language recognition as described in any of the first aspects above.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the smart home control method based on continuous sign language recognition as described in any of the first aspects above.

[0021] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the smart home control method based on continuous sign language recognition as described in any of the first aspects above.

[0022] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the smart home control method based on continuous sign language recognition provided in an embodiment of this application. Figure 2 This is a flowchart illustrating the process for determining whether continuous sign language input has ended, provided in an embodiment of this application. Figure 1 ; Figure 3 This is a flowchart illustrating the process for determining whether continuous sign language input has ended, provided in an embodiment of this application. Figure 2 ; Figure 4 This is a flowchart illustrating the process of determining whether a sign language recognition result is an executable command, as provided in an embodiment of this application. Figure 1 ; Figure 5 This is a schematic diagram of the process for determining the target smart home device provided in an embodiment of this application; Figure 6 This is a flowchart illustrating the process of determining whether a sign language recognition result is an executable command, as provided in an embodiment of this application. Figure 2 ; Figure 7This is a schematic diagram of the process for controlling the target smart home device provided in the embodiments of this application; Figure 8 This is a schematic diagram of the system architecture of the smart home control method based on continuous sign language recognition provided in the embodiments of this application; Figure 9 This is a schematic diagram of the system flow of the smart home control method based on continuous sign language recognition provided in the embodiments of this application; Figure 10 This is a schematic diagram of the overall process of the smart home control method based on continuous sign language recognition provided in the embodiments of this application; Figure 11 This is a structural block diagram of a smart home control device based on continuous sign language recognition provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0026] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0027] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0029] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0031] With the popularization of smart homes, voice interaction has become the mainstream control method. However, users with hearing or speech impairments find it difficult to use this method. Most existing gesture controls only support a few fixed gestures and cannot express complex semantics. Even continuous sign language recognition systems that can output text still have many shortcomings when directly used for smart home control. On the one hand, existing technologies have problems such as unclear boundaries of continuous sign language movements, which easily break complete control commands into multiple segments, triggering device control by recognizing partial words, causing erroneous operations, lacking a command integrity judgment mechanism, and being unable to distinguish between incomplete input and executable commands. On the other hand, they cannot rely on the previous controlled object and scene to carry out continuous interaction, resulting in poor interaction naturalness. At the same time, the interfaces and control protocols of different smart home platforms are not unified, resulting in poor system compatibility.

[0032] To address the aforementioned technical issues, this application provides a smart home control method based on continuous sign language recognition. This method uses a data glove to collect sign language action data, outputs a word sequence through continuous sign language recognition, caches the recognition results using action boundary detection to prevent premature device triggering, and then performs command integrity determination. It combines a device knowledge base, grammar templates, and an optional large language model for intent parsing. Furthermore, it can automatically complete instructions that omit the controlled object using context. Finally, based on the capabilities of the target smart home device, it automatically selects at least one of the following methods to issue control commands: direct interface control, indirect control based on voice commands, gateway control, or infrared forwarding control. This achieves smart home control, thereby solving the problems of accidental triggering, ambiguous command boundaries, poor interaction naturalness, and weak compatibility with smart home platforms associated with continuous sign language recognition, and serving users with hearing impairments or other difficulties using voice interaction.

[0033] See Figure 1 This is a flowchart illustrating a smart home control method based on continuous sign language recognition provided in an embodiment of this application. It is intended as an example and not a limitation. The method may include the following steps: S101 collects continuous sign language data from the user through a data glove; the continuous sign language data includes the user's finger bending data, palm posture data, and hand movement data.

[0034] In this embodiment of the application, a data glove hardware device is used to collect a user’s complete set of coherent sign language commands. The information collected includes not only finger bending data, but also palm posture data, hand movement data and other hand action information, which are used as the raw input data for subsequent sign language recognition.

[0035] For example, the data glove is equipped with a flexible bending sensor, an inertial measurement unit, a microcontroller, and a wireless communication module. The flexible bending sensor collects the bending values ​​of each finger joint, and the inertial measurement unit outputs the palm posture angle, three-axis acceleration, and three-axis angular velocity to characterize the palm posture and hand movement state. The microcontroller completes multi-sensor synchronous sampling and timestamps each frame of data. Then, it continuously sends out the time-series frame data containing finger bending data, palm posture data, and hand movement data through wireless communication to form a raw data stream of continuous sign language action data.

[0036] For example, when a user performs the entire set of continuous sign language actions such as "turn on the living room light", the data glove will continuously record the bending changes of each finger, the flipping and tilting posture of the palm, and the movement state of the hand in the air, and output frame by frame of sensor data in chronological order, completely recording the entire process of this set of sign language, and providing it to the subsequent continuous sign language recognition module for filtering, normalization and other preprocessing.

[0037] S102, input the continuous sign language action data into the trained continuous sign language recognition model to obtain the sign language recognition result; wherein, the sign language recognition result includes at least one candidate word sequence, the confidence score corresponding to each candidate word sequence, and intermediate recognition results; wherein, the intermediate recognition results include at least one of the following: the time step word category probability distribution corresponding to each sliding acquisition window, candidate word sequence, candidate word sequence confidence score, blank label probability, edit distance between candidate word sequences, and action boundary prediction value; In this embodiment, the preprocessed temporal feature data of sign language actions is input into the trained continuous sign language recognition model for recognition. The output is not only a sequence of words translated from sign language, but also the credibility of each candidate word sequence, as well as the internal intermediate data generated by the model during each sliding window operation. This intermediate data records the reasoning and decoding process of the model under each window.

[0038] For example, the sign language temporal feature matrix after filtering, normalization, timestamp alignment, and sliding window segmentation is input into the trained continuous sign language recognition model. The model can use temporal networks such as Convolutional Neural Network (CNN), Temporal Convolutional Network (TCN), Transformer, and Long Short-Term Memory (LSTM) in combination with decoding algorithms such as Connectionist Temporal Classification (CTC) or bundle search to extract features and model temporal dependencies of hand features within each sliding window, output multiple sets of candidate word sequences, calculate the confidence score of each set of candidate word sequences, and output intermediate recognition results. The intermediate recognition results include at least one of the following: word category probability distribution at each time step, candidate word sequence, candidate word sequence confidence score, blank label probability, candidate word sequence edit distance, action boundary prediction value, and cached word sequence, for subsequent action boundary detection and recognition result stability assessment.

[0039] For example, if a user's sign language is "turn on the living room light", the model outputs the candidate word sequence "open, living room, light" with a corresponding confidence level of 0.92. At the same time, it outputs another set of candidate word sequences with a confidence level of 0.41: "open, living room, light". The intermediate recognition results include internal calculation information such as the word probability, blank label probability, and candidate word ranking for each step of each sliding window. Subsequent modules can use this intermediate data to determine whether the sign language action has ended, rather than just using the final output word sequence.

[0040] S103, determine whether the continuous sign language input has ended based on the continuous sign language action data and sign language recognition results.

[0041] In this embodiment, the user's actual sign language action data (continuous sign language action data) is combined with the sign language recognition results output by the model. The two are combined to determine whether the user has completed the entire set of sign language commands and whether the hand movements have stopped, thereby determining the end time of a complete control command. The judgment is not based solely on the recognized text results, but also takes into account the original hand movement signals. Before the end is determined, the recognition results are only cached and will not trigger the smart home device to perform operations.

[0042] In one embodiment, see Figure 2 This is a flowchart illustrating the process for determining whether continuous sign language input has ended, provided in an embodiment of this application. Figure 1 ,like Figure 2 As shown, step S103 includes: S201, extract hand movement state features based on continuous sign language action data; among which, hand movement state features represent hand movement information and posture change information.

[0043] In this embodiment of the application, various features that can reflect the intensity of hand movements and changes in posture are calculated and extracted from the raw continuous sign language movement data (sensor time series data) collected from the data glove. These features are used to measure whether the hand is still making large-scale sign language movements or has become static, providing a motion-level basis for subsequent judgment on whether the sign language command has ended.

[0044] For example, using a sliding window as a processing unit, the finger bending value, palm angular velocity, three-axis acceleration, and palm posture angle time series data are read from the raw sensor data of continuous sign language action data. The changes in finger bending, angular velocity, acceleration, and window motion energy within the window are calculated to obtain the hand motion state characteristics, thereby characterizing the hand motion information and posture change information.

[0045] For example, when a user is performing the sign language "turn on the living room light," their hand is constantly moving and their fingers are repeatedly flexing and extending. The calculated motion energy and the values ​​of changes in various sensors are relatively high, indicating that the hand is in a state of strong motion. When the entire set of sign language is completed and the hand remains still, the changes in bending, angular velocity, and acceleration calculated by the sliding window decrease significantly, and the motion energy becomes lower. This motion state characteristic can serve as an important basis for judging that the sign language action is about to end.

[0046] The formula for calculating the degree of decrease in kinetic energy is as follows:

[0047] in: The degree of decline in physical energy; The reference motion energy is determined based on the motion energy of one or more sliding windows preceding the current sliding window, for example, by taking the average motion energy of the one or more sliding windows. : The motion energy of the current sliding window; : To prevent extremely small positive numbers with a denominator of zero; Limit the result to between 0 and 1.

[0048] In addition, the formulas for calculating finger flexion value, palm angular velocity, and triaxial acceleration are as follows:

[0049]

[0050]

[0051] in: : No. The vector of finger bending values ​​in the frame; : No. The angular velocity vector of the palm in the frame; : No. The three-axis acceleration vector of the frame; The number of frames currently contained in the sliding window; : The magnitude of the vector; : Accumulate from frame 2 to frame 3 frame.

[0052] For example, a triaxial acceleration vector consists of acceleration components in three directions:

[0053] The change in acceleration between two adjacent frames is:

[0054] S202, extract sign language recognition state features based on the sign language recognition results; wherein, the sign language recognition state features characterize the convergence and fluctuation information of the sign language recognition results.

[0055] In this embodiment of the application, sign language recognition state features are extracted from the sign language recognition results output by the continuous sign language recognition model. These features are used to reflect whether the recognition output results have stabilized and converged or are still fluctuating. They work in conjunction with hand movement state features to help determine whether the user's continuous sign language input has ended.

[0056] For example, using a sliding window as the processing unit, the candidate word sequence, confidence level, and intermediate recognition results in the sign language recognition results are read. The consistency of the optimal candidate word sequence in multiple consecutive windows, the edit distance between candidate words in adjacent windows, the fluctuation range of candidate word confidence level, and the probability of blank labels are extracted as sign language recognition state features to characterize the convergence and fluctuation information of the sign language recognition results.

[0057] For example, if a user is gesturing "turn on the living room light" and the gesture is not yet complete, the candidate word sequence output by the sliding windows keeps changing, the editing distance between adjacent windows is large, and the confidence level fluctuates. At this time, the recognition state features reflect that the result is in a fluctuating and unconverged state, and the system believes that the sign language command has not ended. When the user completes the entire set of sign language actions, the optimal candidate word sequence output by multiple sliding windows stabilizes as "turn on, living room, light", the editing distance between adjacent windows is very small, the confidence level remains at a high value and no longer jumps significantly, the probability of blank labels increases, and the recognition state features show convergence and stability, which provides a recognition-level basis for determining the end of the sign language command.

[0058] S203 extracts the changes in finger flexion, palm angular velocity, and acceleration from the hand motion state features, and normalizes these changes.

[0059] In this embodiment of the application, three types of indicators are further calculated from the already obtained hand movement state characteristics: the change in finger flexion, the change in palm angular velocity, and the change in acceleration. The changes in finger flexion, the change in palm angular velocity, and the change in acceleration are normalized to eliminate the difference in dimensions and scale the physical data of different magnitudes to a uniform numerical range. The normalized indicators are used to measure the intensity of the hand movement changes, thereby determining whether the hand is still moving or has become static, providing a quantitative basis for determining whether the sign language command has ended.

[0060] For example, within the sliding window, based on the original sensor data of each frame collected by the data glove, the magnitude of the difference between the finger bending vector, palm angular velocity vector and triaxial acceleration vector corresponding to adjacent frames is calculated respectively. Then, the average value of the changes in all adjacent frames within the window is calculated to obtain the changes in finger bending, palm angular velocity and acceleration respectively, and the above three types of changes are normalized.

[0061] For example, when a user is performing the sign language for "turn on the living room light," their fingers are constantly flexing and extending, and their palms are swinging and turning. The calculated and normalized values ​​of the changes in finger flexion, angular velocity, and acceleration are all relatively large, indicating that the hand is in motion. When the user finishes the entire set of sign language gestures and the hand remains still, all three types of changes calculated by the window decrease to below a small threshold, indicating that the hand movement is approaching stillness. This can be used as an important criterion for judging the end of the sign language action.

[0062] S204. Based on the characteristics of hand movement state, sign language recognition state, and the normalized changes in finger bending, palm angular velocity, and acceleration, determine whether the continuous sign language input has ended.

[0063] In this embodiment, determining whether a user's continuous sign language gestures have been completed is not based on a single condition, but rather on a comprehensive assessment of multiple types of information. Firstly, it considers hand movement characteristics reflecting the strength of the hand movements, as well as sign language recognition characteristics indicating the stability of the recognition result. Secondly, it combines normalized sensor-calculated indicators such as changes in finger flexion, palm angular velocity, and acceleration. Only when the hand approaches stillness and the sign language recognition result converges and stabilizes is the sign language command considered complete. If the condition is not met, the sign language is considered to be still in progress, and the recognition result is only cached, without triggering subsequent smart home control processes.

[0064] The above method combines information on hand movement posture changes and convergence fluctuations in sign language recognition results, and introduces normalized multi-dimensional hand movement changes to comprehensively determine whether a sign language command has ended. This reduces the probability of misjudgment caused by single-dimensional judgment and improves the accuracy and reliability of detecting the end of continuous sign language input.

[0065] In one embodiment, see Figure 3 This is a flowchart illustrating the process for determining whether continuous sign language input has ended, provided in an embodiment of this application. Figure 2 ,like Figure 3 As shown, step S204 includes: S301, normalize the hand motion state features and sign language recognition state features, and then weight and fuse the normalized hand motion state features and sign language recognition state features according to preset weights to obtain the action boundary probability.

[0066] In this embodiment, the hand movement state features representing the actual hand movements and the sign language recognition state features representing the stability of the recognition output are normalized and weighted respectively for fusion calculation, outputting a movement boundary probability between 0 and 1. The higher the probability value, the more likely the current moment is to be the end position of the entire set of sign language commands, providing a unified quantitative score for judging whether the sign language command has ended.

[0067] For example, the hand motion state features and sign language recognition state features are first normalized, mapping both types of features to the value range of 0 to 1. Then, pre-set or adaptively updated weight coefficients are configured respectively, and the normalized two types of features are weighted and summed. The weighted sum is then mapped through the Sigmoid function to finally obtain the action boundary probability with a value between 0 and 1.

[0068] Among them, the boundary score corresponding to the hand movement state characteristics is determined based on the degree of decrease in kinetic energy and the normalized changes in finger flexion, palm angular velocity, and acceleration, specifically:

[0069] in, Scoring is given for the hand boundary. The degree of decrease in exercise energy. , and These represent the normalized changes in finger flexion, palm angular velocity, and acceleration, respectively. , , and The corresponding preset weights.

[0070] The smaller the change in motion, the higher the hand boundary score. The boundary score corresponding to the sign language recognition state features is determined based on the consistency of the candidate word sequence, the edit distance of the candidate word sequence, the degree of confidence fluctuation, and the probability of blank labels. The more stable the candidate word sequence and the smaller the confidence fluctuation, the higher the sign language recognition boundary score.

[0071] The action boundary probability can be determined based on the weighted result of the hand boundary score and the sign language recognition boundary score, specifically as follows:

[0072] in, The hand boundary score is determined based on the degree of decrease in kinetic energy and the amount of change in kinetic energy. To score the boundary for sign language recognition, and For the corresponding weights, For bias terms, The Sigmoid function represents the degree of decrease in kinetic energy. The larger the size and the smaller the changes in various movements, the higher the hand boundary score. The higher the probability, the higher the action boundary probability. Increase.

[0073] For example, when a user signs "turn on the living room light," the hand movements are vigorous and the recognition results fluctuate continuously. The boundary contribution values ​​corresponding to the hand movement state features and the sign language recognition state features are low. After normalization and weighted fusion, the action boundary probability is only 0.2. When the user is about to complete the sign language, the hand movements weaken, and the recognition results converge and stabilize. The contribution values ​​of the two types of features increase simultaneously. After weighted fusion, the action boundary probability reaches 0.85. This high probability indicates that the current position is likely the action end boundary of the sign language command.

[0074] S302, compare the action boundary probability with the first preset threshold, and compare the normalized changes in finger bending, palm angular velocity, and acceleration with the second, third, and fourth preset thresholds, respectively.

[0075] In this embodiment, the action boundary probability is compared with a first preset threshold, and the normalized changes in finger flexion, palm angular velocity, and acceleration are compared with a second, third, and fourth preset threshold, respectively. By combining probability determination conditions with hand movement stillness conditions, the system comprehensively verifies whether the end time of the sign language command has been reached, avoiding misjudgments of the sign language action's end due to relying solely on a single action boundary probability.

[0076] For example, the weighted fusion output action boundary probability is obtained and compared with a first preset threshold P. Simultaneously, the normalized changes in finger flexion, palm angular velocity, and acceleration within the sliding window are compared with their corresponding second, third, and fourth preset thresholds, respectively. Only when the action boundary probability is greater than the first preset threshold P, and the normalized changes in finger flexion, palm angular velocity, and acceleration are not greater than the second, third, and fourth preset thresholds, respectively, is the basic condition for the termination of the sign language action met.

[0077] S303, if the action boundary probability is greater than the first preset threshold, and the normalized finger bending change is not greater than the second preset threshold, the normalized palm angular velocity change is not greater than the third preset threshold, and the normalized acceleration change is not greater than the fourth preset threshold, then the current acquisition window is determined to meet the preset indicators, and the number of sliding acquisition windows that continuously meet the preset indicators is incremented by one.

[0078] In this embodiment, determining whether continuous sign language input has ended requires simultaneously satisfying two types of constraints. The first type is a probability condition, where the probability of the action boundary is greater than a first preset threshold, indicating that the sign language action has reached its end based on comprehensive feature analysis. The second type is a hand movement condition, where the normalized changes in finger flexion, palm angular velocity, and acceleration are not greater than their respective second, third, and fourth preset thresholds, indicating that the physical movement of the hand is approaching stillness. Only when both conditions are met simultaneously is the current sliding acquisition window considered to meet the preset criteria, and the count of sliding acquisition windows that meet the preset criteria is accumulated. If either condition is not met, the count is not accumulated, and the sign language input is considered to be still ongoing.

[0079] For example, the system obtains the action boundary probability output by the current sliding window, as well as the normalized changes in finger bending, palm angular velocity, and acceleration. The action boundary probability is compared with a first preset threshold, and the three normalized motion changes are compared with the corresponding second, third, and fourth preset thresholds, respectively. Only when the action boundary probability is greater than the first preset threshold, and the normalized changes in finger bending, palm angular velocity, and acceleration are not greater than the second, third, and fourth preset thresholds, respectively, is the current acquisition window determined to meet the preset criteria, and the number of sliding acquisition windows that meet the preset criteria is incremented by one.

[0080] For example, the first preset threshold is configured as 0.7, the second preset threshold as 0.1, the third preset threshold as 0.1, and the fourth preset threshold as 0.1. When the user completes the sign language "turn on the living room light," the action boundary probability is 0.82, which is greater than 0.7. The normalized finger bending change is 0.06, the palm angular velocity change is 0.05, and the acceleration change is 0.04. All three values ​​are not greater than their respective thresholds. The current window meets the preset criteria, and the number of sliding collection windows that meet the preset criteria is incremented by one. Subsequently, this count value can be used to confirm the end of the sign language command and enter the command integrity judgment process. If the sign language action is not yet completed, even if the action boundary probability reaches 0.76, which is greater than 0.7, the normalized acceleration change is 0.12, which exceeds the fourth preset threshold. The constraints cannot be fully met, so the current window is not counted as meeting the preset criteria, the count is not accumulated, and the sign language recognition results continue to be collected and cached.

[0081] S304 If the current acquisition window does not meet the preset index, the number of sliding acquisition windows that continuously meet the preset index will be cleared to zero.

[0082] In this embodiment of the application, a sliding acquisition window is used to determine the indicators during the data acquisition process. When it is detected that the data collected in the current sliding acquisition window does not meet the preset indicator requirements, the count value used to count the number of consecutively compliant windows is directly cleared to zero. The previously accumulated number of sliding acquisition windows that continuously meet the preset indicators is no longer retained. The subsequent windows need to start counting again. Only when the subsequent sliding acquisition windows meet the preset indicators again will the continuous counting be restarted.

[0083] S305, when K consecutive sliding acquisition windows meet the preset indicators, the continuous sign language input is determined to end, where K is a preset positive integer.

[0084] In this embodiment of the application, in order to avoid misjudging the end of the sign language by chance when a single sliding window meets the conditions, it is set that the sign language command is finally determined to be over only when all K consecutive windows meet the preset indicators; K is a pre-configured positive integer. Through continuous verification of multiple windows, false triggers caused by instantaneous noise are filtered out, thereby improving the reliability of action boundary detection.

[0085] For example, after the indicator judgment of a single sliding acquisition window is completed, the cumulative number of sliding acquisition windows that meet the preset indicators is continuously counted. When it is detected that K consecutive sliding acquisition windows meet the preset indicators, the current continuous sign language input is determined to end, where K is a pre-configured positive integer. If any sliding acquisition window in the middle no longer meets the preset indicators, the cumulative count is cleared to zero and the continuous window statistics are restarted.

[0086] For example, if K is set to 3, the sign language command is considered to have ended only if all three sliding windows meet the preset criteria. After the user signs "turn on the living room light," the first, second, and third windows successively meet the preset criteria, and the continuous count reaches 3. Thus, the continuous sign language input is considered to have ended, and the system proceeds to the subsequent command integrity judgment process. If only the first window happens to meet the preset criteria, but the hand movement index of the second window exceeds the limit and no longer meets the preset criteria, the continuous count is directly reset to zero, the sign language command is not considered to have ended, and the system continues to collect sign language data.

[0087] In the above method, the action boundary probability is obtained by weighted fusion of multiple features, and the joint judgment mechanism of multi-dimensional normalized motion index and K consecutive sliding windows is combined to avoid misjudgment caused by instantaneous noise of a single feature, effectively improving the accuracy and anti-interference ability of continuous sign language input end time detection.

[0088] S104, after determining that the continuous sign language input has ended, determine whether the sign language recognition result constitutes an executable control command.

[0089] In this embodiment of the application, the content of the sign language recognition will be further verified only after it has been determined that the user has completed the entire set of sign language actions, to see if it conforms to the pre-set rules and can be regarded as a valid control command that can be directly issued to the smart home device for execution; even if the sign language action has been completed, if the semantics are incomplete or lack key elements, the device control will not be executed.

[0090] In one embodiment, see Figure 4 This is a flowchart illustrating the process of determining whether a sign language recognition result is an executable command, provided in an embodiment of this application. Figure 1 ,like Figure 4 As shown, step S104 includes: S401, extract the target smart home device, target control action, and target grammatical structure corresponding to each candidate word sequence.

[0091] In this embodiment of the application, each candidate word sequence output by the model is semantically decomposed, and the smart home device to be controlled, the corresponding control action, and the grammatical structure formed by the word arrangement are directly parsed from the sign language recognition vocabulary, providing basic feature information for subsequent semantic, logical, and grammatical compliance judgments.

[0092] For example, for the candidate word sequence "off, bedroom, air conditioner", keyword recognition can directly extract the target smart home device as air conditioner, the target control action as off, and the target grammatical structure as a compliant sentence of "control action + location + device". For the candidate word sequence "light, turn on, living room", the target device can also be extracted as light and the target action as turn on, but the matched target grammatical structure has a disordered word order and does not conform to the standard command sentence, which provides a basis for subsequent grammatical structure verification and logical screening.

[0093] S402, obtain multiple candidate word sequences and their corresponding confidence scores from the sign language recognition results, and sort the candidate word sequences from high to low confidence scores.

[0094] In this embodiment, all candidate word sequences and their corresponding confidence scores are extracted from the sign language recognition results and sorted in descending order according to the confidence scores, with the candidate word sequences with higher confidence scores placed first, so that the element analysis and verification of the high-confidence candidate words can be carried out first in the subsequent process.

[0095] For example, all candidate word sequences and the confidence score of each candidate word sequence are read from the sign language recognition results. The confidence score is used as the sorting criterion, and the candidate word sequences are sorted in descending order of confidence score. After sorting, the candidate word sequences with the highest confidence scores are selected first for subsequent element extraction and legality verification. For example, if the sign language recognition outputs three sets of candidate word sequences, "turn on the living room light" with a confidence score of 0.86, "turn on the bedroom light" with a confidence score of 0.71, and "turn off the living room light" with a confidence score of 0.53, after sorting them in descending order of confidence score, the order is "turn on the living room light", "turn on the bedroom light", and "turn off the living room light". The candidate word sequence with the highest confidence score is then processed first.

[0096] In one embodiment, see Figure 5 This is a flowchart illustrating the process of determining the target smart home device provided in an embodiment of this application, such as... Figure 5 As shown, step S402, which involves extracting the target smart home device corresponding to each candidate word sequence, includes: S501, if the controlled device exists in the candidate word sequence, then the controlled device is identified as the target smart home device.

[0097] In this embodiment of the application, the word of the controlled device is identified in the candidate word sequence. If it is found, the controlled device is directly identified as the target smart home device in this round of analysis.

[0098] For example, iterate through each word in the current candidate word sequence and determine whether there is a word for controlled device in the candidate word sequence. If there is a controlled device in the candidate word sequence, then the controlled device is identified as the target smart home device; if no controlled device is identified in the candidate word sequence, then the current candidate word sequence lacks device elements and the target smart home device is not output.

[0099] For example, if the candidate word sequence is "open, living room, curtains", and the word "curtains" is the controlled device word in the sequence, then the curtains will be identified as the target smart home device; if the candidate word sequence is only "turn on, open", and the word "controlled device" is not in the sequence, then the candidate word sequence cannot identify the target smart home device, and the command element will be determined to be missing.

[0100] S502, if the controlled device is not found in the candidate word sequence, then obtain the historical control context information corresponding to the control command that was successfully executed in the previous round.

[0101] In this embodiment of the application, when the name of the controlled device is not identified in the candidate word sequence, the system will not directly determine that the command is invalid. Instead, it will read the saved historical control context information of the previous round and try to fill in the missing controlled device object based on the previous interaction record for subsequent command parsing.

[0102] For example, the candidate word sequence is traversed to perform keyword retrieval. If no device-related words are matched after the retrieval, the historical control context information corresponding to the control command that was successfully executed in the previous round is retrieved from the system memory. This context records at least the target smart home device, device location, control action, control time, control scenario, and control execution result that were successfully executed in the previous round, in order to fill in the currently missing controlled device.

[0103] S503, when the historical control context information is within a preset valid time range and the current control action corresponding to the candidate word sequence matches the device corresponding to the historical control context information, the target smart home device is determined based on the historical control context information.

[0104] In this embodiment, when the historical control context is still within the valid time and the control action in the candidate word can be adapted to the device recorded in the historical context, the device in the historical context is reused as the target smart home device for this time, so as to achieve the effect of omitting the device name in the sign language command.

[0105] For example, it is determined whether the historical control context information is within a preset valid time range, and whether the current control action corresponding to the candidate word sequence matches the device corresponding to the historical control context information; when the historical control context information is within a preset valid time range, and the current control action matches the device corresponding to the historical context, the target smart home device is determined based on the historical control context information.

[0106] For example, if the previous sign language control was "open the living room curtains", the historical control context records the target smart home device as the living room curtains, with a preset effective time of 10 seconds. Within the 10-second effective time, if the user inputs sign language that corresponds only to the candidate word sequence "close", the current control action is extracted as "close". The closing action matches the living room curtain device in the historical context, so the historical control context is reused to identify the living room curtains as the target smart home device for this round. If more than 10 seconds have passed since the last control, the historical context is no longer used to complete the device, and the candidate word sequence is deemed incomplete due to the lack of controlled device elements.

[0107] In the above method, the controlled device is extracted directly from the candidate word sequence, or the target smart home device can be completed by using the historical control context within the effective time when the candidate word lacks device vocabulary. This supports simplified sign language input that omits the device, improves the ease of use of sign language control, and ensures the reliability of device identification.

[0108] S403, according to the sorting results, the candidate word sequence is sequentially subjected to device recognition, control action matching and grammatical structure verification in order to determine whether the sign language recognition result constitutes an executable control command.

[0109] In this embodiment, based on the candidate word sequence ordered from high to low confidence, device identification, control action matching, and grammatical structure verification are performed on each candidate word sequence one by one. The identification results with high confidence are verified first, and the verification results are used to determine whether the sign language command can be a control command that can be directly executed.

[0110] In the above method, the candidate word sequences are sorted in descending order of confidence and then device identification, control action matching, and grammatical structure verification are carried out in sequence. The identification results with high confidence are processed first, realizing multi-dimensional verification and screening, and improving the accuracy and processing efficiency of executable control command determination.

[0111] In one embodiment, see Figure 6 This is a flowchart illustrating the process of determining whether a sign language recognition result is an executable command, provided in an embodiment of this application. Figure 2 ,like Figure 6 As shown, step S403 includes: S601, for the candidate word sequence of the identified target smart home device, according to the sorting result, the target control entry consisting of the target smart home device and the target control action corresponding to the candidate word sequence is matched with the preset device knowledge base; wherein, the device knowledge base contains device attribute information and supported control actions of various smart home devices.

[0112] In this embodiment of the application, the target smart home device and the target control action are combined into a target control entry according to the candidate word sequence order after confidence level. The entry is then compared and verified with the device knowledge base. The device attributes and the control actions that the device is allowed to perform are stored in the knowledge base to verify whether the "device-action" combination is legal and valid.

[0113] For example, according to the candidate word sequence sorted by confidence, the target smart home device and target control action obtained from each candidate word sequence are combined to form a target control entry. The target control entry is then matched and queried with a preset device knowledge base. The device knowledge base stores the device attribute information corresponding to various smart home devices and all the control actions supported by the device itself. This verifies whether the control action in the current target control entry is an action allowed to be executed by the corresponding smart home device.

[0114] S602, if the device knowledge base records information that the target smart home device supports the target control action, then it is determined that the target smart home device and the target control action match.

[0115] In this embodiment of the application, the device knowledge base is queried to see if it records that the target smart home device can perform the target control action; if such a record exists, it means that the device and the control action match, and the device-action combination is legal.

[0116] For example, a preset device knowledge base is queried to determine whether there is a record in the device knowledge base that the target smart home device supports the target control action. If the device knowledge base contains a record that the target smart home device supports the target control action, then the target smart home device and the target control action are determined to match, meaning that the device-action combination is legal and usable. Otherwise, if the device knowledge base does not contain a corresponding supporting record, then the two are determined to be mismatched, and the control command corresponding to the candidate word sequence cannot be executed.

[0117] For example, if the target smart home device is a living room air conditioner and the target control action is to raise the temperature, the device knowledge base is searched. If the knowledge base records that the living room air conditioner supports the control action of raising the temperature, then the living room air conditioner and raising the temperature are matched. However, if the target smart home device is a human body sensor and the target control action is to adjust the brightness of the lights, and there is no record in the device knowledge base that the human body sensor supports turning off the brightness, then the device and the control action are not matched, and the candidate word sequence is discarded.

[0118] S603 After determining the target smart home device and the target control action match, the target grammatical structure corresponding to the candidate word sequence is compared with the preset control command grammatical template.

[0119] In this embodiment of the application, after the legality matching verification of the device and the control action is completed, the grammar verification is carried out. The target grammar structure parsed from the candidate word sequence is compared with the pre-set control command grammar template to determine whether the combination of words conforms to the sentence pattern of smart home control commands and to filter out candidates with disordered word order and abnormal sentence pattern.

[0120] For example, based on the determination of the target smart home device and the target control action, the target grammatical structure obtained by parsing the candidate word sequence is obtained, and the target grammatical structure is compared with multiple preset control command grammatical templates one by one to determine whether the current sentence structure matches any legal grammatical template.

[0121] S604. If the target grammatical structure conforms to the structural rules corresponding to the control command grammatical template, then the sign language recognition result is determined to constitute an executable control command.

[0122] In this embodiment of the application, if the target grammatical structure of the candidate word sequence meets the structural rules of the preset control command grammatical template, and the sentence semantic compliance is further confirmed based on device-action matching, the current candidate word sequence is determined to constitute an executable control command.

[0123] For example, assuming that the target smart home device and the target control action have been matched, it is determined whether the target grammatical structure corresponding to the candidate word sequence conforms to the structural rules corresponding to the control command grammatical template. If the target grammatical structure conforms to the structural rules corresponding to the control command grammatical template, the candidate word sequence, i.e. the sign language recognition result, is determined to constitute an executable control command and can proceed to the subsequent intent parsing and instruction generation stage. If the target grammatical structure does not meet the structural rules of the grammatical template, it is determined not to constitute an executable control command.

[0124] For example, if the candidate word sequence is "close, living room, curtains", the device and action verification match, and the target grammatical structure "control action-position-controlled device" conforms to the structural rules of the preset grammatical template, then the candidate word sequence is determined to constitute an executable control command; however, if the candidate word sequence is "curtains, living room, close", although the device and control action match, the target grammatical structure formed by the word arrangement does not conform to the structural rules of the template, then it is determined not to constitute an executable control command.

[0125] S605, when all candidate word sequences fail the verification, the sign language recognition result is determined not to constitute an executable control command.

[0126] In this embodiment of the application, after traversing all candidate word sequences from high to low confidence, if each candidate word sequence fails to pass all verification items such as device action matching and grammatical structure verification, it is determined that the sign language recognition result does not constitute an executable control command and will not enter the subsequent intent parsing and device instruction generation process.

[0127] For example, device action matching and grammatical structure verification are performed on each candidate word sequence in order of confidence. If, after traversing all candidate word sequences, none of them pass the complete verification, it is determined that the sign language recognition result does not constitute an executable control command and will not enter the subsequent intent parsing and device instruction generation process.

[0128] For example, the sign language recognition outputs two candidate word sequences. The first candidate word sequence is "open, temperature sensor". The device and action matching verification fails. The second candidate word sequence is "curtains, raise, bedroom". Although the device action matching is successful, the target grammatical structure does not conform to the preset grammatical template. Neither candidate word sequence passes all verifications. At this time, it is determined that the sign language recognition result does not constitute an executable control command, and the system waits to receive the next round of sign language input.

[0129] In the above method, by matching the target control entries composed of devices and actions with the device knowledge base and combining them with grammatical template verification, the candidate word sequences are filtered layer by layer, which can effectively filter out illegal combinations of device actions and sign language recognition results with abnormal sentence structures, thereby improving the reliability of the determination of executable control commands.

[0130] S105, after determining that the sign language recognition result constitutes an executable control command, controls the target smart home device according to the sign language recognition result.

[0131] In this embodiment, once the entire sign language command has passed all the preceding checks and is confirmed to be a legal and valid executable control command, the system generates corresponding control instructions based on the sign language recognition results to operate the pre-determined target smart home device and complete the device operation corresponding to the sign language. If the preceding checks fail, no control operation will be performed on the device.

[0132] In the above method, executable control commands are verified by pre-set rules. The legality of the controlled device, control action, and grammatical structure of the command is screened to filter out invalid commands that are incomplete, mismatched, or do not conform to grammatical norms. Then, multiple candidate word sequences and confidence levels are used to provide alternative recognition results to make up for the defects of errors in a single recognition result. The combined effect of multiple mechanisms reduces the issuance of erroneous commands, lowers the probability of sign language mis-triggering, and thus significantly improves the interaction stability and actual usability of hearing-impaired users when using sign language to control smart home devices.

[0133] In one embodiment, see Figure 7 This is a schematic diagram of the process for controlling the target smart home device provided in the embodiments of this application, such as... Figure 7 As shown, step S105 includes: S701, perform intent parsing on the target candidate word sequence of the sign language recognition result to obtain structured control information; wherein, the target candidate word sequence is the first candidate word sequence that passes the executable control command verification after being sorted from high to low confidence; the structured control information includes control intent, target smart home device, location and control parameters.

[0134] In this embodiment, the first candidate word sequence that passes all verifications after confidence ranking is selected as the target candidate word sequence. Intent parsing is performed on it to output standardized structured control information. This information organizes the control intent, controlled device, location, and control parameters for subsequent generation of device control commands.

[0135] For example, the target candidate word sequence is semantically split and parsed to extract the control intent, target smart home device, location, and control parameters, which are then assembled into key-value structured control information. For instance, if the target candidate word sequence is "brighten, living room, light, 70%", after intent parsing, the structured control information is obtained, namely, the control intent is brightness adjustment, the target smart home device is the living room light, the location is the living room, and the control parameter is brightness 70%. Subsequently, the system directly reads this structured control information and sends a command to the living room light to adjust the brightness to 70%, without having to repeatedly parse the original word sequence.

[0136] In another embodiment, the intent parsing module inputs at least one of the following into a large language model: a target candidate word sequence, a target smart home device, a device location, and a target control action. The large language model then outputs structured control information containing the control intent, the target smart home device, the location, and control parameters. The output structured control information can also be validated against a device knowledge base and a control command syntax template; if the output does not conform to device capabilities or syntax rules, the output is discarded or intent parsing is performed again.

[0137] S702, determine the control strategy for the target smart home device according to the preset device capability table; wherein, the control strategy includes a first control strategy adopted when the target smart home device has an open interface, or a second control strategy adopted when the target smart home device does not have an open interface; the second control strategy includes at least one of voice command-based control, gateway control and infrared forwarding control.

[0138] In this embodiment, a pre-configured device capability table is queried, and a corresponding control strategy is selected based on whether the target smart home device has an open interface. If the device has an open interface, the first control strategy is used; if it does not have an open interface, the second control strategy is used. The second control strategy includes at least one of voice command-based control, gateway control, and infrared forwarding control, thereby adapting to devices with different hardware conditions and selecting the execution method for subsequent control commands.

[0139] For example, using a identified target smart home device as an index, a preset device capability table is retrieved. This table stores the interface capability attributes of each device. Based on the records in the table, it is determined whether the device has an open interface. If it has an open interface, a first control strategy is selected; otherwise, a second control strategy is selected. For instance, if the target smart home device is a living room air conditioner, the device capability table indicates that the device has an open network interface, so the first control strategy is selected, allowing direct control by sending structured control information to the device interface. If the target smart home device is a modified bedroom fan, the device capability table indicates that the device does not have an open interface, so the second control strategy is selected, requiring indirect methods such as voice command control, gateway relay, and infrared forwarding to achieve device control.

[0140] In another embodiment, when the target smart home device does not have an open interface but supports voice control, the second control strategy is an indirect control strategy based on voice commands. The control terminal generates corresponding voice control commands based on structured control information. These voice control commands include at least one of the following: target smart home device, device location, control action, and control parameters. The voice control commands are then output through a speaker, allowing the smart speaker, voice control gateway, or the voice receiving module of the target smart home device to collect and recognize the commands, and control the target smart home device based on the recognition result. For example, when the structured control information is "living room light brightness 70%", the control terminal generates a voice control command "adjust the living room light brightness to 70%", plays the command through a speaker, and the smart speaker recognizes and controls the living room light.

[0141] S703 controls target smart home devices based on structured control information and control strategies.

[0142] In this embodiment, the parsed structured control information is used in conjunction with the previously selected control strategy to send instructions to the target smart home device to complete the control. Different control strategies will use different instruction sending methods to implement the control intent on the device.

[0143] For example, the parsed structured control information is encapsulated and sent as instructions according to the selected control strategy. If it is the first control strategy, the structured control information is converted into a device-recognizable message and sent directly according to the communication protocol of the device's open interface. If it is the second control strategy, the structured control information is forwarded to the infrared gateway or relay device, and the relay device performs the control.

[0144] For example, if the structured control information is: control intent is to brighten, target smart home device: living room light, location: living room, control parameter: brightness 70%, and the first control strategy is selected, the system directly calls the living room light's open interface and sends the brightness 70% command, and the light fixture directly performs the brightness adjustment operation; if the second control strategy is selected, the control information is sent to the infrared gateway, and the gateway emits an infrared signal to adjust the brightness of the light fixture.

[0145] In the above method, the first verified target candidate word sequence is selected to parse the structured control information, and the corresponding control strategy is adaptively selected according to the device capability table. This method can be adapted to different smart home devices with and without open interfaces, thus broadening the device compatibility range of sign language control and ensuring the effective execution of device control.

[0146] See Figure 8 This is a schematic diagram of the system architecture of the smart home control method based on continuous sign language recognition provided in the embodiments of this application, as shown below. Figure 8 As shown, it includes: Input acquisition layer: It includes a data acquisition module, which is responsible for collecting raw data of the user's sign language actions (posture, gesture timing data) and outputting the collected data to the intermediate command understanding layer.

[0147] The command understanding layer (core processing layer) executes four steps sequentially: Continuous sign language recognition: Recognizes the collected continuous sign language action data and outputs sign language recognition results such as candidate word sequences; Action boundary detection: Determine when a sign language action ends and distinguish the start and end of a complete sign language command; Command integrity determination: Verify the recognized sign language commands, check whether the equipment, actions, and syntax are legal and valid, and determine whether they are executable control commands; Intent parsing: Transform the legal sign language text (target candidate word sequence) into structured control information and output it to the control execution layer.

[0148] Control execution layer: Receive the structured information output by the command understanding layer and execute it sequentially: Control strategy generation: Select an appropriate control strategy based on the device's hardware capabilities; Smart home control: Issue commands according to the selected strategy to complete the actual control of smart home devices.

[0149] See Figure 9 This is a schematic diagram of the system flow of the smart home control method based on continuous sign language recognition provided in the embodiments of this application, as shown below. Figure 9 As shown, it includes: User continuous sign language: The system input is a series of continuous sign language gestures performed by the user to control home appliances.

[0150] Data glove acquisition: Relying on data gloves to capture raw sensor data such as hand movements and postures.

[0151] Continuous sign language recognition: Recognize sensor data and output candidate text results corresponding to the sign language.

[0152] Action boundary detection: Determines whether continuous sign language input has ended and locates the end position of a single sign language command.

[0153] Command integrity check: Verify the recognized sign language commands, check whether the device, action, and grammar are legal and valid, and filter out invalid commands.

[0154] Intent parsing: Parsing qualified sign language results into standardized structured control information (actions, equipment, positions, parameters).

[0155] Control strategy generation: Select the appropriate control distribution strategy based on the device's hardware capabilities.

[0156] Smart home device control: Execute commands to complete the actual operation of smart home devices.

[0157] See Figure 10 This is a schematic diagram of the overall process of the smart home control method based on continuous sign language recognition provided in the embodiments of this application, as follows: Figure 10 As shown, it includes: Process start: Start → User continuous sign language input → Data gloves collect hand sensor data → Data preprocessing: clean the raw sensor data and send it to the continuous sign language recognition module.

[0158] Loop identification branches: Perform continuous sign language recognition and cache the sign language recognition results; Determine if the action has ended: No (action not completed): Proceed to the "Continue Recognition" branch, return to continuous sign language recognition, and continue to collect and recognize sign language data; Yes (sign language gesture ends): Continue executing the command integrity check.

[0159] Command validity check branch: Determine if the command is complete and valid: No (incomplete or invalid command): Enter "Continue to wait for input", return to continuous sign language recognition, and wait for the user to input a new round of sign language; Yes (command complete and valid): Intent parsing and control policy generation will be executed sequentially.

[0160] The device interface capability branch determines whether the target device supports open interfaces: Yes: Direct interface control is adopted, and commands are issued by calling the device's open interface; No: Control commands are issued using an indirect control method based on voice commands.

[0161] Final execution: The two branches eventually converge to control smart home devices and complete device operation.

[0162] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0163] Corresponding to the smart home control method based on continuous sign language recognition in the above embodiments, Figure 11 This is a structural block diagram of a smart home control device based on continuous sign language recognition provided in the embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0164] Reference Figure 11 The device includes: The data acquisition module 111 is used to collect continuous sign language movement data of the user through the data glove; wherein, the continuous sign language movement data includes the user's finger bending data, palm posture data and hand movement data; The recognition result acquisition module 112 is used to input continuous sign language action data into the trained continuous sign language recognition model to obtain sign language recognition results; wherein, the sign language recognition results include at least one candidate word sequence, the confidence score corresponding to each candidate word sequence, and intermediate recognition results; wherein, the intermediate recognition results include at least one of the following: time step word category probability distribution corresponding to each sliding acquisition window, candidate word sequence, candidate word sequence confidence score, blank label probability, edit distance between candidate word sequences, and action boundary prediction value; The first data determination module 113 is used to determine whether the continuous sign language input has ended based on the continuous sign language action data and the sign language recognition result. The second data determination module 114 is used to determine whether the sign language recognition result constitutes an executable control command of a preset rule after the determination of continuous sign language input ends. The intelligent control module 115 is used to control the target smart home device based on the sign language recognition result after determining that the sign language recognition result constitutes an executable control command.

[0165] Optionally, the first data determination module 113 is also used for: Hand movement state features are extracted from continuous sign language action data; among them, hand movement state features represent hand movement information and posture change information; Sign language recognition state features are extracted based on the sign language recognition results; among them, the sign language recognition state features represent the convergence and fluctuation information of the sign language recognition results; The changes in finger flexion, palm angular velocity, and acceleration are extracted from the characteristics of hand movement, and these changes are then normalized. Based on the characteristics of hand movement, sign language recognition, and normalized changes in finger bending, palm angular velocity, and acceleration, it is determined whether continuous sign language input has ended.

[0166] Optionally, the first data determination module 113 is also used for: The hand motion state features and sign language recognition state features are normalized, and the normalized hand motion state features and sign language recognition state features are then weighted and fused according to preset weights to obtain the action boundary probability. The action boundary probability is compared with the first preset threshold, and the normalized changes in finger bending, palm angular velocity, and acceleration are compared with the second, third, and fourth preset thresholds, respectively. If the action boundary probability is greater than the first preset threshold, and the normalized finger bending change is not greater than the second preset threshold, the normalized palm angular velocity change is not greater than the third preset threshold, and the normalized acceleration change is not greater than the fourth preset threshold, then the current acquisition window is determined to meet the preset index, and the number of sliding acquisition windows that continuously meet the preset index is incremented by one. If the current acquisition window does not meet the preset indicators, the number of sliding acquisition windows that continuously meet the preset indicators will be cleared to zero. When K consecutive sliding collection windows meet the preset criteria, the continuous sign language input is determined to have ended, where K is a preset positive integer.

[0167] Optionally, the second data determination module 114 is also used for: Extract the target smart home device, target control action, and target grammatical structure corresponding to each candidate word sequence; Multiple candidate word sequences and their corresponding confidence scores are obtained from the sign language recognition results, and the candidate word sequences are sorted from high to low confidence scores. Based on the sorting results, the candidate word sequences are sequentially subjected to device identification, control action matching, and grammatical structure verification to determine whether the sign language recognition results constitute executable control commands.

[0168] Optionally, the second data determination module 114 is also used for: If the controlled device is present in the candidate word sequence, then the controlled device is identified as the target smart home device; If the controlled device is not found in the candidate word sequence, the historical control context information corresponding to the control command that was successfully executed in the previous round is obtained; If the historical control context information is within a preset valid time range, and the current control action corresponding to the candidate word sequence matches the device corresponding to the historical control context information, the target smart home device is determined based on the historical control context information.

[0169] Optionally, the second data determination module 114 is also used for: For the candidate word sequence of the identified target smart home device, the target control entry consisting of the target smart home device and the target control action corresponding to the candidate word sequence is matched with a preset device knowledge base according to the sorting result; wherein, the device knowledge base contains device attribute information and supported control actions of various smart home devices; If the device knowledge base records information that the target smart home device supports the target control action, then the target smart home device and the target control action are determined to match. After determining the target smart home device and the target control action, the target grammatical structure corresponding to the candidate word sequence is compared with the preset control command grammatical template; If the target grammatical structure conforms to the structural rules corresponding to the control command grammatical template, then the sign language recognition result is determined to constitute an executable control command. If all candidate word sequences fail the verification, the sign language recognition result is determined not to constitute an executable control command.

[0170] Optionally, the intelligent control module 115 is also used for: The target candidate word sequence in the sign language recognition results is parsed to obtain structured control information. The target candidate word sequence is the first candidate word sequence that passes the executable control command verification after being sorted from high to low confidence. The structured control information includes control intent, target smart home device, location, and control parameters. The control strategy for the target smart home device is determined according to a preset device capability table; wherein, the control strategy includes a first control strategy adopted when the target smart home device has an open interface, or a second control strategy adopted when the target smart home device does not have an open interface; the second control strategy includes at least one of voice command-based control, gateway control, and infrared forwarding control; the target smart home device is controlled according to structured control information and the control strategy.

[0171] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0172] in addition, Figure 11 The smart home control device based on continuous sign language recognition shown can be a software unit, a hardware unit, or a combination of software and hardware built into an existing terminal device. It can also be integrated into the terminal device as an independent accessory, or exist as a standalone terminal device.

[0173] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0174] Figure 12 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. For example... Figure 12 As shown, the terminal device 12 of this embodiment includes: at least one processor 120 ( Figure 12 (Only one is shown in the image), memory 121, and computer program 122 stored in memory 121 and executable on at least one processor 120, wherein processor 120 executes computer program 122 to implement the steps in any of the above embodiments of the smart home control method based on continuous sign language recognition.

[0175] The terminal device can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 12 This is merely an example of terminal device 12 and does not constitute a limitation on terminal device 12. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0176] The processor 120 may be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0177] In some embodiments, memory 121 may be an internal storage unit of terminal device 12, such as a hard disk or memory of terminal device 12. In other embodiments, memory 121 may be an external storage device of terminal device 12, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on terminal device 12. Furthermore, memory 121 may include both internal and external storage units of terminal device 12. Memory 121 is used to store operating system, application programs, boot loader, data, and other programs, such as program code of computer programs. Memory 121 may also be used to temporarily store data that has been output or will be output.

[0178] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0179] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0180] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0181] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0182] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0183] In the embodiments provided in this application, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0184] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A smart home control method based on continuous sign language recognition, characterized in that, The method includes: The user's continuous sign language movement data is collected through a data glove; wherein, the continuous sign language movement data includes the user's finger bending data, palm posture data, and hand movement data; The continuous sign language action data is input into the trained continuous sign language recognition model to obtain sign language recognition results; wherein, the sign language recognition results include at least one candidate word sequence, the confidence score corresponding to each candidate word sequence, and intermediate recognition results; wherein, the intermediate recognition results include at least one of the following: time step word category probability distribution corresponding to each sliding acquisition window, candidate word sequence, candidate word sequence confidence score, blank label probability, edit distance between candidate word sequences, and action boundary prediction value; Based on the continuous sign language action data and the sign language recognition result, it is determined whether the continuous sign language input has ended; After the continuous sign language input has ended, it is determined whether the sign language recognition result constitutes an executable control command; After determining that the sign language recognition result constitutes the executable control command, the target smart home device is controlled according to the sign language recognition result.

2. The smart home control method based on continuous sign language recognition as described in claim 1, characterized in that, Based on the continuous sign language movement data and the sign language recognition result, determine whether the continuous sign language input has ended, including: Hand movement state features are extracted from the continuous sign language action data; wherein, the hand movement state features represent hand movement information and posture change information; Sign language recognition state features are extracted based on the sign language recognition results; wherein, the sign language recognition state features characterize the convergence and fluctuation information of the sign language recognition results; The changes in finger flexion, palm angular velocity, and acceleration are extracted from the characteristics of hand movement, and these changes are then normalized. Based on the hand movement state characteristics, the sign language recognition state characteristics, and the normalized changes in finger bending, palm angular velocity, and acceleration, it is determined whether the continuous sign language input has ended.

3. The smart home control method based on continuous sign language recognition as described in claim 2, characterized in that, The step of determining whether continuous sign language input has ended based on the hand movement state features, the sign language recognition state features, and the normalized changes in finger flexion, palm angular velocity, and acceleration includes: The hand movement state features and the sign language recognition state features are normalized, and the normalized hand movement state features and the sign language recognition state features are weighted and fused according to preset weights to obtain the action boundary probability. The action boundary probability is compared with the first preset threshold, and the normalized changes in finger bending, palm angular velocity, and acceleration are compared with the second, third, and fourth preset thresholds, respectively. If the action boundary probability is greater than the first preset threshold, and the normalized change in finger bending is not greater than the second preset threshold, the normalized change in palm angular velocity is not greater than the third preset threshold, and the normalized change in acceleration is not greater than the fourth preset threshold, then the current acquisition window is determined to meet the preset index, and the number of sliding acquisition windows that continuously meet the preset index is incremented by one. If the current sliding acquisition window does not meet the preset index, the number of sliding acquisition windows that continuously meet the preset index will be cleared to zero. When K consecutive sliding acquisition windows meet the preset indicators, the continuous sign language input is determined to have ended, where K is a preset positive integer.

4. The smart home control method based on continuous sign language recognition as described in claim 1, characterized in that, The step of determining whether the sign language recognition result constitutes an executable control command includes: Extract the target smart home device, target control action, and target grammatical structure corresponding to each candidate word sequence; Multiple candidate word sequences and their corresponding confidence scores are obtained from the sign language recognition results, and the candidate word sequences are sorted from high to low confidence scores. Based on the sorting results, the candidate word sequence is sequentially subjected to device identification, control action matching, and grammatical structure verification to determine whether the sign language recognition result constitutes an executable control command.

5. The smart home control method based on continuous sign language recognition as described in claim 4, characterized in that, The steps for extracting the target smart home device corresponding to each candidate word sequence include: If a controlled device exists in the candidate word sequence, then the controlled device is identified as the target smart home device; If the controlled device is not found in the candidate word sequence, then the historical control context information corresponding to the control command that was successfully executed in the previous round is obtained; If the historical control context information is within a preset valid time range, and the current control action corresponding to the candidate word sequence matches the device corresponding to the historical control context information, the target smart home device is determined based on the historical control context information.

6. The smart home control method based on continuous sign language recognition as described in claim 5, characterized in that, The step of sequentially performing device identification, control action matching, and grammatical structure verification on the candidate word sequence according to the sorting results to determine whether the sign language recognition result constitutes an executable control command includes: For the candidate word sequence of the identified target smart home device, the target control entry consisting of the target smart home device and the target control action corresponding to the candidate word sequence is matched with a preset device knowledge base according to the sorting result; wherein, the device knowledge base contains device attribute information and supported control actions of various smart home devices; If the device knowledge base records information that the target smart home device supports the target control action, then it is determined that the target smart home device and the target control action match. After determining that the target smart home device and the target control action match, the target grammatical structure corresponding to the candidate word sequence is compared with the preset control command grammatical template; If the target grammatical structure conforms to the structural rules corresponding to the control command grammatical template, then the sign language recognition result is determined to constitute the executable control command; If all candidate word sequences fail the verification, the sign language recognition result is determined not to constitute an executable control command.

7. The smart home control method based on continuous sign language recognition as described in any one of claims 1 to 6, characterized in that, The step of controlling the target smart home device based on the sign language recognition result includes: The target candidate word sequence in the sign language recognition result is parsed to obtain structured control information; wherein, the target candidate word sequence is the first candidate word sequence that passes the executable control command verification after being sorted from high to low confidence; the structured control information includes control intent, target smart home device, location, and control parameters; The control strategy for the target smart home device is determined according to a preset device capability table; wherein, the control strategy includes a first control strategy adopted when the target smart home device has an open interface, or a second control strategy adopted when the target smart home device does not have an open interface; the second control strategy includes at least one of voice command-based control, gateway control, and infrared forwarding control; The target smart home device is controlled according to the structured control information and the control strategy.

8. A smart home control device based on continuous sign language recognition, characterized in that, include: The data acquisition module is used to collect continuous sign language movement data of the user through a data glove; wherein, the continuous sign language movement data includes the user's finger bending data, palm posture data, and hand movement data; The recognition result acquisition module is used to input the continuous sign language action data into the trained continuous sign language recognition model to obtain the sign language recognition result; wherein, the sign language recognition result includes at least one candidate word sequence, the confidence score corresponding to each candidate word sequence, and intermediate recognition results; wherein, the intermediate recognition results include at least one of the following: time step word category probability distribution corresponding to each sliding acquisition window, candidate word sequence, candidate word sequence confidence score, blank label probability, edit distance between candidate word sequences, and action boundary prediction value; The first data determination module is used to determine whether the continuous sign language input has ended based on the continuous sign language action data and the sign language recognition result. The second data determination module is used to determine whether the sign language recognition result constitutes an executable control command after the continuous sign language input has ended. The intelligent control module is used to control the target smart home device according to the sign language recognition result after determining that the sign language recognition result constitutes the executable control command.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.