Intelligent control method and device, electronic equipment and readable storage medium

By performing multi-dimensional and contextual normalization on gestures and voice commands, and combining multi-attention mechanisms and confidence selection, the problem of data structure differences and conflict handling in multimodal data fusion is solved, achieving more efficient user intent determination and interactive experience.

CN121956605APending Publication Date: 2026-05-01GREE ELECTRIC APPLIANCE INC OF ZHUHAI +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202511942320.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-12-22
Publication Date
2026-05-01

AI Technical Summary

Technical Problem

In intelligent control scenarios involving multimodal data fusion, the differences in data structure and distribution characteristics between visual and auditory information make it difficult to accurately adapt them and lack dynamic adjustment capabilities. This makes it difficult to determine the user's true intentions and makes it impossible to efficiently and accurately handle command conflicts.

Method used

By performing multi-dimensional normalization and contextual semantic awareness normalization on gestures and voice commands respectively, a fusion vector is generated. A multi-attention mechanism is then used for fusion, and the target command is selected based on confidence.

Benefits of technology

It improves the accuracy of data fusion and the precision of command execution, and can intelligently and flexibly handle modal data conflicts, providing a convenient and efficient user interaction experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121956605A_ABST
    Figure CN121956605A_ABST
Patent Text Reader

Abstract

The invention discloses an intelligent control method and device, electronic equipment and a readable storage medium, and belongs to the technical field of intelligent control. Under the condition that the gesture instruction does not conflict with the voice instruction, performing multi-dimensional normalization on the gesture feature vector to obtain a to-be-fused gesture vector; performing normalization processing on the voice feature vector based on context semantic perception to obtain a to-be-fused voice vector; fusing the to-be-fused gesture vector and the to-be-fused voice vector to obtain a fusion vector, and determining a fusion instruction corresponding to the fusion vector as a target execution instruction; and under the condition that the gesture instruction conflicts with the voice instruction, selecting a target execution instruction according to a first confidence coefficient corresponding to the gesture instruction and a second confidence coefficient corresponding to the voice instruction. The self-adaptive normalization strategy is designed for different modal data, the data fusion accuracy is improved, meanwhile, the gesture and voice instruction conflict can be intelligently and flexibly processed, and the accuracy of determining the real intention of the user is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of intelligent control technology, specifically relating to an intelligent control method, device, electronic device, and readable storage medium. Background Technology

[0002] With the widespread application of intelligent control technology in fields such as human-computer interaction, smart homes, and industrial automation, the requirements for system accuracy and natural interactivity are becoming increasingly stringent.

[0003] In intelligent control scenarios involving multimodal data fusion, visual information (such as the precise location and angle of gestures) and auditory information (such as the semantic features of voice commands) have drastically different data structures and distribution characteristics. Current methods for processing gesture or voice features using uniform normalization are insufficient for fine-grained adaptation to these differences. Furthermore, when faced with multimodal command conflicts, execution often relies on pre-set fixed priorities, lacking dynamic adjustment capabilities. Consequently, in complex and ever-changing real-world intelligent control scenarios, it is difficult to efficiently and accurately determine the user's true intent, failing to meet the user's actual needs for convenient and intelligent handling of command conflicts during interaction. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this application provides an intelligent control method, device, electronic device, and readable storage medium.

[0005] In a first aspect, embodiments of this application provide an intelligent control method, the method comprising: If the gesture command corresponding to the gesture action does not conflict with the voice command corresponding to the input voice, the gesture feature vector corresponding to the gesture action is subjected to multi-dimensional normalization to obtain the gesture vector to be fused. The speech feature vector corresponding to the input speech is normalized based on context semantic awareness to obtain the speech vector to be fused. The gesture vector to be fused and the speech vector to be fused are fused to obtain a fused vector, and the fused instruction corresponding to the fused vector is determined as the target execution instruction; In the event of a conflict between the gesture command and the voice command, a target execution command is selected from the gesture command and the voice command based on the first confidence level corresponding to the gesture command and the second confidence level corresponding to the voice command.

[0006] Secondly, embodiments of this application provide an intelligent control device, the device comprising: The first processing module is used to perform multi-dimensional normalization processing on the gesture feature vector corresponding to the gesture action, so as to obtain the gesture vector to be fused, when the gesture command corresponding to the gesture action does not conflict with the voice command corresponding to the input voice. The second processing module is used to perform context-semantic awareness-based normalization processing on the speech feature vector corresponding to the input speech to obtain the speech vector to be fused. The first determining module is used to fuse the gesture vector to be fused with the speech vector to be fused to obtain a fused vector, and to determine the fused instruction corresponding to the fused vector as the target execution instruction; The first selection module is used to select a target execution instruction from the gesture instruction and the voice instruction when the gesture instruction and the voice instruction conflict, based on a first confidence level corresponding to the gesture instruction and a second confidence level corresponding to the voice instruction.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0010] In this embodiment, when the gesture command corresponding to the gesture action and the voice command corresponding to the input speech do not conflict, the gesture feature vector corresponding to the gesture action is normalized in multiple dimensions to obtain the gesture vector to be fused; the voice feature vector corresponding to the input speech is normalized based on contextual semantic awareness to obtain the voice vector to be fused; the gesture vector to be fused and the voice vector to be fused are fused to obtain a fused vector, and the fused command corresponding to the fused vector is determined as the target execution command; when the gesture command and the voice command conflict, the target execution command is selected from the gesture command and the voice command based on the first confidence level corresponding to the gesture command and the second confidence level corresponding to the voice command. Thus, when the gesture command and the voice command do not conflict, a dedicated and adaptive normalization strategy is designed for the characteristics of different modal data, fully considering the differences between gesture and voice data in different dimensions and contexts, enabling the two modal data to better retain their respective features during fusion, avoiding fusion deviations due to data differences, improving data fusion accuracy, and thereby improving the accuracy of subsequent command execution. When gesture commands and voice commands conflict, by calculating the first confidence level of the gesture command and the second confidence level of the voice command, the target command is selected for execution. This enables more intelligent and flexible handling of gesture and voice command conflicts, improves the accuracy of determining the user's true intention, and provides users with a more convenient and efficient interactive experience in complex and ever-changing real-world scenarios, thus meeting the intelligent needs of users for handling command conflicts during natural interaction. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the steps of an intelligent control method provided in an embodiment of this application; Figure 2 This is a specific example provided in the embodiments of this application; Figure 3 This is a structural diagram of an intelligent control device provided in an embodiment of this application; Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0014] The intelligent control method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0015] Figure 1 This is a flowchart illustrating the steps of an intelligent control method provided in an embodiment of this application, as follows: Figure 1 As shown, the method may include: Step 101: If the gesture command corresponding to the gesture action does not conflict with the voice command corresponding to the input voice, perform multi-dimensional normalization processing on the gesture feature vector corresponding to the gesture action to obtain the gesture vector to be fused.

[0016] In this embodiment, during the operation of the multimodal interaction system, the system monitors hand gestures and input voice in real time. When both hand gestures and input voice are detected simultaneously, for hand gestures, feature extraction is first performed to obtain the corresponding hand gesture feature vector, and then the vector is converted or mapped into a corresponding hand gesture command using a preset hand gesture recognition algorithm. For input voice, speech recognition technology and natural language processing technology are used to obtain the corresponding speech feature vector, and then the speech feature vector is converted or mapped into a speech command.

[0017] After recognizing both gesture and voice commands, it's necessary to determine if they conflict. Specifically, this can be determined by whether the object indicated by the gesture command and the object indicated by the voice command are the same, and whether their functions contradict each other. For example, if the gesture command is "turn up the volume" and the voice command is "turn down the volume," these two commands are functionally contradictory and therefore conflict. However, if the gesture command is "change channel" and the voice command is "play music," these two commands are functionally independent and therefore not conflicting.

[0018] When gesture commands and voice commands do not conflict, the gesture feature vector can be pre-normalized. This includes normalizing the feature vectors for different dimensions and smoothing normalization based on historical statistics. A gesture feature vector is a digital description of a gesture, containing feature information across multiple dimensions, such as finger bending, palm orientation, and arm position. For example, when describing a hand movement in three-dimensional space, the gesture feature vector might include the hand's position on the X, Y, and Z axes, the force of the gesture, and the angle of the hand joints. These different dimensional feature information comprehensively and accurately depict the key elements of a gesture, such as its shape, trajectory, and posture.

[0019] Considering the significant differences in the feature distribution characteristics of gesture data across different dimensions—for example, the numerical ranges of hand position information and angle information differ considerably. Taking hand position information as an example, its position on the X-axis may fall within the range of 0 to 100, while the angle information of a certain joint in the hand may fall within the range of 0 to 360 degrees—in order to eliminate the impact of these numerical range differences on subsequent processing, independent means and variances are calculated for each dimension. Specifically, for each dimension in the gesture feature vector, all values ​​of the current batch of gesture data in that dimension are collected, and then the mean and variance of these values ​​are calculated using statistical methods. The calculated mean and variance are then used to normalize the data for that dimension, converting the data into a standard form with the same statistical properties, such as mapping the data to a specific interval, like [0,1] or [-1,1].

[0020] This dimensional normalization process makes data from different dimensions comparable on a numerical scale, avoiding feature distortion caused by cross-dimensional distribution interference and providing a unified data foundation for subsequent fusion processing. For example, when processing hand position and angle information, after dimensional normalization, both position and angle information can be represented within a similar numerical range, avoiding the problem of one dimension being over-amplified or under-amplified during the fusion process due to excessively large or small numerical ranges.

[0021] After dimensional normalization, to improve the stability and accuracy of normalization, smoothing can be performed by incorporating statistical information from historical data. During normalization, not only are statistics (such as mean and variance) calculated based on the current batch of data, but statistical information from historical data is also introduced. Specifically, a weighted average method can be used, assigning different weights to the statistics of the current batch and the statistics of historical data, and then summing them to obtain the final statistic for normalization. This approach fully utilizes historical data information, reducing fluctuations in statistics caused by factors such as small sample sizes or outliers in the current batch, thus making the normalization results more stable and reliable. In some gesture recognition applications, due to the relative stability of user gesture habits, historical data can reflect the general patterns of user gesture characteristics. Smoothing normalization by incorporating historical statistical information can better adapt to the gesture characteristics of different users, improving the accuracy and stability of gesture recognition.

[0022] For example, you can first calculate the mean of each dimension of the current batch of data. and variance Then retrieve the historical statistical information (historical mean) for that dimension. Historical variance The final normalized parameters are generated using a weighted smoothing formula, and normalization is then performed based on these parameters. For example:

[0023]

[0024] Here, β represents the historical data weight, which can be dynamically adjusted based on data stability. Correcting the current statistics by incorporating historical information can reduce the impact of noise in a single batch.

[0025] After dimensionality normalization and smoothing normalization incorporating historical statistics, the original gesture feature vector is transformed into a more standardized, stable, and comparable form, resulting in the gesture vector to be fused. This gesture vector retains the key feature information of the gesture action while eliminating numerical differences and statistical fluctuations between different dimensions, enabling effective fusion with feature vectors from other modalities. For example, when fusing with speech feature vectors, the gesture vector to be fused can be calculated and analyzed on the same numerical scale and statistical characteristics as other modal feature vectors, thereby more accurately achieving the complementarity and synergy of multimodal information and providing more comprehensive and accurate information support for system decision-making and control.

[0026] Step 102: Perform context-aware normalization on the speech feature vector corresponding to the input speech to obtain the speech vector to be fused.

[0027] In this embodiment of the application, in order to achieve effective fusion of speech modality and gesture modality data, it is necessary to normalize the speech feature vector corresponding to the input speech to obtain the speech vector to be fused.

[0028] For speech modalities, the characteristics of each semantic feature dimension in the speech feature vector are first analyzed. Considering the differences in the information meaning and numerical range represented by different semantic feature dimensions—for example, semantic feature dimensions related to time and space often have different numerical ranges and distribution patterns than other ordinary semantic feature dimensions—the system adopts a context-based normalization parameter adjustment strategy. This strategy can dynamically adjust the normalization parameters according to the specific meaning of the semantic feature and contextual information to adapt to the processing needs of different semantic features. For example, when processing semantic feature dimensions related to room location, the numerical range and distribution of corresponding semantic features will differ due to the different spatial distributions of different rooms (such as bedrooms and living rooms) and user habits. The context-based normalization parameter adjustment strategy allows for more accurate normalization of these features, making them comparable at the same scale.

[0029] After receiving a voice command input, the system uses a natural language processing module to analyze the semantics of the command in real time. The specific steps are as follows: 1) Word segmentation and key entity extraction: The text corresponding to the voice command is segmented using a word segmentation algorithm to extract key entities. For example, for the voice command "turn on the bedroom light," the key entities extracted after word segmentation are ["bedroom", "light"]. 2) Association with the scene database: The extracted key entities are associated with the scene database in the system. The scene database stores various scene-related information, such as the context information of the current room and device status information. For example, if the current room is a bedroom, then the key entity "bedroom" can be associated with the context information that the current room is a bedroom. 3) Parsing time / space dimensions: For key entities related to time and space, their corresponding time or space dimension information is parsed. For example, the entity "bedroom" can be associated with historical scene location data to determine its specific location in space; if the voice command contains time information, such as "turn on the light in the morning," then the time dimension is parsed as "morning." 4) Generating scene context vectors: Based on the parsing results, scene context vectors are generated. A scene context vector is a vector form that comprehensively represents instruction scene information, such as {room:“bedroom”, time:“evening”}, where “room” represents room information and “time” represents time information. This scene context vector will be used for subsequent normalization parameter adjustment, providing contextual basis for the normalization processing of different semantic features.

[0030] After obtaining the scene context vector, the system performs exponential smoothing on the time- and space-related dimensions of the semantic feature vector (such as the location information of "bedroom" in "bedroom light"), combined with historical batch statistics, to calculate the weighted average of the mean and variance of the current batch, generating dynamic normalization parameters. The specific calculation process is as follows: Assuming historical batch average Historical batch variance Current batch average Current batch variance The historical data weight α (e.g., α=0.7, indicating that historical data accounts for 70%, which can be dynamically adjusted based on the frequency of historical "bedroom lighting" commands) is as follows: New mean calculation:

[0031] For example, assuming historical data , If α = 0.7, then: .

[0032] New variance calculation:

[0033] For example, suppose , ,but: .

[0034] Using the calculated dynamic normalization parameter (new mean) New variance This involves normalizing the time- and space-related dimensions of the speech feature vector. Specifically, normalization can be achieved by subtracting the new mean from the data in each dimension and then dividing by the square root of the new variance. This maps the data to a standard form with the same statistical properties, typically transforming it to a standard normal distribution with a mean of 0 and a standard deviation of 1. This normalization process eliminates the numerical distribution differences between different semantic feature dimensions, making them comparable on the same scale, ultimately yielding the speech vector to be fused.

[0035] Step 103: Fuse the gesture vector to be fused with the speech vector to be fused to obtain a fused vector, and determine the fused instruction corresponding to the fused vector as the target execution instruction.

[0036] In this embodiment, after adaptively normalizing the gesture feature vector and the speech feature vector, the gesture vector to be fused and the speech vector to be fused are fused to obtain a fused vector, and the fusion instruction corresponding to the fused vector is determined as the target execution instruction. The fusion method can employ a Multi-Attention Based Fusion Network (MABN). By introducing a multi-attention mechanism, it can adaptively focus on important feature information in different modalities and perform weighted fusion based on the importance of this information, thereby effectively integrating the advantages of different modalities and improving the fusion effect. After normalizing the gesture vector and the speech vector to be fused, processing the numerical scale to a similar level, and unifying the data format, the gesture vector and the speech vector to be fused can be fused. This is achieved by automatically learning the importance of each feature dimension in different modalities and assigning different weights to different features based on their importance. The attention-weighted representations of the gesture vector and the speech vector are concatenated or added to obtain a preliminary fused feature representation. This preliminary fused feature representation is then subjected to nonlinear transformation, dimensionality reduction, or dimensionality increase to obtain the final fused vector.

[0037] After obtaining the fusion vector, the fusion instruction corresponding to the fusion vector can be identified, and the fusion instruction can be determined as the target execution instruction. For example, the similarity between the fusion vector and the instruction vector corresponding to the preset instruction can be calculated, and the preset instruction corresponding to the instruction vector with the highest similarity can be determined as the fusion instruction, and this fusion instruction can be determined as the target execution instruction.

[0038] Step 104: In the event of a conflict between the gesture command and the voice command, select the target execution command from the gesture command and the voice command based on the first confidence level corresponding to the gesture command and the second confidence level corresponding to the voice command.

[0039] In this embodiment, when gesture commands and voice commands conflict, a first confidence level corresponding to the gesture command and a second confidence level corresponding to the voice command are obtained. The first confidence level measures the reliability of the gesture command, and the second confidence level measures the reliability of the voice command. Based on the calculated first and second confidence levels, a target execution command is selected from the conflicting gesture and voice commands. If the first confidence level is significantly higher than the second confidence level (i.e., the first confidence level is greater than the second confidence level and the difference between the first and second confidence levels is greater than a preset threshold), the gesture command is considered more reliable, and the gesture command is selected as the target execution command. Conversely, if the second confidence level is significantly higher than the first confidence level (i.e., the first confidence level is less than the second confidence level and the difference between the first and second confidence levels is greater than a preset threshold), the voice command is selected as the target execution command. When the absolute difference between the first and second confidence levels is less than the preset threshold, making it difficult to directly determine which command to prioritize, a multi-round confirmation mechanism can be initiated. By sending confirmation information to the user and receiving user feedback, a target execution command is selected from the gesture and voice commands based on the feedback information.

[0040] In summary, in this embodiment, when the gesture command corresponding to the gesture action and the voice command corresponding to the input speech do not conflict, the gesture feature vector corresponding to the gesture action is normalized in multiple dimensions to obtain the gesture vector to be fused; the voice feature vector corresponding to the input speech is normalized based on contextual semantic awareness to obtain the voice vector to be fused; the gesture vector to be fused and the voice vector to be fused are fused to obtain a fused vector, and the fused command corresponding to the fused vector is determined as the target execution command; when the gesture command and the voice command conflict, the target execution command is selected from the gesture command and the voice command based on the first confidence level corresponding to the gesture command and the second confidence level corresponding to the voice command. Thus, when the gesture command and the voice command do not conflict, a dedicated and adaptive normalization strategy is designed for the characteristics of different modal data, fully considering the differences between gesture and voice data in different dimensions and contexts, enabling the two modal data to better retain their respective features during fusion, avoiding fusion deviations due to data differences, improving data fusion accuracy, and thereby improving the accuracy of subsequent command execution. When gesture commands and voice commands conflict, by calculating the first confidence level of the gesture command and the second confidence level of the voice command, the target command is selected for execution. This enables more intelligent and flexible handling of gesture and voice command conflicts, improves the accuracy of determining the user's true intention, and provides users with a more convenient and efficient interactive experience in complex and ever-changing real-world scenarios, thus meeting the intelligent needs of users for handling command conflicts during natural interaction.

[0041] Optionally, embodiments of this application may further include the following steps: Step 201: Based on the capsule network, perform multi-level capsule encoding and dynamic feature fusion processing on the gesture image corresponding to the gesture action to obtain the gesture feature vector.

[0042] In this embodiment, after obtaining the gesture image corresponding to the gesture action, a multi-level capsule encoding and dynamic feature fusion process based on capsule networks can be used to obtain the gesture feature vector. Multi-level capsule encoding and dynamic feature fusion refers to progressively extracting and fusing features from the gesture image through multiple capsule layers. Different capsule layers output feature information of different dimensions; for example, lower-level capsule layers may extract basic features such as edges and textures, while higher-level capsule layers extract more complex gesture posture features. Dynamic feature fusion organically combines the feature information of each capsule layer based on continuously adjusted weights during the iteration process. This processing method can more accurately capture the spatial information of the gesture, such as position, angle, and force, avoiding feature distortion caused by uniform normalization in traditional methods, and providing more accurate data support for subsequent gesture recognition and command execution.

[0043] Optionally, step 201 may include the following steps: Step 301: In any iteration, for any capsule layer in the capsule network, obtain the output vector of the capsule layer for the gesture image corresponding to the gesture action.

[0044] Step 302: Determine the consistency score corresponding to the output vector based on the similarity between the output vector and the expected vector corresponding to the capsule layer, the weight coefficient of the capsule layer in this iteration, and the historical consistency score; the historical consistency score is used for weighted correction; different capsule layers are used to output feature information corresponding to different dimensions.

[0045] In this embodiment, during any iteration, a consistency score is determined for the output vector of any capsule layer in the capsule network. Different capsule layers output feature information of different dimensions, such as the degree of finger bending, palm orientation, and arm position. First, the similarity between the capsule layer's output vector and the expected vector is calculated, typically using cosine similarity, which is the cosine of the angle between the two vectors. This value ranges from -1 to 1, with values ​​closer to 1 indicating greater similarity. Simultaneously, historical batch statistics, such as the average consistency of past gesture data, are used to weight and correct the current consistency. The average consistency of past gesture data can refer to the average consistency score of the current capsule layer for similar gestures during historical training / inference. The historical consistency score is used to weight and correct the currently calculated similarity; it stores the consistency of the capsule layer in past iterations and reflects a certain degree of stability and trend. For example, the formula for calculating the consistency score can be: Consistency score = (Current cosine similarity × weight coefficient) + (Historical average consistency × (1 - weight coefficient)) The weighting coefficients can be dynamically adjusted based on the feature confidence of the current frame.

[0046] Consistency score = [cos(output vector·expected vector)×α] + [historical average consistency × (1 / 2)] α)] Wherein, cos(·): cosine similarity, used to calculate the angle between the output vector and the vector expected to be output by the capsule; α: dynamic weight coefficient, which can be set based on the feature confidence of the current frame, ranging from 0.5 to 0.8. For example, if the current frame image is clear and the gesture is obvious, the feature confidence is high, and the weight coefficient can be set to 0.7.

[0047] For example, assuming the output vector = (0.8, 0.6) and the expected vector = (0.9, 0.4), then cos_sim = 0.88; historical average consistency = 0.85, α = 0.6, and therefore the consistency score corresponding to the output vector of this capsule layer = 0.88 × 0.6 + 0.85 × 0.4 = 0.868. Step 303: Based on the consistency score and routing factor of each capsule layer in this iteration, determine the weight coefficient of each capsule layer in the next iteration.

[0048] In this embodiment, the weight coefficients for each capsule layer in the next iteration are determined based on the consistency score and routing factor of each capsule layer in the current iteration. The routing factor can be adaptively set according to the dimensional characteristics of the gesture data. For example, for important dimensions such as position and angle, the routing factor is set to be larger, such as 1.5, to strengthen the weight of these spatial features.

[0049] An iterative softmax update mechanism is used to calculate weights. For example, the formula could be: Weight coefficient = exp(consistency score × routing factor) / Σ(exp(consistency score × routing factor)). In each iteration, based on the consistency score and routing factor of each capsule layer in the current iteration, the weight for the next round of each capsule layer is calculated according to the formula. For instance, if a capsule layer has a consistency score of 0.8 and a routing factor of 1.5, the weights of other capsule layers are calculated in the same way, summed, and then the weight coefficient for the next round of that capsule layer is obtained using the formula. This allows capsule layers with high consistency to obtain higher weights in subsequent iterations, achieving dynamic weight optimization.

[0050] Step 304: When the stopping condition is met, the feature information output by each capsule layer corresponding to the last iteration is determined as the gesture feature vector output by the capsule network.

[0051] In this embodiment, upon reaching the stopping condition, the feature information output by each capsule layer corresponding to the last iteration is determined as the gesture feature vector output by the capsule network. The stopping condition can be set as a weight coefficient change of <0.01 for three consecutive iterations to ensure weight stability and avoid infinite iteration. For example, after multiple iterations, the weight changes of each capsule layer are checked three times consecutively; if all changes are less than 0.01, a stable state is considered reached. Upon reaching the stopping condition, the feature information output by each capsule layer has fully integrated the features of the gesture image in different dimensions, such as finger bending, palm orientation, and arm position. Combining these features yields a gesture feature vector that comprehensively and accurately represents the gesture characteristics.

[0052] In this embodiment, by determining the consistency score by combining the similarity between the output vector and the expected vector, the weight coefficient, and the historical score during iteration, the reliability of the capsule layer output can be evaluated more accurately. At the same time, the weight of the next round is determined based on the consistency score and the routing factor, realizing the dynamic optimization of the weight coefficient of the capsule layer in different rounds of iteration. After reaching the stopping condition, the features of each layer are fused to obtain the gesture feature vector, which can fully retain the multi-dimensional information of the gesture and improve the accuracy and comprehensiveness of feature extraction.

[0053] Step 202: Perform semantic recognition and feature extraction on the input speech to obtain the speech feature vector.

[0054] In this embodiment, semantic recognition and feature extraction are performed on the input speech to obtain a speech feature vector. A speech recognition engine based on a deep learning model processes the input speech. Trained on a large amount of speech data, the speech recognition engine can quickly and accurately convert the acquired speech into text. During processing, the speech recognition engine performs real-time framing of the speech signal, dividing the continuous speech signal into short frames, analyzing the frequency, amplitude, and other features of each frame, and combining acoustic and language models to predict the most likely text content. For example, the acoustic model can be used to describe the correspondence between the acoustic characteristics of the speech signal and speech units (such as phonemes). Through learning from a large amount of speech data, it can predict possible sequences of speech units based on the acoustic features of the input speech. The language model considers the statistical laws of language, such as word collocation and sentence grammatical structure. It further filters and adjusts the speech unit sequences output by the acoustic model to predict the most likely text content, thereby quickly and accurately converting speech into text.

[0055] After speech is converted to text, pre-trained word vector models, such as BERT (Bidirectional Encoder Representations from Transformers), can be used to map each word in the text to a high-dimensional vector space. In this high-dimensional space, words with similar semantics are mapped to nearby positions, thus capturing the semantic relationships between words. Then, techniques such as syntactic analysis and semantic role labeling are used to further understand the semantic structure and intent of the voice commands. Syntactic analysis mainly analyzes the grammatical structure of the sentence, determining the relationships between its various components, such as subject-verb-object structures. Semantic role labeling further analyzes the semantic roles of each component in the sentence, such as the executor, receiver, location, and time of the action. For example, when a user says, "Please turn up the brightness of the bedroom light," through syntactic analysis and semantic role labeling, the system can accurately parse the subject of the command as "bedroom light" and the action as "turn up the brightness," clearly identifying the user's precise need.

[0056] After completing the deep semantic analysis of the text, the word vectors and semantic structure information of each word in the text are integrated and fused and encoded using specific algorithms and models to obtain speech feature vectors. These speech feature vectors are used to represent the semantic information of the input speech.

[0057] In this embodiment, the multi-level encoding and dynamic fusion of gesture images through capsule networks can accurately capture multi-dimensional features of gestures; and the speech processing can accurately identify semantics and extract features, thereby comprehensively acquiring gesture and speech information.

[0058] Optionally, embodiments of this application may further include the following steps: Step 401: Determine the gesture coordinates corresponding to the gesture action based on the position information in the gesture feature vector.

[0059] In this embodiment, positional information is extracted from the gesture feature vector output by the capsule network to determine the gesture coordinates corresponding to the gesture action. The positional information can be presented in the form of a spatial coordinate vector, such as 3D coordinates (x, y, z). The positional information can be the coordinates of key hand points in the gesture action (such as fingertip coordinates).

[0060] Since the location information is generated based on the coordinate system set within the capsule network, it can be mapped using camera calibration parameters to convert it into coordinates in actual space. Camera calibration parameters can be parameters used to convert image pixel coordinates into actual physical coordinates, such as the pixel-to-meter conversion factor. Assuming the capsule network outputs coordinates (120, 80, 50) and the pixel-to-meter conversion factor is 0.01m / pixel, the actual position can be calculated using the formula "Actual coordinates = Capsule output coordinates × Conversion factor (e.g., camera calibration value = 0.01m / pixel)". That is, the gesture coordinates corresponding to the gesture action are (1.2m, 0.8m, 0.5m).

[0061] Step 402: Based on the gesture coordinates and the object coordinates corresponding to each controlled object, determine the instruction object indicated by the gesture action.

[0062] In this embodiment, the gesture coordinates are compared with the object coordinates corresponding to each controlled object to determine the instruction object indicated by the gesture. The controlled object can refer to a device or element that can be controlled by gestures, such as air conditioners or lights in a smart home system. Each controlled object has its corresponding physical coordinates, which are preset during system installation and configuration.

[0063] Assuming the physical coordinates of the air conditioner are (1.0m, 0.8m, 0.6m), when the system receives the gesture coordinates as (1.2m, 0.8m, 0.5m), it will calculate the difference between the gesture coordinates and the air conditioner's physical coordinates. The coordinate difference is usually measured by calculating the distance between the two points, for example, using the Euclidean distance formula: "Coordinate difference = √[(x2-x1)]". 2 +(y2-y1) 2 +(z2-z1) 2 Substituting the hand gesture coordinates (1.2m, 0.8m, 0.5m) and the air conditioner's physical coordinates (1.0m, 0.8m, 0.6m) into the formula, we can obtain the coordinate difference as √[(0.2)²+(0)²+(0.1)²]=0.22m.

[0064] A pre-set object matching threshold is used. When the coordinate difference is less than the threshold, the object indicated by the gesture is considered the controlled object. For example, if the threshold is set to 0.3m, since 0.22m < 0.3m, the object indicated by the gesture is determined to be the air conditioner. Furthermore, the specific command, such as turning on, turning off, or adjusting the temperature, can be determined based on other characteristics of the gesture (e.g., the shape and direction of the gesture).

[0065] In this embodiment, the gesture coordinates output by the capsule network are compared with the physical coordinates of the controlled object. The object of the instruction indicated by the gesture is determined according to the relationship between the coordinate difference and a preset threshold. This does not depend on a specific gesture shape or action pattern, thus improving the versatility and flexibility of intelligent control.

[0066] Optionally, embodiments of this application may further include the following steps: Step 501: Obtain gesture feature vectors corresponding to at least two gesture images; the gesture feature vectors include position information, angle information, and force information.

[0067] In this embodiment of the application, gesture feature vectors corresponding to at least two gesture images output by the capsule network are obtained. The gesture feature vectors contain position information, angle information, and force information.

[0068] Step 502: Based on the position information, angle information and force information corresponding to the at least two gesture images, determine the gesture trajectory, gesture angle and gesture force corresponding to the gesture action.

[0069] In this embodiment, the system tracks key points of the hand, such as fingertip coordinates, in consecutive gesture image frames. For each frame, the coordinate positions of the key points are recorded, and then the coordinate differences of the key points between adjacent frames are calculated.

[0070] For example, suppose the fingertip coordinates in frame t are... The fingertip coordinates in frame t-1 are The coordinate difference is The coordinate differences of multiple consecutive frames are accumulated to form a trajectory vector. For example, if the fingertip coordinates in frame 1 are (100, 200) and in frame 2 are (105, 205), then Δx=5, Δy=5, and the accumulated value is the trajectory vector (5,5). If the fingertip coordinates in frame 3 are (110, 210), then the trajectory vector of frames 2 and 3 is (5,5). By concatenating the trajectory vectors of multiple consecutive frames, a complete sliding trajectory is generated. Based on the characteristics of the trajectory vectors in the sliding trajectory, the specific sliding trajectory can be identified. For example, [(5,5), (5,5)] can represent a horizontal straight line sliding trajectory.

[0071] Based on the different characteristics of the x and y axis components in the trajectory vector, different types of gesture trajectories can be identified. Specifically, when the y-axis component is dominant, for example, Δy is significant and Δx≈0 (e.g., (0,10), (0,12)), the gesture trajectory can be identified as a vertical swipe. When the x and y axis components are balanced, for example, Δx≈Δy (e.g., (5,5), (7,7)), the gesture trajectory can be identified as a diagonal swipe. When the vector angle rotates smoothly over time, for example, when the trajectory vector direction changes continuously, the gesture trajectory can be identified as a circle. When the vector length is less than a preset length threshold, for example, in a single frame or a few frames, the displacement is minimal, and the gesture trajectory can be identified as a click gesture.

[0072] The determination of gesture angles primarily relies on the angle information within the gesture feature vector. This angle information can be derived through gesture posture analysis, such as the hand's rotation angle and tilt angle. In some systems, the gesture angle can also be determined by calculating the direction angle of the trajectory vector.

[0073] The inference of gesture strength is based on the rate of change of inter-frame distance. First, the rate is calculated using the formula: Rate = √(Δx² + Δy²) / Δt, where Δt is the inter-frame time interval, which can be set according to the frame rate of the video stream. For example, for a 60fps video stream, Δt = 1 / 60s.

[0074] The force of a gesture can be determined by calculating the rate and comparing it with a preset force threshold. For example, if Δx=5 and Δy=5, then the distance = √(25+25) = 7.07 pixels, and the rate = 7.07 / (1 / 60) = 424.2 pixels / s. If the preset threshold for fast swipe is 200 pixels / s, then the gesture can be determined to be a fast swipe.

[0075] Step 503: Based on the gesture trajectories corresponding to the at least two gestures, the gesture angles, and the gesture strength, calculate the matching degree between the gesture action and each preset gesture corresponding to the instruction object indicated by the gesture action; the instruction object indicated by the gesture action is determined based on the position information in the gesture feature vector.

[0076] In this embodiment, the matching degree between the gesture action and each preset gesture corresponding to the instruction object is calculated based on the gesture trajectory, gesture angle, and gesture force of at least two gestures. The preset gestures are a series of gesture actions predefined by the system, and each preset gesture has its corresponding gesture trajectory, gesture angle, and gesture force feature description.

[0077] When calculating the matching degree, similarity calculation methods can be used. For example, cosine similarity can be used to calculate the matching degree between the current gesture trajectory vector and the predefined gesture template. The formula is: Similarity = (V_current·V_template) / (||V_current||2×||V_template||2), where V_current is the current gesture trajectory vector, V_template is the predefined gesture template vector, and ||V_current||2 and ||V_template||2 are their L2 norms, respectively. In addition to calculating the similarity of the trajectory vector, the matching of the gesture angle and gesture strength also needs to be considered. The differences between the gesture angle and gesture strength and the gesture angle and gesture strength of the gesture template corresponding to the preset gesture can also be quantified. For example, the absolute or relative values ​​of the angle difference and strength difference can be calculated. Then, the trajectory similarity, angle matching degree, and strength matching degree are combined according to their respective weights to obtain the final matching degree.

[0078] Step 504: Determine the operation instruction corresponding to the preset gesture with the highest matching degree and a matching degree greater than the preset matching degree threshold as the gesture instruction of the gesture action indication.

[0079] In this embodiment, after calculating the matching degree between the gesture action and each preset gesture, the preset gesture with the highest matching degree is determined, and then it is checked whether the matching degree is greater than a preset matching degree threshold. The preset matching degree threshold is a standard set by the system based on actual application scenarios and requirements, used to determine whether a gesture action effectively matches a preset gesture. If the matching degree is the highest and greater than the preset matching degree threshold, the operation instruction corresponding to the preset gesture is determined as the gesture instruction indicated by the gesture action; if the matching degree of all preset gestures is less than the preset matching degree threshold, the gesture action is considered invalid, and no operation instruction is executed.

[0080] In this embodiment, gesture images are collected, gesture feature vectors are extracted, and information such as gesture trajectory, angle, and force are parsed. Then, they are matched with preset gestures to finally determine the gesture command corresponding to the gesture action. By analyzing from multiple dimensions such as position, angle, and force, the various feature information of the gesture can be fully utilized to improve the accuracy of matching degree calculation, thereby improving the accuracy and reliability of gesture recognition.

[0081] Optionally, embodiments of this application may further include the following steps: Step 601: Determine the first confidence level corresponding to the gesture command based on the output stability of each capsule layer, the reliability of dynamic routing, and the consistency of gesture actions.

[0082] In this embodiment of the application, the confidence level of a gesture command can be calculated by comprehensively considering the output stability of each capsule layer in the capsule network, the reliability of dynamic routing, and the consistency of gesture actions to determine the first confidence level corresponding to the gesture command.

[0083] The output stability of a capsule layer can be determined by the degree of change in the capsule output vector across consecutive frames. The output vectors of each capsule layer are obtained. The magnitude variance and directional angle variation of these output vectors are calculated. The magnitude variance reflects the degree of fluctuation in the length of the output vector; if the variance is less than a set threshold (e.g., 0.05), it indicates that the length of the capsule layer's output vector is relatively stable. The directional angle variation characterizes the change in the direction of the output vector; when the angle variation is less than 5°, it indicates that the vector direction is relatively stable. Only when both the magnitude variance and the directional angle variation meet the corresponding conditions is the capsule layer's output considered stable.

[0084] Dynamic routing is a crucial mechanism in capsule networks for determining the connection weights between capsule layers. Its reliability can be evaluated based on two aspects: First, the convergence of the routing iterations. The system sets convergence conditions for routing iterations; for example, if the weight convergence condition is met within 3-5 iterations, and the weight change is less than 0.01 for three consecutive iterations, the routing iteration is considered to have converged well. Second, the overall level of consistency scores during the iteration process. A historical gesture database can be maintained to store the feature vectors of gesture templates and corresponding historical consistency statistics. During the dynamic routing iteration process, the consistency score for each round is calculated. If the average consistency score is higher than 0.8, the routing result is considered reliable.

[0085] The consistency of gesture actions mainly includes two aspects: first, the matching degree between the current gesture trajectory and posture and the predefined gesture template; and second, the coherence of gesture features between consecutive frames, such as no abrupt changes in trajectory direction and posture parameters. After the current frame is input, the capsule output vector is extracted first, and its Euclidean distance with the historical average vector is calculated. The formula is d = ||current capsule output vector|| The historical average vector is ||2, where d is the Euclidean distance. Simultaneously, current frame features are considered, such as hand position stability; if the coordinate change is less than 5 pixels for three consecutive frames, the stability is high.

[0086] Then, a score is generated based on the consistency scoring formula: .

[0087] in, Used to represent the standard deviation of historical gesture data, such as the finger bending dimension. , Used to represent the stability weight of the current frame; when the coordinate change is less than 5 pixels for 3 consecutive frames. The value can be 0.9, and the stability weight can be determined by the confidence level of the current frame features.

[0088] Based on the output stability of the capsule layer, the reliability of dynamic routing, and the consistency of gesture actions, the scores of the three factors are weighted and summed according to the preset weight allocation rules to obtain the first confidence level corresponding to the gesture command.

[0089] Step 602: Determine the second confidence level corresponding to the voice command based on the accuracy of speech recognition, the rationality of semantic analysis, and the matching score of the language model.

[0090] In this embodiment, the speech recognition engine can calculate the matching probability between the acoustic features of the recognized text and the speech signal, and combine it with the posterior probability of the acoustic model to generate a confidence score in the range of 0-1. This confidence score is then determined as the speech recognition accuracy. For example, when the matching degree between the acoustic features of the recognized text and the speech signal reaches 95%, the speech recognition accuracy is 0.95.

[0091] The validity of semantic analysis can be determined using pre-trained language models (such as BERT) and syntactic analysis tools. First, the syntactic analysis tool parses the syntactic structure of the speech command, checking if it has a complete "action + object" syntactic structure. For example, for the command "turn on the light," "turn on" is the action, and "light" is the object, resulting in a complete syntactic structure. Then, the similarity between the semantic vector and common command pattern vectors is calculated. If the structure is complete and the similarity is above a threshold (e.g., 0.9), it is considered valid, and a corresponding validity score is output.

[0092] The matching score of a language model can be calculated using N-gram or Transformer-type language models. These models can be trained on large-scale text corpora and learn the rules and patterns of natural language. When input text is recognized, the model outputs the probability (or log probability) of that text appearing in the corpus; a higher probability indicates that the text conforms more closely to natural language usage. This probability, after normalization, becomes the matching score of the language model.

[0093] The accuracy of speech recognition, the rationality of semantic analysis, and the matching score of the language model are combined. Based on the preset weighting rules, the scores of these three factors are weighted and summed to obtain the second confidence level corresponding to the speech command.

[0094] In this embodiment of the application, a multi-dimensional and comprehensive evaluation method can more accurately determine the confidence level of gesture commands and voice commands, providing a more reliable basis for command recognition and execution for multimodal interaction systems, and improving the stability and accuracy of the system.

[0095] Optionally, step 104 may include the following steps: Step 701: If the first confidence level is greater than the second confidence level and the difference between the first confidence level and the second confidence level is greater than a preset difference threshold, the gesture instruction is determined as the target execution instruction.

[0096] In this embodiment, when a first confidence level is greater than a second confidence level, and the difference between the first and second confidence levels is greater than a preset difference threshold, this indicates that the gesture command is significantly more reliable than the voice command. Therefore, the gesture command is determined to be more reliable and is thus identified as the target execution command. The preset difference threshold can be determined based on the needs of the actual application scenario, providing a reasonable boundary for judging the difference in reliability between gesture commands and voice commands.

[0097] Step 702: If the first confidence level is less than the second confidence level and the difference between the first confidence level and the second confidence level is greater than the preset difference threshold, the voice command is determined as the target execution command.

[0098] In this embodiment of the application, when the first confidence level is less than the second confidence level, and the difference between the first confidence level and the second confidence level is greater than a preset difference threshold, it means that the voice command is significantly more reliable than the gesture command. Therefore, the voice command is considered more reliable, and the voice command is determined as the target execution command.

[0099] In this embodiment, when the difference between the first confidence level and the second confidence level is greater than a preset difference threshold, the instruction with higher confidence level is selected as the target execution instruction. This can effectively avoid erroneous operations caused by misidentification and improve the accuracy and stability of the system.

[0100] Optionally, step 104 may also include the following steps: Step 801: If the absolute difference between the first confidence level and the second confidence level is less than or equal to the preset difference threshold, generate a first confirmation request.

[0101] In this embodiment, when the absolute difference between the first confidence level and the second confidence level is less than or equal to a preset difference threshold, it indicates that the confidence levels of the two instructions are relatively close, and the system has difficulty directly determining which instruction to execute first. To further determine the actual needs, a first confirmation request can be generated. The first confirmation request is used to request the user to determine whether to execute a gesture instruction or a voice instruction.

[0102] Step 802: Determine the instruction indicated by the first feedback information corresponding to the first confirmation request as the target execution instruction.

[0103] In this embodiment, after generating the first confirmation request, the system will provide the user with the corresponding first feedback information in various ways. For example, the system can provide feedback to the user via voice regarding detected conflicts, such as, "Some conflicts have been detected. Would you like to turn on the lights or operate the air conditioner?" Simultaneously, the system can visually display two possible command options on the smart device's screen, such as displaying buttons for "Turn on the lights" and "Operate the air conditioner," allowing the user to confirm via gesture or voice selection. After receiving the first confirmation request, the user can provide corresponding feedback via gesture or voice. Upon receiving this first feedback information, the command indicated by the first feedback information can be determined as the target execution command. For example, if the user hears the voice confirmation request and says "Operate the air conditioner," then the command to operate the air conditioner will be determined as the target execution command and executed.

[0104] Step 803: If the first feedback information is not received within a preset time, a second confirmation request is generated, and the instruction indicated by the second feedback information corresponding to the second confirmation request is determined as the target execution instruction.

[0105] In this embodiment, if the user's first feedback is not received within a preset time, it indicates that the user may not have noticed the system's feedback in a timely manner or has not given a clear response. In this case, the scope of confirmation can be further narrowed down, generating a second confirmation voice message. The content of the second confirmation request can differ from the content of the first confirmation request, and the question content of the second confirmation request is more explicit than that of the first confirmation request. For example, the second confirmation request could be: "You just wanted to operate an electrical appliance, right? If it's a light-related operation, please say 'lights'; if it's an air conditioner-related operation, please say 'air conditioner'." The user can provide second feedback information in response to the second confirmation request, and the instruction indicated by the second feedback information is determined as the target execution instruction. For example, if the user says "air conditioner," then the instruction to operate the air conditioner is determined as the target execution instruction and executed.

[0106] In this embodiment, by generating a first confirmation request and a second confirmation request, users can confirm according to their actual needs. Furthermore, if the user does not respond to the first confirmation request in a timely manner, a second confirmation request is generated to narrow down the scope of confirmation. This step-by-step guidance and refinement of the confirmation process effectively mitigates the uncertainty caused by command conflicts, ensuring that the system accurately executes the user's true intentions.

[0107] For example, such as Figure 2 As shown, this application provides a specific example, such as... Figure 2As shown, when both gesture actions and input speech are simultaneously acquired, a capsule network is used to extract the gesture feature vector corresponding to the gesture action, and to extract the speech feature vector from the input speech. The gesture command corresponding to the gesture action is determined based on the gesture feature vector, and the speech command corresponding to the input speech is determined based on the speech feature vector. In the case of a conflict between the gesture command and the speech command, a first confidence level for the gesture command and a second confidence level for the speech command are obtained. When the difference between the first and second confidence levels is greater than a preset difference threshold, the command with the higher confidence level is determined as the target execution command. When the absolute difference between the first and second confidence levels is less than or equal to the preset difference threshold, a multi-round confirmation mechanism is executed, guiding the user to confirm through speech and visual information, and confirming the target execution command based on user feedback. When the gesture command and the speech command do not conflict, the gesture feature vector undergoes multi-dimensional normalization, and the speech feature vector undergoes context-aware normalization. The normalized gesture vector and the speech vector to be fused are dynamically fused using MABN to obtain a fused vector. The fused command corresponding to the fused vector is determined as the target execution command.

[0108] It should be noted that the intelligent control method provided in this application embodiment can be executed by an intelligent control device, or a control module within the intelligent control device for executing the method of loading intelligent control. This application embodiment uses the execution of the method of loading intelligent control by an intelligent control device as an example to illustrate the intelligent control device provided in this application embodiment.

[0109] Figure 3 This is a structural diagram of an intelligent control device provided in an embodiment of this application, with reference to... Figure 3 The device may include: The first processing module 901 is used to perform multi-dimensional normalization processing on the gesture feature vector corresponding to the gesture action, so as to obtain the gesture vector to be fused, when the gesture command corresponding to the gesture action does not conflict with the voice command corresponding to the input voice. The second processing module 902 is used to perform context-semantic awareness-based normalization processing on the speech feature vector corresponding to the input speech to obtain the speech vector to be fused. The first determining module 903 is used to fuse the gesture vector to be fused with the speech vector to be fused to obtain a fused vector, and to determine the fused instruction corresponding to the fused vector as the target execution instruction; The first selection module 904 is used to select a target execution instruction from the gesture instruction and the voice instruction when the gesture instruction and the voice instruction conflict, based on the first confidence level corresponding to the gesture instruction and the second confidence level corresponding to the voice instruction.

[0110] This application provides an intelligent control device that, when the gesture command corresponding to the gesture action and the voice command corresponding to the input speech do not conflict, performs multi-dimensional normalization on the gesture feature vector corresponding to the gesture action to obtain a gesture vector to be fused; performs context-aware normalization on the voice feature vector corresponding to the input speech to obtain a voice vector to be fused; fuses the gesture vector and the voice vector to be fused to obtain a fused vector, and determines the fused command corresponding to the fused vector as the target execution command; when the gesture command and the voice command conflict, the target execution command is selected from the gesture command and the voice command based on a first confidence level corresponding to the gesture command and a second confidence level corresponding to the voice command. In this way, when the gesture command and the voice command do not conflict, a dedicated and adaptive normalization strategy is designed for the characteristics of different modal data, fully considering the differences between gesture and voice data in different dimensions and contexts, so that the two modal data better retain their respective features during fusion, avoiding fusion deviations caused by data differences, improving the accuracy of data fusion, and thus improving the accuracy of subsequent command execution. When gesture commands and voice commands conflict, by calculating the first confidence level of the gesture command and the second confidence level of the voice command, the target command is selected for execution. This enables more intelligent and flexible handling of gesture and voice command conflicts, improves the accuracy of determining the user's true intention, and provides users with a more convenient and efficient interactive experience in complex and ever-changing real-world scenarios, thus meeting the intelligent needs of users for handling command conflicts during natural interaction.

[0111] Optionally, the device further includes: The first extraction module is used to perform multi-level capsule encoding and dynamic feature fusion processing on the gesture image corresponding to the gesture action based on a capsule network to obtain the gesture feature vector. The second extraction module is used to perform semantic recognition and feature extraction on the input speech to obtain the speech feature vector.

[0112] Optionally, the first extraction module includes: The first acquisition module is used to acquire, in any iteration, the output vector of the capsule layer in the capsule network for the gesture image corresponding to the gesture action; The second determining module is used to determine the consistency score corresponding to the output vector based on the similarity between the output vector and the expected vector corresponding to the capsule layer, the weight coefficient of the capsule layer in this iteration, and the historical consistency score; the historical consistency score is used for weighted correction; different capsule layers are used to output feature information corresponding to different dimensions; The third determination module is used to determine the weight coefficients of each capsule layer in the next iteration based on the consistency score and routing factor of each capsule layer in the current iteration. The fourth determining module is used to determine the feature information output by each capsule layer corresponding to the last iteration as the gesture feature vector output by the capsule network when the stopping condition is met.

[0113] Optionally, the device further includes: The fifth determining module is used to determine the gesture coordinates corresponding to the gesture action based on the position information in the gesture feature vector; The sixth determining module is used to determine the instruction object indicated by the gesture action based on the gesture coordinates and the object coordinates corresponding to each controlled object.

[0114] Optionally, the device further includes: The second acquisition module is used to acquire gesture feature vectors corresponding to at least two gesture images; the gesture feature vectors include position information, angle information, and force information. The seventh determining module is used to determine the gesture trajectory, gesture angle, and gesture force corresponding to the gesture action based on the position information, angle information, and force information corresponding to the at least two gesture images. The first calculation module is used to calculate the matching degree between the gesture action and each preset gesture corresponding to the instruction object indicated by the gesture action, based on the gesture trajectory corresponding to the at least two gestures, the gesture angle, and the gesture force; the instruction object indicated by the gesture action is determined based on the position information in the gesture feature vector. The eighth determining module is used to determine the operation instruction corresponding to the preset gesture with the highest matching degree and a matching degree greater than the preset matching degree threshold as the gesture instruction of the gesture action indication.

[0115] Optionally, the device further includes: The ninth determination module is used to determine the first confidence level corresponding to the gesture command based on the output stability of each capsule layer, the reliability of dynamic routing, and the consistency of gesture actions. The tenth determination module is used to determine the second confidence level corresponding to the voice command based on the accuracy of speech recognition, the rationality of semantic analysis, and the matching score of the language model.

[0116] Optionally, the first selection module 904 includes: The first determining submodule is used to determine the gesture instruction as the target execution instruction when the first confidence level is greater than the second confidence level and the difference between the first confidence level and the second confidence level is greater than a preset difference threshold. The second determining submodule is used to determine the voice instruction as the target execution instruction when the first confidence level is less than the second confidence level and the difference between the first confidence level and the second confidence level is greater than the preset difference threshold.

[0117] Optionally, the first selection module 904 further includes: The first generation module is used to generate a first confirmation request when the absolute difference between the first confidence level and the second confidence level is less than or equal to the preset difference threshold. The third determining submodule is used to determine the instruction indicated by the first feedback information corresponding to the first confirmation request as the target execution instruction; The fourth determining submodule is used to generate a second confirmation request if the first feedback information is not received within a preset time, and to determine the instruction indicated by the second feedback information corresponding to the second confirmation request as the target execution instruction.

[0118] The intelligent control device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0119] The intelligent control device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0120] Optional, such as Figure 4 As shown, this application embodiment also provides an electronic device, including a processor 1010, a memory 1009, and a program or instructions stored in the memory 1009 and executable on the processor 1010. When the program or instructions are executed by the processor 1010, they implement the various processes of the above-described intelligent control method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0121] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0122] Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0123] The electronic device 1000 includes, but is not limited to, components such as: radio frequency unit 1001, network module 1002, audio output unit 1003, input unit 1004, sensor 1005, display unit 1006, user input unit 1007, interface unit 1008, memory 1009, and processor 1010.

[0124] Those skilled in the art will understand that the electronic device 1000 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1010 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0125] It should be understood that, in this embodiment, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042. The GPU 10041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1006 may include a display panel 10061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 1007 includes at least one of a touch panel 10071 and other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.

[0126] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0127] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor 1010.

[0128] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described intelligent control method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0129] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0130] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described intelligent control method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0131] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0132] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the intelligent control method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0133] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0135] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An intelligent control method, characterized in that, The method includes: If the gesture command corresponding to the gesture action does not conflict with the voice command corresponding to the input voice, the gesture feature vector corresponding to the gesture action is subjected to multi-dimensional normalization to obtain the gesture vector to be fused. The speech feature vector corresponding to the input speech is normalized based on context semantic awareness to obtain the speech vector to be fused. The gesture vector to be fused and the speech vector to be fused are fused to obtain a fused vector, and the fused instruction corresponding to the fused vector is determined as the target execution instruction; In the event of a conflict between the gesture command and the voice command, a target execution command is selected from the gesture command and the voice command based on the first confidence level corresponding to the gesture command and the second confidence level corresponding to the voice command.

2. The method according to claim 1, characterized in that, The method further includes: Based on capsule networks, the gesture image corresponding to the gesture action is subjected to multi-level capsule encoding and dynamic feature fusion processing to obtain the gesture feature vector. The input speech is subjected to semantic recognition and feature extraction to obtain the speech feature vector.

3. The method according to claim 2, characterized in that, The capsule network-based method involves performing multi-level capsule encoding and dynamic feature fusion processing on the gesture image corresponding to the gesture action to obtain the gesture feature vector, including: In any iteration, for any capsule layer in the capsule network, obtain the output vector of the capsule layer for the gesture image corresponding to the gesture action; Based on the similarity between the output vector and the expected vector corresponding to the capsule layer, the weight coefficient of the capsule layer in this iteration, and the historical consistency score, the consistency score corresponding to the output vector is determined; the historical consistency score is used for weighted correction; different capsule layers are used to output feature information corresponding to different dimensions; Based on the consistency score and routing factor of each capsule layer in this iteration, determine the weight coefficient of each capsule layer in the next iteration. When the stopping condition is met, the feature information output by each capsule layer corresponding to the last iteration is determined as the gesture feature vector output by the capsule network.

4. The method according to claim 1, characterized in that, The method further includes: Obtain gesture feature vectors corresponding to at least two gesture images; the gesture feature vectors include position information, angle information, and force information; Based on the position information, angle information and force information corresponding to the at least two gesture images, determine the gesture trajectory, gesture angle and gesture force corresponding to the gesture action; Based on the gesture trajectories corresponding to the at least two gestures, the gesture angles, and the gesture strength, the matching degree between the gesture action and each preset gesture corresponding to the instruction object indicated by the gesture action is calculated; the instruction object indicated by the gesture action is determined based on the position information in the gesture feature vector. The operation instruction corresponding to the preset gesture with the highest matching degree and a matching degree greater than the preset matching degree threshold is determined as the gesture instruction of the gesture action.

5. The method according to claim 4, characterized in that, The method further includes: Based on the position information in the gesture feature vector, determine the gesture coordinates corresponding to the gesture action; Based on the gesture coordinates and the object coordinates corresponding to each controlled object, the instruction object indicated by the gesture action is determined.

6. The method according to claim 1, characterized in that, The method further includes: The first confidence level corresponding to the gesture command is determined based on the output stability of each capsule layer, the reliability of dynamic routing, and the consistency of gesture actions. The second confidence level corresponding to the voice command is determined based on the accuracy of speech recognition, the rationality of semantic analysis, and the matching score of the language model.

7. The method according to claim 1, characterized in that, The step of selecting a target execution instruction from the gesture instruction and the voice instruction based on the first confidence level corresponding to the gesture instruction and the second confidence level corresponding to the voice instruction includes: If the first confidence level is greater than the second confidence level and the difference between the first confidence level and the second confidence level is greater than a preset difference threshold, the gesture instruction is determined as the target execution instruction. If the first confidence level is less than the second confidence level and the difference between the first confidence level and the second confidence level is greater than the preset difference threshold, the voice instruction is determined as the target execution instruction.

8. The method according to claim 7, characterized in that, The step of selecting a target execution instruction from the gesture instruction and the voice instruction based on the first confidence level corresponding to the gesture instruction and the second confidence level corresponding to the voice instruction further includes: If the absolute difference between the first confidence level and the second confidence level is less than or equal to the preset difference threshold, a first confirmation request is generated; The instruction indicated by the first feedback information corresponding to the first confirmation request is determined as the target execution instruction; If the first feedback information is not received within a preset time, a second confirmation request is generated, and the instruction indicated by the second feedback information corresponding to the second confirmation request is determined as the target execution instruction.

9. An intelligent control device, characterized in that, The device includes: The first processing module is used to perform multi-dimensional normalization processing on the gesture feature vector corresponding to the gesture action, so as to obtain the gesture vector to be fused, when the gesture command corresponding to the gesture action does not conflict with the voice command corresponding to the input voice. The second processing module is used to perform context-semantic awareness-based normalization processing on the speech feature vector corresponding to the input speech to obtain the speech vector to be fused. The first determining module is used to fuse the gesture vector to be fused with the speech vector to be fused to obtain a fused vector, and to determine the fused instruction corresponding to the fused vector as the target execution instruction; The first selection module is used to select a target execution instruction from the gesture instruction and the voice instruction when the gesture instruction and the voice instruction conflict, based on a first confidence level corresponding to the gesture instruction and a second confidence level corresponding to the voice instruction.

10. The apparatus according to claim 9, characterized in that, The device further includes: The first extraction module is used to perform multi-level capsule encoding and dynamic feature fusion processing on the gesture image corresponding to the gesture action based on a capsule network to obtain the gesture feature vector. The second extraction module is used to perform semantic recognition and feature extraction on the input speech to obtain the speech feature vector.

11. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the intelligent control method as described in any one of claims 1-8.

12. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the intelligent control method as described in any one of claims 1-8.

Citation Information

Cited By

  • Air load prediction method, device and equipment for end of air compression station and storage medium

    CN122173846A

  • Intelligent device control method, electronic device, readable storage medium and computer program product

    CN122346004A