Intelligent power grid control system and method based on gesture interaction

Abstract

The invention relates to the technical field of intelligent power grid control, and discloses an intelligent power grid control system and method based on gesture interaction, and the system comprises a multi-modal data collection and preprocessing module which is used for collecting gesture data of a user, including images, radar and inertial measurement unit data, the gesture data acquisition module is used for acquiring gesture data of a user, filtering, denoising and normalizing the acquired gesture data of the user, and transmitting the processed data to the multi-modal data synchronization and alignment module; and the multi-modal data synchronization and alignment module refers to the output data of the multi-modal data acquisition and preprocessing module, and performs time and space alignment on the multi-modal data through a timestamp synchronization and space calibration technology. According to the method, features of the image, the radar and the inertial measurement unit are subjected to weighted fusion to realize efficient integration of multi-modal information, so that the gesture recognition precision of the system in a complex and variable environment is remarkably improved, and the robustness and the recognition stability of the system are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of smart grid control technology, and in particular to a smart grid control system and method based on gesture interaction. Background Art

[0002] In the development of smart grids, achieving efficient, safe and stable grid control is one of the core issues. Traditional grid control systems rely on centralized control and manual intervention, which is inefficient in dealing with the diversity of modern power demand, load fluctuations and sudden failures, and there are delays and safety risks. Smart grids introduce automation and intelligent technologies to achieve efficient management while improving system security. However, traditional smart grid control still relies on fixed interface operation, which has the limitations of complex operation, long response time and lack of flexibility, especially in scenarios with fast response and high safety requirements.

[0003] The development of artificial intelligence and multimodal sensing technology has made it possible to apply gesture recognition technology in smart grid control. Gesture recognition captures and analyzes the user's hand movements, allowing users to directly and efficiently control the grid through natural gestures. Compared with traditional interface operation methods, gesture interaction has the advantages of being intuitive, flexible and fast in response. However, due to the complexity and dynamic changes of the grid environment, gesture recognition faces many challenges in its application: Smart grid control usually involves a variety of environmental conditions, such as dim light, complex background, equipment interference, etc. Single-mode gesture recognition has low recognition accuracy and stability in this situation, and is prone to misrecognition and missed recognition. Different users have different gesture characteristics and usage habits. Therefore, the general gesture recognition model cannot meet the personalized recognition needs. The lack of adaptability of the gesture recognition system will reduce the recognition accuracy in the multi-user and multi-habit operation environment, resulting in poor user experience. Grid control has the characteristics of high frequency and high timeliness. Especially in operations such as load adjustment and equipment switching, the system needs to complete recognition and response in a very short time. Therefore, the real-time requirements for gesture recognition and control command generation are high. The traditional cloud computing model cannot meet the low-latency requirements. The long system response time will cause operation delays and reduce the accuracy and safety of grid control. There are many high-risk critical operations in power grid control. Once misidentification or misoperation occurs, it will cause serious damage to power equipment or safety accidents.

[0004] Therefore, those skilled in the art provide a smart grid control system and method based on gesture interaction to solve the above-mentioned problems. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a smart grid control system and method based on gesture interaction to solve the problems raised in the above background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a smart grid control system and method based on gesture interaction, comprising: The multimodal data acquisition and preprocessing module is used to collect the user's gesture data, including image, radar and inertial measurement unit data, and filter, denoise and normalize the collected user's gesture data. The processed data is transmitted to the multimodal data synchronization and alignment module; The multimodal data synchronization and alignment module refers to the output data of the multimodal data acquisition and preprocessing module, aligns the multimodal data in time and space through timestamp synchronization and spatial calibration technology, and transmits the output data to the feature extraction and enhancement module; The feature extraction and enhancement module uses the alignment data from the multimodal data synchronization and alignment module, extracts gesture features from images, radars, and inertial measurement units using deep learning models and signal processing methods, and improves the quality of features through denoising and enhancement algorithms. The extracted features are then passed to the multimodal feature fusion module. The multimodal feature fusion module refers to the gesture feature data of the feature extraction and enhancement module, integrates the image, radar and inertial measurement unit features through a weighted fusion algorithm, dynamically adjusts the weights of each data source, generates a unified gesture feature expression, and passes the fused features to the adaptive learning and dynamic adjustment module; The adaptive learning and dynamic adjustment module refers to the fusion features of the multimodal feature fusion module, and uses the adaptive learning algorithm to dynamically adjust the weight of feature fusion according to environmental changes and user characteristics. The adjusted features are transmitted to the edge computing and instruction generation module; The edge computing and command generation module uses the optimization features of the adaptive learning and dynamic adjustment module to quickly process gesture features based on the edge computing framework, generates power grid control commands through classification models or decision trees, and transmits the commands to the multi-channel user feedback module; The multi-channel user feedback module refers to the control command data of the edge computing and command generation module, and provides users with real-time feedback on gesture recognition status and control command execution through visual, sound or vibration feedback.

[0007] Preferably, the multimodal data acquisition and preprocessing module includes: An image acquisition unit is used to capture gesture images using an RGB camera to obtain two-dimensional spatial information of the user's hand. The image resolution is set to W*H, where W represents the width of the image and H represents the height of the image. A radar acquisition unit, used to acquire the motion depth and speed information of the gesture through a frequency modulated continuous wave radar; Inertial measurement unit, used to collect the three-axis acceleration of the hand and angular velocity , to obtain the dynamic information of the gesture; is the acceleration component in the x direction, is the acceleration component in the y direction, is the acceleration component in the z direction, is the angular velocity component in the x direction, is the angular velocity component in the y direction, is the angular velocity component in the z direction.

[0008] Preferably, the feature extraction and enhancement module includes: The convolutional neural network used for image feature extraction performs convolution and pooling on the image data I to obtain the image feature vector : , in, is the weight parameter of the convolutional neural network model, is the image feature vector, is the feature extraction function of the convolutional neural network model, is the input image data; Fast Fourier transform processing unit for radar feature extraction, transforming radar signal Convert to frequency domain signal , its frequency domain eigenvector is: , in, represents the feature vector extracted from radar data, represents the amplitude spectrum of the radar signal in the frequency domain, represents the maximum value in the frequency domain amplitude spectrum of the radar signal, represents the standard deviation of the radar signal frequency domain amplitude spectrum, Indicates the frequency position where the frequency domain amplitude spectrum of the radar signal reaches its maximum value; Long short-term memory network for inertial measurement unit feature extraction, processing time series data of inertial measurement unit , generate IMU feature vector : , in, are the parameters of the long short-term memory network model, is the feature vector, It is the feature extraction function of the long short-term memory network model.

[0009] Preferably, the multimodal feature fusion module uses a weighted fusion method to fusion the image feature vector , radar feature vector and the inertial measurement unit eigenvector Fusion is performed to generate a fusion feature vector : , in, , , is the weight of image, radar and feature, satisfying .

[0010] Preferably, the adaptive learning and dynamic adjustment module automatically adjusts the weight of the fusion feature according to environmental changes, based on the light intensity. and background complexity Calculate weight parameters , and : , in, Indicates the ambient light intensity. Represents the background complexity.

[0011] Preferably, the adaptive learning and dynamic adjustment module dynamically optimizes the gesture recognition model through an online learning method and updates the model parameters using a gradient descent method. , the update formula is: , in, is the learning rate, is the cross entropy loss, is the predicted output of the model, Indicated in The model parameters after iterations are represents the model parameters at the tth iteration, represents the true label or target value, Indicates that the gradient is with respect to the parameter The set of partial derivatives of .

[0012] Preferably, the edge computing and instruction generation module is used to quickly process multimodal fusion features And generate control instructions, use lightweight neural network to improve computing efficiency, use knowledge distillation to optimize the lightweight model, and its distillation loss function is: , in, is the cross entropy loss of hard labels, is the divergence loss of soft labels, is the weight coefficient of the cross entropy loss, Represents distillation loss.

[0013] Preferably, the multi-channel user feedback module provides the user with confirmation information of gesture recognition status and control instructions through visual, audio and vibration feedback, wherein: Visual feedback is used to display recognition results and control status on the interface; Audio feedback provides adjustable volume beeps for high priority operations; Vibration feedback is used to provide tactile prompts of varying intensities on mobile devices.

[0014] Preferably, the multi-channel user feedback module further includes intelligent error correction and secondary confirmation functions, by predicting the probability of incorrect operation of gesture operations. , if the probability of misoperation When the second confirmation is triggered: , in, is the error operation threshold, Indicates a confirmation sign; like hour, The value is 1, indicating an event Get confirmation; like hour, The value is 0, indicating an event Not confirmed.

[0015] A smart grid control method based on gesture interaction comprises the following steps: Step 1: Collect user gesture data, including image, radar and inertial measurement unit data, filter, denoise and normalize the collected raw data to remove noise and standardize the data form, and the processed data is passed to step 2; Step 2: refer to the output data in step 1, use the timestamp synchronization technology to time align the data, and use the spatial calibration method to perform spatial consistency processing on the data of each modality. The processed data is passed to step 3; Step 3: Reference the alignment data in step 2, use the deep learning model to extract gesture feature points from the image data, analyze the characteristic waveform in the radar data through the signal processing method, parse the motion change data in the inertial measurement unit, and perform denoising and enhancement on the features. The extracted features are passed to step 4. Step 4: Refer to the gesture features extracted in step 3, use the weighted fusion algorithm to weight the gesture features of the image, radar and inertial measurement unit, adjust the weights according to the importance of different modal data, and pass the fused unified feature expression to step 5; Step 5: refer to the fusion feature in step 4, analyze the characteristics of the current environment and the user based on the adaptive learning algorithm, dynamically adjust the weight ratio of each modal feature in the fusion feature, and the adjusted fusion feature is passed to step 6; Step 6: Refer to the optimized fusion features in step 5, use the edge computing framework to quickly analyze the input features, generate corresponding power grid control instructions through the classification model or decision tree method, and the generated control instructions are passed to step 7; Step 7: Refer to the control instruction in step 6, and transmit the gesture recognition status and the corresponding control instruction information to the user through visual display, sound prompt or vibration feedback to realize the closed loop of information interaction.

[0016] The present invention provides a smart grid control system and method based on gesture interaction, which has the following beneficial effects: 1. The present invention realizes efficient integration of multimodal information by weighted fusion of features of images, radar and inertial measurement units, thereby significantly improving the gesture recognition accuracy of the system in complex and changeable environments, enabling the system to flexibly cope with situations such as dim light and background interference, and enhancing the robustness and recognition stability of the system.

[0017] 2. The present invention introduces an adaptive learning module, which can dynamically optimize model parameters and fusion weights based on environmental changes and user gesture characteristics, realize self-adjustment and adaptation of the model, reduce the occurrence of misrecognition and missed recognition, enable the system to adapt to the user's usage habits and gesture characteristics, and greatly improve the recognition accuracy and user experience.

[0018] 3. The present invention deploys lightweight neural networks and instruction generation modules on edge devices to achieve rapid processing of multimodal fusion features and real-time control instruction generation. The use of edge computing significantly reduces the response delay of gesture recognition and power grid control, ensuring the real-time requirements of the system in smart grid control. It is suitable for high-frequency and high-timeliness power grid operation scenarios, and provides strong support for the efficient management of smart grids.

[0019] 4. The present invention designs an intelligent misoperation detection and secondary confirmation mechanism. The system can automatically detect potential misoperations in the gesture recognition process. If the probability of misoperation exceeds a set threshold, a secondary confirmation prompt is provided to the user. This is of great value in high-risk power grid operations, significantly reducing the risk of misoperation, ensuring the safety of power grid control and the stable operation of equipment, and providing additional security for critical operations in smart grid applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a system framework diagram of the present invention; Figure 2 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0021] In order to make the technical personnel in the technical field understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in combination with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a partial embodiment of the present invention, not a complete embodiment. Based on the embodiment of the present invention, other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0022] The present invention is described in detail below in conjunction with the accompanying drawings: Example: Please refer to the attached Figure 1 and attached Figure 2 , an embodiment of the present invention provides a smart grid control system based on gesture interaction, comprising: The multimodal data acquisition and preprocessing module is used to collect the user's gesture data, including image, radar and inertial measurement unit data, and filter, denoise and normalize the collected user's gesture data. The processed data is transmitted to the multimodal data synchronization and alignment module; The multimodal data synchronization and alignment module refers to the output data of the multimodal data acquisition and preprocessing module, aligns the multimodal data in time and space through timestamp synchronization and spatial calibration technology, and transmits the output data to the feature extraction and enhancement module; The feature extraction and enhancement module uses the alignment data from the multimodal data synchronization and alignment module, extracts gesture features from images, radars, and inertial measurement units using deep learning models and signal processing methods, and improves the quality of features through denoising and enhancement algorithms. The extracted features are then passed to the multimodal feature fusion module. The multimodal feature fusion module refers to the gesture feature data of the feature extraction and enhancement module, integrates the image, radar and inertial measurement unit features through a weighted fusion algorithm, dynamically adjusts the weights of each data source, generates a unified gesture feature expression, and passes the fused features to the adaptive learning and dynamic adjustment module; The adaptive learning and dynamic adjustment module refers to the fusion features of the multimodal feature fusion module, and uses the adaptive learning algorithm to dynamically adjust the weight of feature fusion according to environmental changes and user characteristics. The adjusted features are transmitted to the edge computing and instruction generation module; The edge computing and command generation module uses the optimization features of the adaptive learning and dynamic adjustment module to quickly process gesture features based on the edge computing framework, generates power grid control commands through classification models or decision trees, and transmits the commands to the multi-channel user feedback module; The multi-channel user feedback module refers to the control command data of the edge computing and command generation module, and provides users with real-time feedback on gesture recognition status and control command execution through visual, sound or vibration feedback.

[0023] The benefits of multimodal data acquisition and preprocessing modules help remove noise and standardize input data, providing high-quality basic data for subsequent processing and improving recognition accuracy; The benefits of the multimodal data synchronization and alignment module ensure that each modal data can accurately reflect the gesture action at the same time, effectively avoiding the time and space mismatch problem of multimodal data, and significantly improving the accuracy and stability of gesture recognition; Benefits of the feature extraction and enhancement module: Fully exploit the data characteristics of each modality, improve the expressiveness of gesture features, enable the system to recognize gesture features in multiple dimensions, and further improve recognition accuracy and environmental adaptability; The benefits of the multimodal feature fusion module are to improve the recognition robustness of the system in complex environments, enable the system to flexibly respond to changing usage environments, and improve the overall recognition performance; The benefits of the adaptive learning and dynamic adjustment module are that it can adaptively optimize the gesture recognition process, continuously adapt to the user's gesture habits and environmental changes, reduce the occurrence of misidentification and missed recognition, and improve recognition accuracy and user experience; The benefits of edge computing and instruction generation modules enable the system to have excellent real-time performance in high-frequency and high-timeliness power grid control scenarios, meet the rapid control requirements of smart grids, and ensure efficient operation of the system; Benefits of the multi-channel user feedback module: Provide users with real-time feedback on gesture recognition status and control instructions, improve user interaction experience, especially in incorrect operations or high-risk operations, help users discover and correct problems in a timely manner, and ensure the safety of power grid operations.

[0024] The multimodal data acquisition and preprocessing module includes: An image acquisition unit is used to capture gesture images using an RGB camera to obtain two-dimensional spatial information of the user's hand. The image resolution is set to W*H, where W represents the width of the image and H represents the height of the image. A radar acquisition unit, used to acquire the motion depth and speed information of the gesture through a frequency modulated continuous wave radar; Inertial measurement unit, used to collect the three-axis acceleration of the hand and angular velocity , to obtain the dynamic information of the gesture; is the acceleration component in the x direction, is the acceleration component in the y direction, is the acceleration component in the z direction, is the angular velocity component in the x direction, is the angular velocity component in the y direction, is the angular velocity component in the z direction.

[0025] The image acquisition unit provides a high-resolution spatial information basis for subsequent feature extraction, can identify the static shape and position information of gestures, and accurately capture the overall contour of gestures, which is especially suitable for the recognition of subtle gestures. The radar acquisition unit provides the system with rich depth information and speed change data, which helps improve the system's responsiveness and recognition accuracy to dynamic gestures, and is particularly suitable for fast and continuous gesture detection; The benefits of inertial measurement units can complement the deficiencies of image and radar data in the recognition of fast or complex gestures, enhance the system's ability to recognize gesture acceleration and rotation changes, and help accurately determine the directionality and speed characteristics of gestures.

[0026] The feature extraction and enhancement module includes: The convolutional neural network used for image feature extraction performs convolution and pooling on the image data I to obtain the image feature vector : , in, is the weight parameter of the convolutional neural network model, is the image feature vector, is the feature extraction function of the convolutional neural network model, is the input image data; Fast Fourier transform processing unit for radar feature extraction, transforming radar signal Convert to frequency domain signal , its frequency domain eigenvector is: , in, represents the feature vector extracted from radar data, represents the amplitude spectrum of the radar signal in the frequency domain, represents the maximum value in the frequency domain amplitude spectrum of the radar signal, represents the standard deviation of the radar signal frequency domain amplitude spectrum, Indicates the frequency position where the frequency domain amplitude spectrum of the radar signal reaches its maximum value; Long short-term memory network for inertial measurement unit feature extraction, processing time series data of inertial measurement unit , generate IMU feature vector : , in, are the parameters of the long short-term memory network model, is the feature vector, It is the feature extraction function of the long short-term memory network model.

[0027] Achieve high-precision static gesture recognition under complex backgrounds. Under complex backgrounds and changing light conditions, it can accurately capture the contours and details of gestures, provide stable visual input for multimodal fusion, and effectively improve the recognition accuracy and anti-interference ability of the system. The ability to accurately capture the motion characteristics of gestures in insufficient lighting or when there is occlusion, and the addition of radar features effectively supplements the deficiencies of visual features. It is particularly suitable for detecting fast and continuous dynamic gestures, providing the system with sensitivity to the direction and speed of gesture movement, and enhancing the stability and real-time performance of gesture recognition.

[0028] The introduction of inertial measurement unit features improves the system's ability to analyze fast-moving and rotating gestures, allowing the system to accurately identify the direction and angle changes of gestures. The benefit of this unit lies in its sensitivity to dynamic and continuous gestures, allowing the system to recognize static gestures and efficiently respond to complex gestures containing motion information, thereby expanding the applicability and accuracy of gesture recognition.

[0029] The multimodal feature fusion module uses a weighted fusion method to fusion the image feature vector , radar feature vector and the inertial measurement unit eigenvector Fusion is performed to generate a fusion feature vector : , in, , , is the weight of image, radar and feature, satisfying .

[0030] Image features mainly provide static shape information of gestures, radar features enhance the capture of gesture movement depth and speed, and inertial measurement unit features supplement the dynamic direction and rotation features of gestures, effectively integrating information from different feature sources and significantly improving the system's gesture recognition accuracy in complex and changing environments; Increasing the weight of radar and inertial measurement unit features in low-light environments can make up for the shortcomings of image features and enhance the adaptability and robustness of the system; By realizing accurate recognition of complex gestures and reasonable allocation of feature weights, the system can more flexibly respond to changes in various gestures and provide comprehensive information support for smart grid control; When there is sufficient light, the weight of the radar and inertial measurement unit can be reduced, and the computational consumption of the modality can be reduced. By optimizing the utilization of computing resources, the system can achieve efficient resource allocation while maintaining high recognition accuracy, further improving the overall performance and real-time performance of the system.

[0031] The adaptive learning and dynamic adjustment module automatically adjusts the weight of the fusion features according to environmental changes, based on the light intensity. and background complexity Calculate weight parameters , and : , in, Indicates the ambient light intensity. Indicates background complexity; The adaptive learning and dynamic adjustment module dynamically optimizes the gesture recognition model through online learning methods and updates the model parameters using the gradient descent method. , the update formula is: , in, is the learning rate, is the cross entropy loss, is the predicted output of the model, Indicated in The model parameters after iterations are represents the model parameters at the tth iteration, represents the true label or target value, Indicates that the gradient is with respect to the parameter The set of partial derivatives of .

[0032] The adaptive learning and dynamic adjustment module can dynamically adjust the weight of fusion features according to real-time environmental information to achieve environmental adaptation; In low-light conditions, the weight of radar and inertial measurement units is increased to effectively compensate for the lack of image features, thereby improving the system's robustness to gestures and environmental adaptability; Through online learning, the adaptive module continuously and dynamically optimizes model parameters according to the user's usage habits and gesture characteristics to achieve personalized adjustments; Online learning enables the model to adapt to the user's gesture characteristics and usage habits, reducing misrecognition and missed recognition, improving the system's recognition accuracy and personalized performance, and enhancing the user experience; The adaptive learning module can dynamically optimize according to the feedback of the real-time environment and the recognition error, gradually reducing the recognition error. Through iterative updates, the model's weight allocation of features and parameter optimization will gradually approach the optimal recognition result, effectively reducing the misrecognition rate. Especially in power grid operation, the accuracy of gesture recognition is crucial. Adaptive optimization ensures high recognition accuracy and guarantees the safety and stability of power grid operation. Through online learning and adaptive dynamic adjustment, the system can adapt to environmental changes in a short period of time without frequent full model training, thereby improving the system's response speed. At the same time, adaptive adjustment reduces unnecessary computing consumption and reduces the frequency of feature adjustments when environmental changes are small, thereby achieving efficient use of resources and further enhancing the system's real-time and overall performance.

[0033] Edge computing and command generation modules are used to quickly process multimodal fusion features And generate control instructions, use lightweight neural network to improve computing efficiency, use knowledge distillation to optimize the lightweight model, and its distillation loss function is: , in, is the cross entropy loss of hard labels, is the divergence loss of soft labels, is the weight coefficient of the cross entropy loss, Represents distillation loss.

[0034] The edge computing and command generation module processes multimodal fusion features quickly on edge devices and generates control commands, reducing the time required for data transmission to the cloud and delays. It can meet the requirements of high-frequency and real-time control in smart grid systems and ensure timely response of gesture recognition and control command generation. Lightweight neural networks are used to perform calculations on edge devices. By reducing the complexity of the model, the computing requirements and resource consumption are reduced, and the computing efficiency is improved. The lightweight model can run stably on resource-constrained edge devices, reduce the power consumption of the system, and extend the service life of the device. It is particularly suitable for long-term operation under low energy consumption conditions. The use of knowledge distillation makes the lightweight model close to the complex model in terms of accuracy, maintains high recognition accuracy of gestures, and effectively improves the practicality and reliability of the system; The combination of edge computing and fast instruction generation enables the system to respond to control requirements in complex power grid environments in real time. The lightweight model can adapt to changes in multimodal characteristics and generate stable control instructions, providing accurate and timely control decisions in power grid operations, providing strong support for the efficient management of smart grids, and improving the intelligence level and operational efficiency of power grid management.

[0035] The multi-channel user feedback module provides users with confirmation information of gesture recognition status and control commands through visual, audio and vibration feedback, including: Visual feedback is used to display recognition results and control status on the interface; Audio feedback provides adjustable volume beeps for high priority operations; Vibration feedback is used to provide tactile prompts of varying intensities on mobile devices.

[0036] The multi-channel user feedback module further includes intelligent error correction and secondary confirmation functions, which can predict the probability of incorrect operation of gesture operations. , if the probability of misoperation When the second confirmation is triggered: , in, is the error operation threshold, Indicates a confirmation sign; like hour, The value is 1, indicating an event Get confirmation; If hour, The value is 0, indicating an event Not confirmed.

[0037] The multi-channel user feedback module effectively reduces the risk of misoperation through intelligent error correction and secondary confirmation functions. The system determines whether secondary confirmation is required based on the probability of misoperation of gesture operations. If the probability of misoperation exceeds the preset threshold, the secondary confirmation process is triggered to ensure that the user's key operations are double-confirmed. This is particularly important when it comes to high-risk power grid operations, reducing the possibility of misoperation and ensuring the safety of power grid control and the stability of equipment. Through multi-channel feedback such as vision, audio and touch, the system can provide users with real-time feedback on gesture recognition status and control command execution, allowing users to intuitively understand the operation status. If the system detects the risk of misoperation and triggers secondary confirmation, users can receive confirmation prompts in a variety of ways and respond more quickly, thereby improving users' trust and satisfaction with the system. In the case of high probability of wrong operation, the system will set the confirmation flag to 0 to avoid automatic execution of the operation and ensure that the key control instructions are handled prudently. If the user confirms the operation intention, the system will set the confirmation flag to 1 to eliminate the risk of automatic execution of wrong operation when the probability of wrong operation is high. This function is particularly suitable for important operation links in smart grids. Through the secondary confirmation mechanism, the system ensures the accuracy of high-risk operations and avoids system damage and safety hazards. The intelligent error correction function continuously predicts and learns the user's operation behavior, adjusts the error probability threshold and response mode according to the gesture characteristics, and enables the system to gradually adapt to the habits of different users. Through precise control of the error probability, the system's fault tolerance is improved, and the system's adaptability in complex and changeable power grid environments is significantly enhanced, ensuring the stability and reliability of the system in long-term use.

[0038] A smart grid control method based on gesture interaction, comprising: Step 1: Collect user gesture data, including image, radar and inertial measurement unit data, filter, denoise and normalize the collected raw data to remove noise and standardize the data form, and the processed data is passed to step 2; Step 2: refer to the output data in step 1, use the timestamp synchronization technology to time align the data, and use the spatial calibration method to perform spatial consistency processing on the data of each modality. The processed data is passed to step 3; Step 3: Reference the alignment data in step 2, use the deep learning model to extract gesture feature points from the image data, analyze the characteristic waveform in the radar data through the signal processing method, parse the motion change data in the inertial measurement unit, and perform denoising and enhancement on the features. The extracted features are passed to step 4. Step 4: Refer to the gesture features extracted in step 3, use the weighted fusion algorithm to weight the gesture features of the image, radar and inertial measurement unit, adjust the weights according to the importance of different modal data, and pass the fused unified feature expression to step 5; Step 5: refer to the fusion feature in step 4, analyze the characteristics of the current environment and the user based on the adaptive learning algorithm, dynamically adjust the weight ratio of each modal feature in the fusion feature, and the adjusted fusion feature is passed to step 6; Step 6: Refer to the optimized fusion features in step 5, use the edge computing framework to quickly analyze the input features, generate corresponding power grid control instructions through the classification model or decision tree method, and the generated control instructions are passed to step 7; Step 7: Refer to the control instruction in step 6, and transmit the gesture recognition status and the corresponding control instruction information to the user through visual display, sound prompt or vibration feedback to realize the closed loop of information interaction.

[0039] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A smart grid control system based on gesture interaction, characterized in that: include: The multimodal data acquisition and preprocessing module is used to collect the user's gesture data, including image, radar and inertial measurement unit data, and filter, denoise and normalize the collected user's gesture data. The processed data is transmitted to the multimodal data synchronization and alignment module; The multimodal data synchronization and alignment module refers to the output data of the multimodal data acquisition and preprocessing module, aligns the multimodal data in time and space through timestamp synchronization and spatial calibration technology, and transmits the output data to the feature extraction and enhancement module; The feature extraction and enhancement module uses the alignment data from the multimodal data synchronization and alignment module, extracts gesture features from images, radars, and inertial measurement units using deep learning models and signal processing methods, and improves the quality of features through denoising and enhancement algorithms. The extracted features are then passed to the multimodal feature fusion module. The multimodal feature fusion module refers to the gesture feature data of the feature extraction and enhancement module, integrates the image, radar and inertial measurement unit features through a weighted fusion algorithm, dynamically adjusts the weights of each data source, generates a unified gesture feature expression, and passes the fused features to the adaptive learning and dynamic adjustment module; The adaptive learning and dynamic adjustment module refers to the fusion features of the multimodal feature fusion module, and uses the adaptive learning algorithm to dynamically adjust the weight of feature fusion according to environmental changes and user characteristics. The adjusted features are transmitted to the edge computing and instruction generation module; The edge computing and command generation module uses the optimization features of the adaptive learning and dynamic adjustment module to quickly process gesture features based on the edge computing framework, generates power grid control commands through classification models or decision trees, and transmits the commands to the multi-channel user feedback module; The multi-channel user feedback module refers to the control command data of the edge computing and command generation module, and provides users with real-time feedback on gesture recognition status and control command execution through visual, sound or vibration feedback.

2. The smart grid control system based on gesture interaction according to claim 1, characterized in that: The multimodal data acquisition and preprocessing module includes: An image acquisition unit is used to capture gesture images using an RGB camera to obtain two-dimensional spatial information of the user's hand. The image resolution is set to W*H, where W represents the width of the image and H represents the height of the image. A radar acquisition unit, used to acquire the motion depth and speed information of the gesture through a frequency modulated continuous wave radar; Inertial measurement unit, used to collect the three-axis acceleration of the hand and angular velocity , to obtain the dynamic information of the gesture; is the acceleration component in the x direction, is the acceleration component in the y direction, is the acceleration component in the z direction, is the angular velocity component in the x direction, is the angular velocity component in the y direction, is the angular velocity component in the z direction.

3. The smart grid control system based on gesture interaction according to claim 2, characterized in that: The feature extraction and enhancement module includes: The convolutional neural network used for image feature extraction performs convolution and pooling on the image data I to obtain the image feature vector : ， in, is the weight parameter of the convolutional neural network model, is the image feature vector, is the feature extraction function of the convolutional neural network model, is the input image data; Fast Fourier transform processing unit for radar feature extraction, transforming radar signal Convert to frequency domain signal , its frequency domain eigenvector is: ， in, represents the feature vector extracted from radar data, represents the amplitude spectrum of the radar signal in the frequency domain, represents the maximum value in the frequency domain amplitude spectrum of the radar signal, represents the standard deviation of the radar signal frequency domain amplitude spectrum, Indicates the frequency position where the frequency domain amplitude spectrum of the radar signal reaches its maximum value; Long short-term memory network for inertial measurement unit feature extraction, processing time series data of inertial measurement unit , generate IMU feature vector : ， in, are the parameters of the long short-term memory network model, is the feature vector, It is the feature extraction function of the long short-term memory network model.

4. The smart grid control system based on gesture interaction according to claim 3 is characterized in that: The multimodal feature fusion module uses a weighted fusion method to fusion the image feature vector , radar feature vector and the inertial measurement unit eigenvector Fusion is performed to generate a fusion feature vector : ， in, , , is the weight of image, radar and feature, satisfying .

5. The smart grid control system based on gesture interaction according to claim 4, characterized in that: The adaptive learning and dynamic adjustment module automatically adjusts the weight of the fusion feature according to environmental changes, based on the light intensity. and background complexity Calculate weight parameters , and : ， in, Indicates the ambient light intensity. Represents the background complexity.

6. The smart grid control system based on gesture interaction according to claim 1, characterized in that: The adaptive learning and dynamic adjustment module dynamically optimizes the gesture recognition model through an online learning method and updates the model parameters using the gradient descent method. , the update formula is: ， in, is the learning rate, is the cross entropy loss, is the predicted output of the model, Indicated in The model parameters after iterations are represents the model parameters at the tth iteration, represents the true label or target value, Indicates that the gradient is with respect to the parameter The set of partial derivatives of .

7. The smart grid control system based on gesture interaction according to claim 1, characterized in that: The edge computing and instruction generation module is used to quickly process multimodal fusion features. And generate control instructions, use lightweight neural networks to improve computing efficiency, and use knowledge distillation to optimize the lightweight model. Its distillation loss function is: ， in, is the cross entropy loss of hard labels, is the divergence loss of soft labels, is the weight coefficient of the cross entropy loss, Represents distillation loss.

8. The smart grid control system based on gesture interaction according to claim 1, characterized in that: The multi-channel user feedback module provides the user with confirmation information of gesture recognition status and control instructions through visual, audio and vibration feedback, wherein: Visual feedback is used to display recognition results and control status on the interface; Audio feedback provides adjustable volume beeps for high priority operations; Vibration feedback is used to provide tactile prompts of varying intensities on mobile devices.

9. The smart grid control system based on gesture interaction according to claim 1, characterized in that: The multi-channel user feedback module further includes intelligent error correction and secondary confirmation functions, which predicts the probability of incorrect operation of gesture operations. , if the probability of misoperation When the second confirmation is triggered: ， in, is the error operation threshold, Indicates a confirmation sign; like hour, The value is 1, indicating an event Get confirmation; like hour, The value is 0, indicating an event Not confirmed.

10. A smart grid control method based on gesture interaction, according to a smart grid control system based on gesture interaction according to any one of claims 1 to 9, characterized in that: include: Step 1: Collect user gesture data, including image, radar and inertial measurement unit data, filter, denoise and normalize the collected raw data to remove noise and standardize the data form, and the processed data is passed to step 2; Step 2: refer to the output data in step 1, use the timestamp synchronization technology to time align the data, and use the spatial calibration method to perform spatial consistency processing on the data of each modality. The processed data is passed to step 3; Step 3: Reference the alignment data in step 2, use the deep learning model to extract gesture feature points from the image data, analyze the characteristic waveform in the radar data through the signal processing method, parse the motion change data in the inertial measurement unit, and perform denoising and enhancement on the features. The extracted features are passed to step 4. Step 4: Refer to the gesture features extracted in step 3, use the weighted fusion algorithm to weight the gesture features of the image, radar and inertial measurement unit, adjust the weights according to the importance of different modal data, and pass the fused unified feature expression to step 5; Step 5: refer to the fusion feature in step 4, analyze the characteristics of the current environment and the user based on the adaptive learning algorithm, dynamically adjust the weight ratio of each modal feature in the fusion feature, and the adjusted fusion feature is passed to step 6; Step 6: Refer to the optimized fusion features in step 5, use the edge computing framework to quickly analyze the input features, generate corresponding power grid control instructions through the classification model or decision tree method, and the generated control instructions are passed to step 7; Step 7: Refer to the control instruction in step 6, and transmit the gesture recognition status and the corresponding control instruction information to the user through visual display, sound prompt or vibration feedback to realize the closed loop of information interaction.

Images