Virtual interaction method and system for power grid production command system

Abstract

The invention relates to the field of virtual interaction, and provides a virtual interaction method and system for a power grid production command system, and the method comprises the steps: receiving a user interaction gesture video through a virtual technology, and obtaining an interaction intention supplement text inputted by a user; carrying out semantic coding on the user interaction gesture video by adopting a data processing technology based on a deep neural network, and carrying out semantic coding on the interaction intention supplementary text; based on the cross-modal joint enhanced coding representation between the user interaction gesture semantic feature and the interaction intention supplementary text semantic feature, the category label of the interaction intention is automatically evaluated, and the interaction operation content of the power grid production command system is determined. Thus, through a dual confirmation mechanism of gesture and text input, the intention of the user can be identified more accurately, the risk of misoperation is further reduced, and the intelligent level of the power grid production command system applied to training simulation is improved.

Description

Technical Field

[0001] This application relates to the field of virtual interaction, and more specifically, to a virtual interaction method and system for a power grid production command system. Background Art

[0002] The power grid production command system involves complex power network management and scheduling, and it is necessary to ensure the high reliability of the system. Any operation error may trigger a safety accident and affect the stability of power supply. Therefore, production command training plays an extremely important role in the power system. Through training, operators can be familiar with various operation processes in the power grid production command system, including daily maintenance, equipment monitoring, data recording, etc., and simulate the normal, abnormal, and accident states of equipment.

[0003] Power grid production command training has gradually incorporated advanced simulation technologies to further improve training effectiveness and safety. Through simulation technologies, a highly realistic simulation environment of the power grid production command system can be constructed, enabling operators to receive training in a nearly real working scenario. In power grid production command training simulation, virtual technologies can simulate various operating states of the power grid, including normal, abnormal, and accident states. Operators can perform relevant operations in the virtual environment and observe the responses and changes of the system in real time. This simulation can help operators handle various emergencies without actual risks, improving their emergency handling capabilities and decision-making accuracy. In addition, virtual technologies can also customize different scenarios and cases according to training needs, meeting the requirements of operators at different levels and with different needs.

[0004] However, in traditional power grid production command training simulation systems, a single interaction method (such as relying only on gestures or manual input) may not be sufficient to fully express complex operation instructions. For example, relying only on gesture recognition may not accurately capture subtle gesture differences, especially in cases where fine control is required. Although manual input is precise, it may not respond in time in emergency situations, and the operation process is cumbersome. The single interaction method lacks redundancy and flexibility. Once there is a problem with the input of a certain modality, the system cannot effectively recognize the true intention of the user, which may lead to operation errors.

[0005] Therefore, an optimized virtual interaction solution for the power grid production command system is desired. Summary of the Invention

[0006] This application aims at the deficiencies in the prior art and provides a virtual interaction method and system for a power grid production command system.

[0007] According to one aspect of this application, a virtual interaction method for a power grid production command system is provided, which includes:

[0008] Obtain the user interaction gesture video collected by the camera;

[0009] In response to receiving the user interaction gesture video, pop up an interaction intention text supplement box;

[0010] Receive the interaction intention supplementary text input by the user in the interaction intention text supplement box;

[0011] Perform interaction intention gesture semantic encoding on the user interaction gesture video to obtain an interaction intention gesture semantic encoding feature map;

[0012] Perform semantic encoding on the interaction intention supplementary text to obtain an interaction intention supplementary text semantic encoding feature vector;

[0013] Perform text-video cross-modal joint enhancement encoding on the interaction intention supplementary text semantic encoding feature vector and the interaction intention gesture semantic encoding feature map to obtain an interaction intention multi-modal joint enhancement encoding feature, including: performing linear transformation and gradient magnitude-based masking processing on the interaction intention supplementary text semantic encoding feature vector to obtain a masked first interaction intention supplementary text semantic meta-modal auxiliary feature and a masked second interaction intention supplementary text semantic meta-modal auxiliary feature; based on the masked first interaction intention supplementary text semantic meta-modal auxiliary feature and the masked second interaction intention supplementary text semantic meta-modal auxiliary feature, perform feature screening on the interaction intention gesture semantic encoding feature map to obtain the interaction intention multi-modal joint enhancement encoding feature;

[0014] Based on the interaction intention multi-modal joint enhancement encoding feature, obtain an interaction intention recognition result;

[0015] Based on the interaction intention recognition result, determine the interaction operation content of the power grid production command system.

[0016] According to another aspect of the present application, there is provided a virtual interaction system for a power grid production command system, which includes:

[0017] A user interaction video data acquisition module for obtaining a user interaction gesture video collected by a camera;

[0018] An interaction intention supplement module for popping up an interaction intention text supplement box in response to receiving the user interaction gesture video;

[0019] An interaction intention supplementary text data receiving module for receiving the interaction intention supplementary text input by the user in the interaction intention text supplement box;

[0020] A user interaction video data semantic encoding module for performing interaction intention gesture semantic encoding on the user interaction gesture video to obtain an interaction intention gesture semantic encoding feature map;

[0021] An interactive intention supplementary text semantic encoding module, configured to perform semantic encoding on the interactive intention supplementary text to obtain an interactive intention supplementary text semantic encoding feature vector;

[0022] An interactive intention multi-modal joint enhancement encoding feature generation module, configured to perform text-video cross-modal joint enhancement encoding on the interactive intention supplementary text semantic encoding feature vector and the interactive intention gesture semantic encoding feature map to obtain an interactive intention multi-modal joint enhancement encoding feature, including: an interactive intention semantic encoding feature mask processing unit, configured to perform linear transformation and gradient magnitude-based mask processing on the interactive intention supplementary text semantic encoding feature vector to obtain a masked first interactive intention supplementary text semantic meta-modal auxiliary feature and a masked second interactive intention supplementary text semantic meta-modal auxiliary feature; an interactive intention gesture semantic encoding feature screening unit, configured to perform feature screening on the interactive intention gesture semantic encoding feature map based on the masked first interactive intention supplementary text semantic meta-modal auxiliary feature and the masked second interactive intention supplementary text semantic meta-modal auxiliary feature to obtain the interactive intention multi-modal joint enhancement encoding feature;

[0023] An interactive intention recognition result generation module, configured to obtain an interactive intention recognition result based on the interactive intention multi-modal joint enhancement encoding feature;

[0024] An interactive operation content determination module, configured to determine the interactive operation content of the power grid production command system based on the interactive intention recognition result.

[0025] Due to the adoption of the above technical solutions, the present application has significant technical effects:

[0026] The virtual interaction method and system for the power grid production command system provided by the present application receive a user interaction gesture video through virtual technology and obtain an interactive intention supplementary text input by the user, and use a data processing technology based on a deep neural network to perform semantic encoding on the user interaction gesture video and perform semantic encoding on the interactive intention supplementary text, so as to automatically evaluate the category label of the interactive intention based on the cross-modal joint enhancement encoding representation between the user interaction gesture semantic feature and the interactive intention supplementary text semantic feature, and determine the interactive operation content of the power grid production command system. In this way, through the dual confirmation mechanism of gesture and text input, the user's intention can be more accurately recognized, thereby reducing the risk of misoperation and improving the intelligent level of the power grid production command system in training simulation applications. Description of the Drawings

[0027] The above and other objects, features, and advantages of the present application will become more apparent by describing the embodiments of the present application in more detail with reference to the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation to the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0028] Figure 1 It is a flowchart of a virtual interaction method for a power grid production command system according to an embodiment of the present application.

[0029] Figure 2 It is a schematic diagram of data flow of a virtual interaction method for a power grid production command system according to an embodiment of the present application.

[0030] Figure 3 It is a flowchart of performing text-video cross-modal joint enhancement coding on the supplementary text semantic coding feature vector of the interaction intention and the gesture semantic coding feature map of the interaction intention to obtain the multi-modal joint enhancement coding feature of the interaction intention in the virtual interaction method for a power grid production command system according to an embodiment of the present application.

[0031] Figure 4 It is a flowchart of performing linear transformation and mask processing based on gradient magnitude on the supplementary text semantic coding feature vector of the interaction intention to obtain the masked first supplementary text semantic meta-modal auxiliary feature of the interaction intention and the masked second supplementary text semantic meta-modal auxiliary feature of the interaction intention in the virtual interaction method for a power grid production command system according to an embodiment of the present application.

[0032] Figure 5 It is a flowchart of performing feature screening on the gesture semantic coding feature map of the interaction intention based on the masked first supplementary text semantic meta-modal auxiliary feature of the interaction intention and the masked second supplementary text semantic meta-modal auxiliary feature of the interaction intention to obtain the multi-modal joint enhancement coding feature of the interaction intention in the virtual interaction method for a power grid production command system according to an embodiment of the present application.

[0033] Figure 6 It is a system block diagram of a virtual interaction system for a power grid production command system according to an embodiment of the present application. Detailed implementation manners

[0034] Next, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. It should be understood that the present application is not limited by the example embodiments described herein.

[0035] The power grid production command system is a key link in power network management and dispatching, and the accuracy of its operation is crucial for the stability of power supply. Therefore, it is essential to provide systematic training to operators to ensure that they can proficiently master various operation processes, including daily maintenance, equipment monitoring, and data recording, and can handle different states of equipment.

[0036] Nowadays, advanced simulation technology is integrated into power grid production command training to improve effectiveness and safety. This technology can construct a realistic simulation environment, enabling operators to train in a near-real scenario. Virtual technology can simulate various operating states of the power grid. Operators can operate in the virtual environment and observe the changes in system responses, enhancing their emergency handling and decision-making capabilities. It can also customize scenario cases to meet the needs of different personnel.

[0037] However, traditional power grid production command training simulation systems may rely on a single interaction method, such as gesture recognition or manual input, which may not be sufficient to accurately convey complex operation instructions in some cases. The single interaction method lacks redundancy and flexibility. Once a problem occurs, it may cause the system to fail to correctly recognize the user's intention, thereby increasing the risk of operation errors.

[0038] To address the above technical problems, the technical concept of this application is to receive the user interaction gesture video collected by the camera through virtual technology and pop up an interactive intention text supplement box, and input the interactive intention supplement text in the interactive intention text supplement box. And adopt video analysis and processing technology based on a deep neural network to perform semantic encoding on the user interaction gesture video and perform semantic encoding on the interactive intention supplement text. Based on the cross-modal joint enhanced encoding representation between the semantic features of the user interaction gesture and the semantic features of the interactive intention supplement text, automatically evaluate the category label of the interactive intention, and determine the interactive operation content of the power grid production command system. In this way, a simple and intuitive gesture can speed up the response speed. And through the dual confirmation mechanism of gesture and text input, the system can more accurately identify the user's intention, reduce the risk of misoperation caused by a single input method, and improve the intelligent level of the power grid production command system in training simulation applications.

[0039] Figure 1 It is a flowchart of a virtual interaction method for a power grid production command system according to an embodiment of the present application. Figure 2 It is a schematic diagram of data flow of a virtual interaction method for a power grid production command system according to an embodiment of the present application. As Figure 1 and Figure 2As shown, a virtual interaction method for a power grid production command system according to an embodiment of the present application includes: S110, obtaining a user interaction gesture video collected by a camera; S120, in response to receiving the user interaction gesture video, popping up an interaction intention text supplement box; S130, receiving interaction intention supplement text input by the user in the interaction intention text supplement box; S140, performing interaction intention gesture semantic encoding on the user interaction gesture video to obtain an interaction intention gesture semantic encoding feature map; S150, performing semantic encoding on the interaction intention supplement text to obtain an interaction intention supplement text semantic encoding feature vector; S160, performing text-video cross-modal joint enhancement encoding on the interaction intention supplement text semantic encoding feature vector and the interaction intention gesture semantic encoding feature map to obtain an interaction intention multi-modal joint enhancement encoding feature; S170, obtaining an interaction intention recognition result based on the interaction intention multi-modal joint enhancement encoding feature; S180, determining the interaction operation content of the power grid production command system based on the interaction intention recognition result.

[0040] In steps S110 and S120, a user interaction gesture video collected by a camera is obtained, and in response to receiving the user interaction gesture video, an interaction intention text supplement box is popped up. It should be understood that during power grid production command training simulation, the system will create a highly realistic virtual environment for power grid production command. In the virtual environment, a virtual camera is used to capture the user's interaction gesture data. When the system receives the user interaction gesture video data collected by the virtual camera, an event will be triggered. This event will cause an interaction intention text supplement box to pop up on the virtual interaction interface, so that the user can input additional interaction intention information. Combining the user gesture data and the text input data can help the system more accurately understand the user's intention.

[0041] In step S130, the interaction intention supplement text input by the user in the interaction intention text supplement box is received. Correspondingly, considering that although the camera can capture the user's gestures, the meaning of some gestures may not be clear enough or there may be ambiguities. That is, for some complex operations or situations that require detailed explanations, relying solely on gestures may not be sufficient to convey enough information. Based on this, in the technical solution of the present application, the interaction intention supplement text input by the user in the interaction intention text supplement box is received. That is, by allowing the user to input text to describe their intentions, it can help the system more accurately understand the user's specific needs.

[0042] In step S140, perform interactive intention gesture semantic encoding on the user interactive gesture video to obtain an interactive intention gesture semantic encoding feature map. Specifically, in the embodiment of the present application, performing interactive intention gesture semantic encoding on the user interactive gesture video to obtain an interactive intention gesture semantic encoding feature map includes: inputting the user interactive gesture video into an interactive intention gesture semantic encoder based on a three-dimensional convolutional neural network model to obtain the interactive intention gesture semantic encoding feature map. It should be understood that the user interactive gesture video is a dynamic time-sequential change process, that is, there is time-sequential feature change information in the dimension of time for the interactive gesture. However, the traditional two-dimensional convolutional neural network (2DCNN) can only process single-frame images, while the three-dimensional convolutional neural network model can process continuous video frames, that is, it can capture the continuity in time, which is very important for understanding dynamic gestures because gestures often contain continuous actions in time. Based on this, in the technical solution of the present application, the user interactive gesture video is input into an interactive intention gesture semantic encoder based on a three-dimensional convolutional neural network model to capture the changes in the gesture at different time points and obtain an interactive intention gesture semantic encoding feature map.

[0043] In step S150, perform semantic encoding on the interactive intention supplementary text to obtain an interactive intention supplementary text semantic encoding feature vector. Correspondingly, considering that the interactive intention supplementary text contains semantic information of the user's true intention, therefore, in order to better capture and extract the semantic meaning between the contexts in the interactive intention supplementary text input by the user, in the technical solution of the present application, perform semantic encoding on the interactive intention supplementary text to obtain an interactive intention supplementary text semantic encoding feature vector, so that the actual semantics expressed by the user can be better understood, thereby improving the accuracy of the system's gesture intention recognition. In particular, in a specific embodiment of the present application, a transformer-based encoder including an embedding layer is used to perform semantic encoding on the interactive intention supplementary text to obtain an interactive intention supplementary text semantic encoding feature vector.

[0044] In step S160, text-video cross-modal joint enhancement coding is performed on the supplementary text semantic coding feature vector of the interaction intention and the semantic coding feature map of the interaction intention gesture to obtain the multi-modal joint enhancement coding feature of the interaction intention. It should be understood that the supplementary text semantic coding feature vector of the interaction intention contains the semantic information of the text content input by the user, which may be a specific explanation of a certain gesture action or a further description of the operation. The semantic coding feature map of the interaction intention gesture contains the key features of the gesture action extracted from the video frame, such as the position, posture, and movement trajectory of the hand. That is, the semantic coding feature of the interaction intention gesture may not be clear enough or there may be multiple interpretations, while the supplementary text semantic coding feature of the interaction intention can provide specific context. Therefore, in order to perform feature screening and enhancement on the semantic coding feature of the interaction intention gesture based on the supplementary text semantic coding feature of the interaction intention, so as to more accurately and clearly understand the user's interaction intention, in the technical solution of this application, text-video cross-modal joint enhancement coding is performed on the supplementary text semantic coding feature vector of the interaction intention and the semantic coding feature map of the interaction intention gesture to obtain the multi-modal joint enhancement coding feature map of the interaction intention as the multi-modal joint enhancement coding feature of the interaction intention.

[0045] Specifically, first, perform a linear transformation on the supplementary text semantic coding feature vector of the interaction intention based on the first weight modifier matrix and the second weight modifier matrix to obtain a first supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector and a second supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector of the interaction intention. That is, use two different weight modifier matrices for linear transformation, thereby broadening the dimension of feature expression and enabling the model to identify and understand the mutual relationship between the supplementary text semantic coding features of the interaction intention and the semantic coding features of the interaction intention gesture from multiple perspectives. Then, based on the gradient magnitudes at each position in the first and second meta-modal auxiliary feature mapping pattern representation vectors respectively, perform mask processing on the eigenvalues in each representation vector based on the gradient magnitudes. In particular, the gradient magnitude reflects the degree of change of each element within the feature mapping pattern representation vector, and the use of the mask technique can emphasize those features with significant changes, so that in the subsequent feature screening stage, more attention can be focused on key features. Subsequently, use the masked first supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector of the interaction intention and combine it with the hyperbolic tangent function to preliminarily screen the semantic coding feature map of the interaction intention gesture to obtain a preliminarily screened semantic coding feature map of the interaction intention gesture. It should be understood that the hyperbolic tangent function, as an activation function, has an output value range between -1 and +1, which can clearly distinguish different regions within the feature map, and thus highlight the features closely related to the semantic coding features of the interaction intention gesture during the preliminary screening process. Furthermore, use the masked second supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector of the interaction intention and combine it with the Sigmoid function to further screen the features after the preliminary screening. That is, the Sigmoid function is also a commonly used activation function, and its output value range is between 0 and 1, which is suitable for probability prediction and binary decision scenarios. Therefore, in this step, the Sigmoid function can assist in identifying the truly key features, thereby forming the final multi-modal joint enhanced coding feature map of the interaction intention.

[0046] Specifically, Figure 3 The figure is a flowchart for text-video cross-modal joint enhancement coding of the supplementary text semantic coding feature vector of the interaction intention and the semantic coding feature map of the interaction intention gesture to obtain a multi-modal joint enhanced coding feature of the interaction intention in the virtual interaction method for the power grid production command system according to an embodiment of the present application. As Figure 3As shown, text-video cross-modal joint enhancement encoding is performed on the supplementary text semantic encoding feature vector of the interaction intention and the gesture semantic encoding feature map of the interaction intention to obtain the multi-modal joint enhancement encoding feature of the interaction intention, including: S161. Perform linear transformation and gradient magnitude-based masking processing on the supplementary text semantic encoding feature vector of the interaction intention to obtain a masked first supplementary text semantic meta-modal auxiliary feature of the interaction intention and a masked second supplementary text semantic meta-modal auxiliary feature of the interaction intention; S162. Based on the masked first supplementary text semantic meta-modal auxiliary feature of the interaction intention and the masked second supplementary text semantic meta-modal auxiliary feature of the interaction intention, perform feature screening on the gesture semantic encoding feature map of the interaction intention to obtain the multi-modal joint enhancement encoding feature of the interaction intention.

[0047] More specifically, Figure 4 In the virtual interaction method for the power grid production command system according to the embodiment of the present application, the flowchart of performing linear transformation and gradient magnitude-based masking processing on the supplementary text semantic encoding feature vector of the interaction intention to obtain a masked first supplementary text semantic meta-modal auxiliary feature of the interaction intention and a masked second supplementary text semantic meta-modal auxiliary feature of the interaction intention. As Figure 4 shown, performing linear transformation and gradient magnitude-based masking processing on the supplementary text semantic encoding feature vector of the interaction intention to obtain a masked first supplementary text semantic meta-modal auxiliary feature of the interaction intention and a masked second supplementary text semantic meta-modal auxiliary feature of the interaction intention includes: S1611. Perform linear transformation on the supplementary text semantic encoding feature vector of the interaction intention based on the first weight modifier matrix and the second weight modifier matrix to obtain a first supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector and a second supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector; S1612. Based on the gradient magnitude at each position in the first supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector, perform gradient magnitude-based masking processing on the first supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector to obtain a masked first supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector as the masked first supplementary text semantic meta-modal auxiliary feature; S1613. Based on the gradient magnitude at each position in the second supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector, perform gradient magnitude-based masking processing on the second supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector to obtain a masked second supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector as the masked second supplementary text semantic meta-modal auxiliary feature.

[0048] More specifically, in the embodiments of the present application, a linear transformation of the supplementary text semantic coding feature vector of the interaction intention is performed based on a first weight modifier matrix and a second weight modifier matrix to obtain a first supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector and a second supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector of the interaction intention, including: multiplying the supplementary text semantic coding feature vector of the interaction intention by the first weight modifier matrix, and adding the obtained first supplementary text semantic modifier vector of the interaction intention to the first modifier bias vector position by position to obtain the first supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector of the interaction intention; multiplying the supplementary text semantic coding feature vector of the interaction intention by the second weight modifier matrix, and adding the obtained second supplementary text semantic modifier vector of the interaction intention to the second modifier bias vector position by position to obtain the second supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector of the interaction intention.

[0049] More specifically, in the embodiments of the present application, based on the gradient magnitudes at each position in the first supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector of the interaction intention, a mask processing based on the gradient magnitudes is performed on the first supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector of the interaction intention to obtain a masked first supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector, including: calculating the square of the difference between the feature values at the (i + 1)-th position and the (i - 1)-th position in the first supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector of the interaction intention to obtain a first supplementary text semantic gradient magnitude representation vector of the interaction intention; calculating the absolute value of the square root of each feature value in the first supplementary text semantic gradient magnitude representation vector of the interaction intention to obtain a first supplementary text semantic gradient magnitude modulation representation vector of the interaction intention; performing mask processing on the first supplementary text semantic gradient magnitude modulation representation vector of the interaction intention, and in response to the feature values at each position in the first supplementary text semantic gradient magnitude modulation representation vector of the interaction intention being greater than a predetermined threshold, setting the feature value at this position to 1 and the rest to 0 to obtain the masked first supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector of the interaction intention.

[0050] More specifically, Figure 5 For the virtual interaction method for the power grid production command system according to the embodiments of the present application, a flowchart of feature screening of the interaction intention gesture semantic coding feature map based on the masked first supplementary text semantic meta-modal auxiliary feature and the masked second supplementary text semantic meta-modal auxiliary feature to obtain the interaction intention multi-modal joint enhanced coding feature. As Figure 5As shown, based on the masked first interaction intention supplementary text semantic meta-modal auxiliary feature and the masked second interaction intention supplementary text semantic meta-modal auxiliary feature, feature screening is performed on the interaction intention gesture semantic encoding feature map to obtain the interaction intention multi-modal joint enhanced encoding feature, including: S1621, based on the masked first interaction intention supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector and the hyperbolic tangent function, perform preliminary feature screening on the interaction intention gesture semantic encoding feature map to obtain a preliminarily screened interaction intention gesture semantic encoding feature map; S1622, based on the masked second interaction intention supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector and the Sigmoid function, perform secondary feature screening on the preliminarily screened interaction intention gesture semantic encoding feature map to obtain an interaction intention multi-modal joint enhanced encoding feature map as the interaction intention multi-modal joint enhanced encoding feature.

[0051] More specifically, in the embodiment of the present application, based on the masked first interaction intention supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector and the hyperbolic tangent function, performing preliminary feature screening on the interaction intention gesture semantic encoding feature map to obtain a preliminarily screened interaction intention gesture semantic encoding feature map includes: multiplying the feature values at each position in the masked first interaction intention supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector with each feature matrix along the channel dimension of the interaction intention gesture semantic encoding feature map to obtain an interaction intention gesture semantic encoding weighted feature map; using the hyperbolic tangent function to perform feature screening on the interaction intention gesture semantic encoding weighted feature map to obtain the preliminarily screened interaction intention gesture semantic encoding feature map.

[0052] More specifically, in the embodiment of the present application, based on the masked second interaction intention supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector and the Sigmoid function, performing secondary feature screening on the preliminarily screened interaction intention gesture semantic encoding feature map to obtain an interaction intention multi-modal joint enhanced encoding feature map includes: multiplying the feature values at each position in the masked second interaction intention supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector with each feature matrix along the channel dimension of the preliminarily screened interaction intention gesture semantic encoding feature map to obtain a preliminarily screened interaction intention gesture semantic encoding weighted feature map; using the Sigmoid function to process the preliminarily screened interaction intention gesture semantic encoding weighted feature map to obtain the interaction intention multi-modal joint enhanced encoding feature map.

[0053] In the embodiments of the present application, specifically, text-video cross-modal joint enhancement encoding is performed on the semantic encoding feature vector of the supplementary text of the interaction intention and the semantic encoding feature map of the interaction intention gesture, which can be expressed by the formula:

[0054]

[0055] Where V meta is the semantic encoding feature vector of the supplementary text of the interaction intention, and are the first weight modifier matrix and the second weight modifier matrix respectively, w 0f and w 0g are the first modifier bias vector and the second modifier bias vector respectively, V1 and V2 are the representation vectors of the first supplementary text semantic meta-modal auxiliary feature mapping mode of the interaction intention and the representation vectors of the second supplementary text semantic meta-modal auxiliary feature mapping mode of the interaction intention, V1(i + 1) and V1(i - 1) are the feature values at the (i + 1)-th and (i - 1)-th positions in the representation vector of the first supplementary text semantic meta-modal auxiliary feature mapping mode of the interaction intention,.2(i + 1) and V2(i - 1) are the feature values at the (i + 1)-th and (i - 1)-th positions in the representation vector of the second supplementary text semantic meta-modal auxiliary feature mapping mode of the interaction intention, mask(·) is a masking operation, θ is a predetermined threshold, are the feature values at each position in the masked representation vector of the first supplementary text semantic meta-modal auxiliary feature mapping mode of the interaction intention, are the feature values at each position in the masked representation vector of the second supplementary text semantic meta-modal auxiliary feature mapping mode of the interaction intention, V 1s is the masked representation vector of the first supplementary text semantic meta-modal auxiliary feature mapping mode of the interaction intention, F main is the semantic encoding feature map of the interaction intention gesture, is to weight the feature map using the feature vector, tanh(·) is the hyperbolic tangent function, T gate is the semantic encoding feature map of the interaction intention gesture after preliminary screening, V 2s is the masked representation vector of the second supplementary text semantic meta-modal auxiliary feature mapping mode of the interaction intention, sigmoid(·) is the sigmoid function, S gate is the cross-modal joint enhancement encoding feature map of the interaction intention.

[0056] In steps S170 and S180, based on the multi-modal joint enhanced coding features of the interaction intention, an interaction intention recognition result is obtained, and based on the interaction intention recognition result, the interaction operation content of the power grid production command system is determined. Specifically, in the embodiment of the present application, obtaining the interaction intention recognition result based on the multi-modal joint enhanced coding features of the interaction intention includes: inputting the multi-modal joint enhanced coding feature map of the interaction intention into an interaction intention recognizer based on a classifier to obtain the interaction intention recognition result, and the interaction intention recognition result is used to represent the category label of the interaction intention. That is, the multi-modal joint enhanced coding features of the interaction intention obtained by cross-modal joint of the interaction intention supplementary text semantic coding feature vector and the interaction intention gesture semantic coding feature map are used for classification processing, so as to automatically evaluate the category label of the interaction intention and determine the interaction operation content of the power grid production command system. In this way, the response speed can be accelerated through simple and intuitive gestures. And through the dual confirmation mechanism of gesture and text input, the system can more accurately identify the user's intention, reduce the risk of misoperation caused by a single input method, and improve the intelligent level of power grid production command.

[0057] In the technical solution of the present application, the interaction intention supplementary text semantic coding feature vector and the interaction intention gesture semantic coding feature map respectively represent the interaction intention supplementary text semantic coding information and the interaction intention gesture action semantic information. When performing text-video cross-modal joint enhanced coding on them, due to the cross-modal feature information and the heterogeneity of feature expressions, the obtained multi-modal joint enhanced coding feature map of the interaction intention has cross-modal interaction sparsity, which affects the accuracy of the interaction intention recognition result obtained by inputting it into the interaction intention recognizer based on a classifier.

[0058] In a preferred embodiment, in the process of inputting the multi-modal joint enhanced coding feature map of the interaction intention into an interaction intention recognizer based on a classifier to obtain an interaction intention recognition result, feature manifold modulation is performed on the multi-modal joint enhanced coding feature map of the interaction intention, including:

[0059] Unfolding the multi-modal joint enhanced coding feature map of the interaction intention into a multi-modal joint enhanced coding feature vector;

[0060] Subtracting the sparsity base criterion from the feature dimension magnitude of the multi-modal joint enhanced coding feature vector of the interaction intention to obtain the independent difference quantity of the multi-modal joint enhanced coding of the interaction intention, expressed as:

[0061] n = L - ‖V‖0

[0062] Among them, L represents the feature dimension magnitude of the interactive intention multimodal joint enhanced coding feature vector, ‖V‖0 represents the sparsity base criterion of the interactive intention multimodal joint enhanced coding feature vector, and n represents the independent difference quantity of the interactive intention multimodal joint enhanced coding;

[0063] Calculate the quadratic norm operation of the independent difference quantity of the interactive intention multimodal joint enhanced coding and its own binary discrete entropy value to obtain the interactive intention multimodal joint enhanced coding hierarchical correlation coefficient, which is expressed as:

[0064] ∈ = log2(n 2 + n)

[0065] Among them, ∈ represents the interactive intention multimodal joint enhanced coding hierarchical correlation coefficient;

[0066] Construct an exponential morphism with the eigenvalue at each position of the interactive intention multimodal joint enhanced coding feature vector as the base and the independent difference quantity of the interactive intention multimodal joint enhanced coding minus one as the exponent, and perform an element-wise product with the interactive intention multimodal joint enhanced coding hierarchical correlation coefficient to generate the interactive intention multimodal joint enhanced coding guiding gradient, which is expressed as:

[0067] V1 = V ⊙(n-1) ⊙ ∈

[0068] Among them, V ⊙(n-1) represents constructing an exponential morphism with the eigenvalue at each position of the interactive intention multimodal joint enhanced coding feature vector as the base and the independent difference quantity of the interactive intention multimodal joint enhanced coding minus one as the exponent, ⊙ represents element-wise product, and V1 represents the interactive intention multimodal joint enhanced coding guiding gradient;

[0069] Perform an element-wise product of the interactive intention multimodal joint enhanced coding feature vector and the result of subtracting one from the independent difference quantity of the interactive intention multimodal joint enhanced coding, and then perform an inverse space calibration operation to obtain the interactive intention multimodal joint enhanced coding field constraint gradient, which is expressed as:

[0070] V2 = V ⊙ (n - 1) ⊙ n -1

[0071] Among them, V2 represents the interactive intention multimodal joint enhanced coding field constraint gradient;

[0072] Apply a natural exponential base transformation to the interactive intention multimodal joint enhanced coding field constraint gradient to form an exponential space mapping, and output the interactive intention multimodal joint enhanced coding field compensation gradient, which is expressed as:

[0073] V3 = exp(V2)

[0074] Among them, V3 represents the compensation gradient of the interactive intention multi-modal joint enhanced coding field;

[0075] Finally, through the tensor superposition of the interactive intention multi-modal joint enhanced coding guiding gradient and the compensation gradient of the interactive intention multi-modal joint enhanced coding field, an optimized interactive intention multi-modal joint enhanced coding feature vector is obtained, which is expressed as:

[0076]

[0077] Among them, represents tensor superposition, and V′ represents the optimized interactive intention multi-modal joint enhanced coding feature vector.

[0078] Correspondingly, for the multi-scale feature space topology of the interactive intention multi-modal joint enhanced coding feature vector obtained by expanding the interactive intention multi-modal joint enhanced coding feature map, a dynamic gradient field model is constructed based on the convergence dimension of the attribute parameters of the feature set. Through the extraction of the hierarchical correlation identification of the non-continuous cumulative quantity in the spatial domain of the interactive intention multi-modal joint enhanced coding feature vector, the parameter neighborhood anchoring is realized, and a gradient compensation excitation mechanism for the inversion field stability criterion is introduced to drive the regression distribution of the interactive intention multi-modal joint enhanced coding feature vector to meet the projection path locking requirement. Thereby, it prompts the feature set to perceive the spatial correlation conduction, and improves the fine-grained feature distribution balance degree of its cross-modal interaction by enhancing semantic coupling, so as to improve the accuracy of the interactive intention recognition result obtained by its input interactive intention recognizer based on the classifier.

[0079] In summary, the virtual interaction method for the power grid production command system based on the embodiments of the present application is clarified. It receives the user interaction gesture video through virtual technology and obtains the supplementary text of the interactive intention input by the user, and uses the data processing technology based on the deep neural network to perform semantic coding on the user interaction gesture video and semantic coding on the supplementary text of the interactive intention, so as to automatically evaluate the category label of the interactive intention based on the cross-modal joint enhanced coding representation between the semantic features of the user interaction gesture and the semantic features of the supplementary text of the interactive intention, and determine the interactive operation content of the power grid production command system. In this way, through the dual confirmation mechanism of gesture and text input, the user's intention can be more accurately recognized, thereby reducing the risk of misoperation and improving the intelligent level of the power grid production command system in training simulation applications.

[0080] Figure 6 It is a system block diagram of the virtual interaction system for the power grid production command system according to the embodiments of the present application. As Figure 6As shown in the figure, the virtual interaction system 100 for the power grid production command system according to the embodiments of the present application includes: a user interaction video data acquisition module 110, configured to acquire the user interaction gesture video collected by a camera; an interaction intention supplement module 120, configured to pop up an interaction intention text supplement box in response to receiving the user interaction gesture video; an interaction intention supplement text data receiving module 130, configured to receive the interaction intention supplement text input by the user in the interaction intention text supplement box; a user interaction video data semantic encoding module 140, configured to perform interaction intention gesture semantic encoding on the user interaction gesture video to obtain an interaction intention gesture semantic encoding feature map; an interaction intention supplement text semantic encoding module 150, configured to perform semantic encoding on the interaction intention supplement text to obtain an interaction intention supplement text semantic encoding feature vector; an interaction intention multi-modal joint enhancement encoding feature generation module 160, configured to perform text-video cross-modal joint enhancement encoding on the interaction intention supplement text semantic encoding feature vector and the interaction intention gesture semantic encoding feature map to obtain an interaction intention multi-modal joint enhancement encoding feature, including: an interaction intention semantic encoding feature mask processing unit, configured to perform linear transformation and gradient magnitude-based mask processing on the interaction intention supplement text semantic encoding feature vector to obtain a masked first interaction intention supplement text semantic meta-modal auxiliary feature and a masked second interaction intention supplement text semantic meta-modal auxiliary feature; an interaction intention gesture semantic encoding feature screening unit, configured to perform feature screening on the interaction intention gesture semantic encoding feature map based on the masked first interaction intention supplement text semantic meta-modal auxiliary feature and the masked second interaction intention supplement text semantic meta-modal auxiliary feature to obtain the interaction intention multi-modal joint enhancement encoding feature; an interaction intention recognition result generation module 170, configured to obtain an interaction intention recognition result based on the interaction intention multi-modal joint enhancement encoding feature; an interaction operation content determination module 180, configured to determine the interaction operation content of the power grid production command system based on the interaction intention recognition result.

[0081] Here, those skilled in the art can understand that the specific functions and operations of each unit and module in the above virtual interaction system 100 for the power grid production command system have been introduced in detail in the description of the virtual interaction method for the power grid production command system above with reference to Figures 1 to 5 and thus, the repeated description thereof will be omitted.

[0082] In summary, the virtual interaction system 100 for the power grid production command system based on the embodiments of the present application is elucidated. It receives the user interaction gesture video through virtual technology and obtains the supplementary text of the interaction intention input by the user, and uses the data processing technology based on the deep neural network to semantically encode the user interaction gesture video and the supplementary text of the interaction intention, so as to automatically evaluate the category label of the interaction intention based on the cross-modal joint enhanced encoding representation between the semantic features of the user interaction gesture and the semantic features of the supplementary text of the interaction intention, and determine the interaction operation content of the power grid production command system. In this way, through the dual confirmation mechanism of gesture and text input, the user's intention can be recognized more accurately, thereby reducing the risk of misoperation and improving the intelligent level of the power grid production command system in training simulation applications.

[0083] As described above, the virtual interaction system 100 for the power grid production command system according to the embodiments of the present application can be implemented in various wireless terminals, such as servers for virtual interaction for the power grid production command system. In one example, the virtual interaction system 100 for the power grid production command system according to the embodiments of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the virtual interaction system 100 for the power grid production command system can be a software module in the operating system of the wireless terminal, or can be an application program developed for the wireless terminal; of course, the virtual interaction system 100 for the power grid production command system can also be one of the many hardware modules of the wireless terminal.

[0084] Alternatively, in another example, the virtual interaction system 100 for the power grid production command system and the wireless terminal can also be separate devices, and the virtual interaction system 100 for the power grid production command system can be connected to the wireless terminal through a wired and / or wireless network and transmit interaction information in accordance with a predefined data format.

[0085] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, advantages, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, advantages, effects, etc. are essential for each embodiment of the present application. In addition, the specific details of the above application are only for the purpose of illustration and easy understanding, rather than limitations. The above details do not limit the present application to necessarily adopt the above specific details for implementation.

Claims

1. An intelligent interactive method for power grid production case deduction, characterized in that: include: Get the user interaction gesture video captured by the camera; In response to receiving the user interaction gesture video, popping up an interaction intention text supplement box; receiving the interaction intention supplementary text input by the user in the interaction intention text supplementary box; Performing interaction intention gesture semantic coding on the user interaction gesture video to obtain an interaction intention gesture semantic coding feature map; Performing semantic encoding on the interaction intention supplementary text to obtain a semantic encoding feature vector of the interaction intention supplementary text; The interaction intention supplementary text semantic coding feature vector and the interaction intention gesture semantic coding feature map are subjected to text-video cross-modal joint enhancement coding to obtain interaction intention multimodal joint enhancement coding features, including: linear transformation and gradient amplitude-based masking processing on the interaction intention supplementary text semantic coding feature vector to obtain masked first interaction intention supplementary text semantic metamodal auxiliary features and masked second interaction intention supplementary text semantic metamodal auxiliary features; based on the masked first interaction intention supplementary text semantic metamodal auxiliary features and the masked second interaction intention supplementary text semantic metamodal auxiliary features, feature screening is performed on the interaction intention gesture semantic coding feature map to obtain the interaction intention multimodal joint enhancement coding features; Based on the multimodal joint enhanced coding features of the interaction intention, an interaction intention recognition result is obtained; Based on the interaction intention recognition result, the interaction operation content of the power grid production command system is determined.

2. The intelligent interactive method for power grid production case deduction according to claim 1 is characterized in that: The user interaction gesture video is semantically encoded with interaction intention gesture to obtain an interaction intention gesture semantic encoding feature map, including: inputting the user interaction gesture video into an interaction intention gesture semantic encoder based on a three-dimensional convolutional neural network model to obtain the interaction intention gesture semantic encoding feature map.

3. The intelligent interactive method for power grid production case deduction according to claim 2 is characterized in that: The interaction intention supplementary text semantic encoding feature vector is subjected to linear transformation and gradient amplitude-based masking processing to obtain a masked first interaction intention supplementary text semantic metamodal auxiliary feature and a masked second interaction intention supplementary text semantic metamodal auxiliary feature, including: Performing a linear transformation on the interaction intention supplementary text semantic encoding feature vector based on the first weight modifier matrix and the second weight modifier matrix to obtain a first interaction intention supplementary text semantic metamodal auxiliary feature mapping mode representation vector and a second interaction intention supplementary text semantic metamodal auxiliary feature mapping mode representation vector; Based on the gradient amplitude of each position in the first interaction intention supplementary text semantic metamodal auxiliary feature mapping pattern representation vector, the first interaction intention supplementary text semantic metamodal auxiliary feature mapping pattern representation vector is subjected to gradient amplitude-based masking processing to obtain a masked first interaction intention supplementary text semantic metamodal auxiliary feature mapping pattern representation vector as the masked first interaction intention supplementary text semantic metamodal auxiliary feature; Based on the gradient amplitude of each position in the second interactive intention supplementary text semantic metamodal auxiliary feature mapping pattern representation vector, the second interactive intention supplementary text semantic metamodal auxiliary feature mapping pattern representation vector is masked based on the gradient amplitude to obtain a masked second interactive intention supplementary text semantic metamodal auxiliary feature mapping pattern representation vector as the masked second interactive intention supplementary text semantic metamodal auxiliary feature.

4. The intelligent interactive method for power grid production case deduction according to claim 3 is characterized in that: The interaction intention supplementary text semantic encoding feature vector is subjected to a linear transformation based on a first weight modifier matrix and a second weight modifier matrix to obtain a first interaction intention supplementary text semantic metamodal auxiliary feature mapping mode representation vector and a second interaction intention supplementary text semantic metamodal auxiliary feature mapping mode representation vector, including: Multiplying the interaction intention supplementary text semantic encoding feature vector by the first weight modifier matrix, and adding the obtained first interaction intention supplementary text semantic modification vector and the first modifier bias vector by position to obtain the first interaction intention supplementary text semantic meta-modality auxiliary feature mapping mode representation vector; Multiply the interaction intention supplementary text semantic encoding feature vector with the second weight modifier matrix, and add the obtained second interaction intention supplementary text semantic modification vector and the second modifier bias vector by position to obtain the second interaction intention supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector.

5. The intelligent interactive method for power grid production case deduction according to claim 4 is characterized in that: Based on the gradient amplitude of each position in the first interaction intention supplementary text semantic metamodal auxiliary feature mapping mode representation vector, the first interaction intention supplementary text semantic metamodal auxiliary feature mapping mode representation vector is subjected to gradient amplitude-based masking processing to obtain a masked first interaction intention supplementary text semantic metamodal auxiliary feature mapping mode representation vector, including: Calculate the square of the difference between the eigenvalues ​​at the i+1th position and the i-1th position in the semantic meta-modal auxiliary feature mapping mode representation vector of the first interaction intention supplementary text to obtain the semantic gradient amplitude representation vector of the first interaction intention supplementary text; Calculating the absolute value of the square root of each eigenvalue in the semantic gradient amplitude representation vector of the first interaction intention supplementary text to obtain the semantic gradient amplitude modulation representation vector of the first interaction intention supplementary text; The first interactive intention supplementary text semantic gradient amplitude modulation representation vector is masked, and in response to the eigenvalue of each position in the first interactive intention supplementary text semantic gradient amplitude modulation representation vector being greater than a predetermined threshold, the eigenvalue of the position is set to 1, and the rest are set to 0, to obtain the masked first interactive intention supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector.

6. The intelligent interactive method for power grid production case deduction according to claim 5 is characterized in that: Based on the masked first interaction intention supplementary text semantic metamodal auxiliary feature and the masked second interaction intention supplementary text semantic metamodal auxiliary feature, feature screening is performed on the interaction intention gesture semantic encoding feature map to obtain the interaction intention multimodal joint enhanced encoding feature, including: Based on the masked first interaction intention supplementary text semantic metamodal auxiliary feature mapping mode representation vector and the hyperbolic tangent function, preliminary feature screening is performed on the interaction intention gesture semantic encoding feature map to obtain a preliminary screened interaction intention gesture semantic encoding feature map; Based on the masked second interaction intention supplementary text semantic meta-modal auxiliary feature mapping pattern representation vector and Sigmoid function, the interaction intention gesture semantic encoding feature map after preliminary screening is subjected to further feature screening to obtain the interaction intention multimodal joint enhanced encoding feature map as the interaction intention multimodal joint enhanced encoding feature.

7. The intelligent interactive method for power grid production case deduction according to claim 6 is characterized in that: Based on the masked first interaction intention supplementary text semantic metamodal auxiliary feature mapping mode representation vector and the hyperbolic tangent function, preliminary feature screening is performed on the interaction intention gesture semantic encoding feature map to obtain a preliminary screened interaction intention gesture semantic encoding feature map, including: Multiplying the feature values ​​of each position in the masked first interaction intention supplementary text semantic metamodal auxiliary feature mapping mode representation vector by each feature matrix along the channel dimension of the interaction intention gesture semantic encoding feature map by position to obtain an interaction intention gesture semantic encoding weighted feature map; The hyperbolic tangent function is used to perform feature screening on the interaction intention gesture semantic encoding weighted feature map to obtain the interaction intention gesture semantic encoding feature map after preliminary screening.

8. The intelligent interactive method for power grid production case deduction according to claim 7 is characterized in that: The interactive intention gesture semantic encoding feature map after the preliminary screening is subjected to feature screening again based on the masked second interactive intention supplementary text semantic metamodal auxiliary feature mapping mode representation vector and the Sigmoid function to obtain the interactive intention multimodal joint enhanced encoding feature map, including: Multiplying the feature values ​​of each position in the masked second interaction intention supplementary text semantic metamodal auxiliary feature mapping mode representation vector by each feature matrix along the channel dimension of the interaction intention gesture semantic encoding feature map after preliminary screening by position to obtain a weighted feature map of interaction intention gesture semantic encoding after preliminary screening; The Sigmoid function is used to process the interaction intention gesture semantic encoding weighted feature map after the preliminary screening to obtain the interaction intention multimodal joint enhanced encoding feature map.

9. The intelligent interactive method for power grid production case deduction according to claim 8 is characterized in that: Based on the multimodal joint enhanced coding features of the interaction intention, an interaction intention recognition result is obtained, including: inputting the multimodal joint enhanced coding feature map of the interaction intention into a classifier-based interaction intention identifier to obtain the interaction intention recognition result, and the interaction intention recognition result is used to represent the category label of the interaction intention.

10. An intelligent interactive system for power grid production case deduction, characterized in that: include: A user interaction video data acquisition module is used to obtain the user interaction gesture video collected by the camera; An interaction intention supplement module, configured to pop up an interaction intention text supplement box in response to receiving the user interaction gesture video; An interaction intention supplementary text data receiving module, used for receiving the interaction intention supplementary text input by the user in the interaction intention text supplementary box; A user interaction video data semantic encoding module, used for performing interaction intention gesture semantic encoding on the user interaction gesture video to obtain an interaction intention gesture semantic encoding feature map; An interaction intention supplementary text semantic encoding module, used for semantically encoding the interaction intention supplementary text to obtain an interaction intention supplementary text semantic encoding feature vector; An interactive intention multimodal joint enhanced coding feature generation module, used for performing text-video cross-modal joint enhanced coding on the interactive intention supplementary text semantic coding feature vector and the interactive intention gesture semantic coding feature map to obtain interactive intention multimodal joint enhanced coding features, including: an interactive intention semantic coding feature mask processing unit, used for performing linear transformation and gradient amplitude-based mask processing on the interactive intention supplementary text semantic coding feature vector to obtain masked first interactive intention supplementary text semantic metamodal auxiliary features and masked second interactive intention supplementary text semantic metamodal auxiliary features; an interactive intention gesture semantic coding feature screening unit, used for performing feature screening on the interactive intention gesture semantic coding feature map based on the masked first interactive intention supplementary text semantic metamodal auxiliary features and the masked second interactive intention supplementary text semantic metamodal auxiliary features to obtain the interactive intention multimodal joint enhanced coding features; An interaction intention recognition result generation module is used to obtain an interaction intention recognition result based on the interaction intention multimodal joint enhanced coding feature; The interactive operation content determination module is used to determine the interactive operation content of the power grid production command system based on the interactive intention recognition result.

Images