Anti-misoperation locking system and method based on intelligent identification

Abstract

The invention discloses an anti-misoperation locking system and method based on intelligent recognition, and relates to the technical field of industrial automation, and the method comprises the steps: collecting and preprocessing multi-modal data, and generating a multi-modal data set, the multi-modal data comprising voice data, visual data and equipment state data; based on the multi-modal data set, multi-modal features are extracted through a multi-modal feature coding method, and the multi-modal features generate a multi-modal feature cube through a three-dimensional feature modeling method; inputting the multi-modal feature cube into a double-flow GAN network, generating an adversarial sample, constructing a defense model, and training through the adversarial sample; and inputting the multi-modal feature cube into a defense model, generating a probability distribution vector and an instruction confidence coefficient, performing false touch identification according to the instruction confidence coefficient, and dynamically generating a locking instruction according to an identification result. According to the method, the generalization ability of the defense model and the resistance ability to unknown threats are improved through the antagonism training and mistaken touch recognition mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial automation, and in particular to an anti-misoperation locking system and method based on intelligent identification. Background Art

[0002] In the power grid and industrial automation sectors, anti-error lockout technology is a key technology for ensuring stable operation of power equipment, preventing electrical errors, and protecting the personal safety of operators. Errors can cause equipment damage, grid failures, and even major safety incidents. This is especially true in critical scenarios such as high-voltage transmission and distribution, substation switching operations, and factory automation production line control. Smarter, more precise technologies are needed to improve error prevention capabilities.

[0003] However, the accuracy of intelligent recognition technology's operational commands is affected by a variety of factors, including ambient noise, the operator's accent, and speaking speed, leading to reduced accuracy in complex environments. Furthermore, the threat of adversarial attacks is a pressing issue. Adversarial attacks introduce subtle perturbations to deceive the recognition mechanism, causing it to produce erroneous recognition results. This poses a serious threat to the security of preventing misoperations. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an anti-misoperation locking method based on intelligent recognition to solve the problems of poor recognition accuracy and adversarial attack threats.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides an anti-misoperation locking method based on intelligent recognition, which includes collecting multimodal data and preprocessing it to generate a multimodal data set, wherein the multimodal data includes voice data, visual data and device status data; based on the multimodal data set, multimodal features are extracted by a multimodal feature encoding method, and the multimodal features are generated into a multimodal feature cube by a three-dimensional feature modeling method; the multimodal feature cube is input into a two-stream GAN network to generate adversarial samples, a defense model is constructed, and training is performed using the adversarial samples; the multimodal feature cube is input into the defense model to generate a probability distribution vector and an instruction confidence, and false touch recognition is performed according to the instruction confidence, and a locking instruction is dynamically generated according to the recognition result.

[0008] As a preferred solution of the anti-misoperation locking method based on intelligent identification of the present invention, wherein:

[0009] The multimodal data is collected and preprocessed to generate a multimodal data set. The specific steps are as follows:

[0010] The WaveNet-based sound source separation method is used to reduce noise in speech data, the ESRGAN-based super-resolution reconstruction method is used to deblur visual data, and the wavelet transform method is used to reduce noise in device status data.

[0011] Based on the preprocessed multimodal data, a multimodal dataset is generated through a multimodal data integration method.

[0012] As a preferred solution of the anti-misoperation locking method based on intelligent identification of the present invention, wherein:

[0013] Based on the multimodal dataset, multimodal features are extracted through a multimodal feature encoding method, and the multimodal features are generated into a multimodal feature cube through a three-dimensional feature modeling method. The specific steps are as follows:

[0014] Use multimodal feature encoding methods to encode speech data, visual data, and device status data in multimodal datasets;

[0015] The encoded voice data, visual data, and device status data are concatenated and fused in the feature dimension through a joint encoding method, and mapped to a fixed dimension to generate multimodal features;

[0016] The multimodal features are temporally and spatially aligned based on the timestamp synchronization interpolation method and the RTK differential positioning method;

[0017] Through the spatiotemporal coordinate mapping method, the spatiotemporally aligned multimodal features are mapped to the corresponding positions of the three-dimensional cube according to the time axis resolution and the space axis resolution to generate a multimodal feature cube.

[0018] As a preferred solution of the anti-misoperation locking method based on intelligent identification of the present invention, wherein:

[0019] The multimodal feature cube is input into the two-stream GAN network to generate adversarial samples. The specific steps are as follows:

[0020] Extract temporal and spatial features from the multimodal feature cube through the generator in the two-stream GAN network;

[0021] The temporal features and spatial features are modally concatenated and input into a multi-layer perceptron to generate the initial features of the adversarial sample.

[0022] Use the deconvolution network to upsample the initial features of the adversarial sample to generate a complete adversarial sample.

[0023] As a preferred solution of the anti-misoperation locking method based on intelligent identification of the present invention, wherein:

[0024] The defense model is constructed and trained through adversarial samples. The specific steps are as follows:

[0025] The input layer is defined based on the characteristics of the multimodal feature cube, and the temporal feature extraction layer is defined based on the Transformer architecture;

[0026] The spatial feature extraction layer is defined based on a 3D convolutional neural network, the feature fusion layer is defined based on a fully connected network, and the output layer is defined based on a classifier and a multi-head self-attention mechanism;

[0027] Construct a defense model through the input layer, temporal feature extraction layer, spatial feature extraction layer, feature fusion layer and output layer;

[0028] Using cross entropy loss as the loss function, Softmax as the activation function, and Adam as the optimizer, the complete adversarial sample is input into the defense model for training.

[0029] As a preferred solution of the anti-misoperation locking method based on intelligent identification of the present invention, wherein:

[0030] The multimodal feature cube is input into the defense model to generate the probability distribution vector and instruction confidence. The specific steps are as follows:

[0031] The time feature extraction layer and space feature extraction layer of the defense model extract the time axis features and space axis features of the multimodal feature cube respectively;

[0032] Through the feature fusion layer, the time axis features and the space axis features are fused to generate fused features, and the fused features are calculated through the multi-head self-attention mechanism to generate weighted fused features;

[0033] The Softmax activation function is used to classify the weighted fusion features, generate a probability distribution vector, and analyze the instruction confidence based on the probability distribution vector.

[0034] As a preferred solution of the anti-misoperation locking method based on intelligent identification of the present invention, wherein:

[0035] The false touch recognition is performed according to the command confidence, and the locking command is dynamically generated according to the recognition result. The specific steps are as follows:

[0036] Use the confusion matrix method to set a confidence threshold and evaluate whether the instruction confidence exceeds the confidence threshold;

[0037] According to the evaluation results, the false touch identification rules and the lock instruction generation rules are set and the lock instruction is executed.

[0038] In the second aspect, the present invention provides an anti-misoperation locking system based on intelligent recognition, including a data acquisition module, a feature extraction module, a model construction module and an instruction generation module; the data acquisition module is used to collect multimodal data and preprocess it to generate a multimodal data set, wherein the multimodal data includes voice data, visual data and device status data; the feature extraction module is used to extract multimodal features based on the multimodal data set through a multimodal feature encoding method, and the multimodal features are used to generate a multimodal feature cube through a three-dimensional feature modeling method; the model construction module is used to input the multimodal feature cube into the dual-stream GAN network, generate adversarial samples, build a defense model, and train with adversarial samples; the instruction generation module is used to input the multimodal feature cube into the defense model, generate a probability distribution vector and instruction confidence, perform mistouch recognition according to the instruction confidence, and dynamically generate a locking instruction according to the recognition result.

[0039] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the anti-misoperation locking method based on intelligent identification as described in the first aspect of the present invention.

[0040] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the anti-misoperation locking method based on intelligent identification as described in the first aspect of the present invention.

[0041] The beneficial effects of this invention are as follows: through multimodal data fusion, it overcomes the problem of reduced recognition accuracy caused by environmental factors in complex scenarios, significantly improving the accuracy and robustness of command recognition, and enhancing adaptability and recognition accuracy in complex environments. At the same time, through adversarial training and false touch recognition mechanisms, the defense model's generalization ability and resistance to unknown threats are improved, and its anti-interference ability is enhanced. It can still maintain high accuracy in the face of noise interference or abnormal input, ensuring the safety and reliability of operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 This is a flow chart of the anti-misoperation locking method based on intelligent identification.

[0044] Figure 2 Schematic diagram of the anti-misoperation locking system based on intelligent recognition.

[0045] Figure 3 Flowchart of data collection and preprocessing.

[0046] Figure 4 A flowchart constructed for the defense model. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0048] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0049] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0050] Reference Figures 1 to 4 , is an embodiment of the present invention, which provides an anti-misoperation locking method based on intelligent identification, comprising the following steps:

[0051] S1. Collect multimodal data and perform preprocessing to generate a multimodal dataset. The multimodal data includes voice data, visual data, and device status data.

[0052] The WaveNet-based sound source separation method is used to reduce noise in speech data, the ESRGAN-based super-resolution reconstruction method is used to deblur visual data, and the wavelet transform method is used to reduce noise in device status data.

[0053] It should be noted that WaveNet uses deep neural networks to analyze speech signals, identify and separate useful components in speech, and remove background noise; ESRGAN uses generative adversarial networks to enhance low-resolution images and restore image details; the wavelet transform method decomposes the signal into sub-signals of different frequencies, identifies and removes noise components, and can effectively process high-frequency noise in device status data.

[0054] Based on the preprocessed multimodal data, a multimodal dataset is generated through a multimodal data integration method;

[0055] It should be noted that the preprocessed speech data is standardized using Mel-frequency cepstral coefficients and stored as a two-dimensional floating-point array. The visual data is stored as an RGB image sequence of a three-dimensional integer array based on the input requirements of the convolutional neural network (CNN). The device status data is stored as a two-dimensional floating-point array in a time series format. Then, a multimodal data structure containing speech data fields, visual data fields, and device status data fields is defined through a class object. The standardized speech data, visual data, and device status data are stored according to the defined multimodal data structure. Multiple instances storing data of different modalities are combined into a set to form a multimodal data set.

[0056] S2. Based on the multimodal dataset, multimodal features are extracted through the multimodal feature encoding method, and the multimodal features are generated into a multimodal feature cube through the three-dimensional feature modeling method.

[0057] Use multimodal feature encoding methods to encode speech data, visual data, and device status data in multimodal datasets;

[0058] It should be noted that in the process of multimodal data encoding, it usually refers to converting the original data into a format suitable for processing by machine learning models; linear predictive coding (LPC) can be used to represent the voice features of speech data, and the speech data is pre-emphasized. A first-order FIR filter with a coefficient of 0.95 or 0.97 is usually used. The pre-emphasized speech data is divided into short time frames, usually with a frame length of 20-40ms and a frame shift of 10ms. A Hamming window function is applied to each frame of speech data to reduce spectral leakage, and the linear prediction coefficient (LPC) of each frame of speech data is calculated. The Yule-Walker equation is usually solved to obtain the LPC coefficient, and the LPC coefficient is extracted as the acoustic feature of the speech data. Usually the first 10-16 order coefficients are taken; pixel-level encoding (such as RGB values) can be used. To represent the characteristics of visual data, use an image processing library (such as OpenCV) to read visual data. If the image is not in RGB format, convert the image to RGB format through color space conversion to ensure that each pixel contains the values ​​of the three channels of red, green, and blue. For each pixel, extract the RGB value and arrange the RGB values ​​of all pixels of the image in row priority order to form a one-dimensional feature vector; the device status data is usually time series data, which is encoded using the time series encoding method. The time series data is divided into frames of fixed length. Usually, each frame contains several sampling points (such as 1024 points). The frame shift can be set as needed (such as 512 points). Perform fast Fourier transform (FFT) on each frame of time series data to convert the time domain signal into a frequency domain signal, and calculate the amplitude spectrum and power spectrum of the spectrum.

[0059] The encoded voice data, visual data, and device status data are concatenated and fused in the feature dimension through a joint encoding method, and mapped to a fixed dimension to generate multimodal features;

[0060] It should be noted that feature extraction is performed on the encoded speech data, visual data and device status data. For example, speech feature vectors are extracted from MFCC coefficients, which are usually 12-13 dimensions. Pixel values ​​are extracted from RGB images to form a three-dimensional array. Amplitude spectrum features, such as spectrum peak, spectrum center of gravity, and spectrum bandwidth, are extracted from the spectrum obtained by FFT transformation. Ensure that the feature vectors of each modality have the same dimension. If the feature dimensions are different, they can be adjusted through dimensionality reduction or padding operations. Speech features, visual features and device status features are spliced ​​in the feature dimension, and the spliced ​​feature vectors are mapped to a fixed dimension using a fully connected layer. During the mapping process, activation functions can be used to enhance the expressiveness of multimodal features. The mapped feature vectors are multimodal features.

[0061] The multimodal features are temporally and spatially aligned based on the timestamp synchronization interpolation method and the RTK differential positioning method;

[0062] It should be noted that for speech data and visual data, we first need to obtain their respective timestamp information, find the common time point in the two timestamp sequences or determine a reference time point. We can choose the earliest timestamp in the two sequences as the reference time point, calculate the offset of each timestamp relative to the reference time point, and perform time correction based on the offset to ensure that the speech data and visual data are synchronized in time during the time alignment process. Then, we calculate the speech data at T according to the linear interpolation principle. vk The value at the moment, through such an interpolation operation, the timestamp of the voice data can be aligned with the timestamp of the visual data, thereby achieving the alignment of the voice data and the visual data in the time dimension. The expression is,

[0063]

[0064] Among them, x sk Represents the feature vector of speech data s at the kth time stamp of visual data; x si Represents the feature vector at time stamp i in the speech data s; T vk represents the timestamp of the kth moment in the visual data v; T si Represents the timestamp of the i-th moment in the speech data s, which is the same as T vk Adjacent and earlier timestamp; T s(i+1) represents the time stamp in the speech data s that is adjacent to and later than time i; x s(i+1) Indicates that the timestamp in the voice data s is T s(i+1)The feature vector at the moment; s represents speech data; i represents an index in the speech data timestamp sequence, used to identify a specific timestamp position; v represents visual data; k represents an index in the visual data timestamp sequence, used to identify a specific timestamp position;

[0065] The RTK differential positioning method improves positioning accuracy by calculating the difference between the base station and the rover. If the timestamp in the device status data and the timestamp of the differential correction number are inconsistent, the timestamp synchronization interpolation method is used. Based on the change pattern of the differential correction number over time, the differential correction number corresponding to the timestamp T of the device status data is calculated to obtain more accurate, time-aligned device position coordinates.

[0066] Through the spatiotemporal coordinate mapping method, the multimodal features after spatiotemporal alignment are mapped to the corresponding positions of the three-dimensional cube according to the time axis resolution and the spatial axis resolution to generate a multimodal feature cube;

[0067] It should be noted that for each multimodal feature data point after spatiotemporal alignment, the corresponding timestamp is obtained, and the corresponding coordinate position of the data point on the time axis is calculated according to the time axis resolution; the spatial coordinates of each pixel point are obtained, and the corresponding coordinate position of the corresponding pixel point on the spatial axis is calculated according to the spatial axis resolution, and the time axis coordinate position and the spatial axis coordinate position are combined to determine the position of the multimodal feature in the three-dimensional cube; according to the range of the time axis and the spatial axis and the determined resolution, a three-dimensional array is created to represent the multimodal feature cube, and all elements of the three-dimensional array are initialized to default values. For each multimodal feature data point after spatiotemporal alignment, the characteristic value of the data point is filled into the corresponding position of the three-dimensional cube according to the position mapped to the three-dimensional cube, and finally a multimodal feature cube is generated.

[0068] S3. Input the multimodal feature cube into the two-stream GAN network to generate adversarial samples, build a defense model, and train it with adversarial samples.

[0069] Extract temporal and spatial features from the multimodal feature cube through the generator in the two-stream GAN network;

[0070] It should be noted that the generator consists of multiple layers, including convolutional layers, pooling layers, deconvolutional layers, fully connected layers, etc. The multimodal feature cube first enters the convolutional layer in the generator. The convolutional layer performs convolution operations by sliding a set of learnable convolution kernels on the multimodal feature cube. The size of the convolution kernel can be selected according to the characteristics of the time dimension and the space dimension. The convolution kernel slides on the time dimension and the space dimension of the multimodal feature cube, and performs convolution calculations on the features in each local time window and space window. By operating multiple such convolution kernels in parallel, multiple different time feature maps and spatial feature maps can be obtained. Then, the time feature maps and spatial feature maps are reduced in dimensionality through the pooling layer. After multiple convolution and pooling operations, the time feature maps and spatial feature maps obtained at different levels are spliced ​​and fused to obtain time features and spatial features.

[0071] The temporal features and spatial features are modally concatenated and input into a multi-layer perceptron to generate the initial features of the adversarial sample.

[0072] It should be noted that modal cascade refers to the splicing of features of two different modalities (time and space) in a specific dimension, usually in the channel dimension, and all information of the time feature in the channel dimension and all information of the spatial feature in the channel dimension are arranged in sequence to form a new feature representation that combines the time feature and the space feature; the multi-layer perceptron is composed of multiple fully connected layers, including input layer, hidden layer and output layer; the feature data after modal cascade is input into the multi-layer perceptron, and forward propagation calculation is performed through the hidden layer in sequence. The neurons in each layer will process the input feature data. This processing process includes weighted summation of the input feature data and nonlinear transformation through the activation function to finally generate the initial features of the adversarial sample.

[0073] Use the deconvolution network to upsample the initial features of the adversarial sample to generate a complete adversarial sample;

[0074] It should be noted that the number of layers of the deconvolution network is determined according to the dimension of the initial features of the adversarial sample. For each deconvolution layer, parameters such as the convolution kernel size, step size, and padding method need to be determined. The convolution kernel size determines the size of the input area involved in each convolution operation, the step size determines the stride of the convolution kernel moving on the input data, and the padding method is used to control the size of the output data. The initial features of the adversarial sample are fed into the first layer of the deconvolution network as input data, and a deconvolution operation is performed on the initial features of the adversarial sample to increase the spatial dimension of the initial features of the adversarial sample. After processing by one deconvolution layer, the output data obtained will be used as the input data of the next deconvolution layer, and the deconvolution operation will continue until the predetermined number of deconvolution layers is reached. Each layer of deconvolution operation will gradually increase the spatial dimension of the initial features of the adversarial sample, so that the initial features of the adversarial sample gradually recover to a size and feature representation close to that of the complete adversarial sample. After processing by all deconvolution layers, the final output data obtained is the complete adversarial sample.

[0075] The input layer is defined based on the characteristics of the multimodal feature cube, and the temporal feature extraction layer is defined based on the Transformer architecture;

[0076] It should be noted that to determine the dimensional information of the multimodal feature cube, including the length of the time dimension, the size of the spatial dimension, and the number of feature channels, the input layer needs to receive a multidimensional tensor as input, and the dimension of the tensor corresponds to the dimension of the multimodal feature cube; in the time feature extraction layer, the self-attention mechanism of the Transformer architecture is used to extract the time features in the multimodal feature cube, and the multimodal feature cube is expanded into a sequence in the time dimension. This sequence is then passed as input to the Transformer architecture to determine the key parameters of the Transformer architecture, such as the number of heads, hidden layer dimension, and feedforward neural network dimension, etc., to construct the encoder of the Transformer architecture, and use the encoder part to extract the time features.

[0077] The spatial feature extraction layer is defined based on a 3D convolutional neural network, the feature fusion layer is defined based on a fully connected network, and the output layer is defined based on a classifier and a multi-head self-attention mechanism;

[0078] It should be noted that, first determine the structural parameters of the 3D convolutional neural network, select the appropriate convolution kernel size, the convolution kernel size determines the range covered by each convolution operation in the spatial dimension; determine the step size of the convolution operation, the step size determines the distance the convolution kernel moves in the spatial dimension; select the appropriate padding method, such as "same" padding or "valid" padding, "same" padding will add appropriate zero values ​​to the edge of the input data, so that the output size after the convolution operation is the same as the input size; "valid" padding does not perform padding, and the output size after the convolution operation will be reduced accordingly according to the convolution kernel size and step size; determine the number of input and output channels, the number of input channels corresponds to the number of feature channels of the input data; construct the 3D convolution layer according to the determined structural parameters; determine the input and output dimensions of the fully connected network, the input dimension should be based on the feature dimension output by the spatial feature extraction layer The temporal feature dimension output by the spatial feature extraction layer and the temporal feature extraction layer is determined, and the output dimension is determined according to the task requirements. According to the determined input dimension and output dimension, a fully connected layer is constructed, and the spatial features output by the spatial feature extraction layer and the temporal features output by the temporal feature extraction layer are input into the fully connected network for feature splicing and fusion; a suitable classifier is selected according to the task type. Common classifiers include Softmax classifier and Sigmoid classifier, etc. The parameters of the multi-head self-attention mechanism, such as the number of heads and hidden layer dimension, are determined. The number of heads determines the number of parallel calculations of the attention mechanism, and the hidden layer dimension determines the dimension size of each head. The fused features output by the feature fusion layer are passed as input to the multi-head self-attention mechanism layer to obtain the fused feature representation after attention weighting, and the weighted fused features output by the multi-head self-attention mechanism layer are input into the classifier for classification.

[0079] Construct a defense model through the input layer, temporal feature extraction layer, spatial feature extraction layer, feature fusion layer and output layer;

[0080] Use cross entropy loss as the loss function, Softmax as the activation function, and Adam as the optimizer to input the complete adversarial sample into the defense model for training;

[0081] It should be noted that the complete adversarial sample is divided into a training set, a validation set, and a test set. Before starting training, all parameters of the defense model need to be initialized, and the number of training rounds is set. The training set is divided according to the batch size, for example, the batch size is set to 32. A batch of selected adversarial samples is input into the defense model, and the probability distribution vectors of each category are generated through the output layer through forward propagation. The parameters of the defense model are optimized by minimizing the cross entropy loss, and the gradient of the loss to the parameters of each layer of the defense model is calculated by the back propagation algorithm. The back propagation starts from the output layer and calculates the gradient of each layer in sequence until the gradient of the relevant parameters of the input layer is calculated. The parameters of the defense model are updated according to the calculated gradient using the Adam optimizer. The Adam optimizer adaptively adjusts the learning rate based on the historical gradient information of each parameter, and updates the weights, biases and other parameters of the defense model according to certain update rules. After reaching the maximum number of training rounds, the validation set is input into the defense model, and the loss value of the defense model on the validation set is calculated. If the performance of the defense model on the validation set no longer improves or begins to decline, it is necessary to adjust the hyperparameters of the defense model (such as learning rate, batch size, etc.). Finally, the test set is input into the defense model to obtain the prediction results of the defense model, and the evaluation indicators of the defense model on the test set, such as accuracy, are calculated. The evaluation indicators are used to measure the performance of the defense model in defending against complete adversarial samples and to evaluate whether the defense model has achieved the expected effect. The expression is,

[0082]

[0083] Where A represents the ratio of the number of complete adversarial samples correctly predicted by the defense model to the total number of complete adversarial samples; n represents the total number of complete adversarial samples participating in the defense model evaluation, and y 1l represents the true label of the lth complete adversarial sample; y 2l represents the defense model prediction label of the lth complete adversarial sample; I(y 1l =y 2l ) is an indicator function, when y 1l =y 2l When y 1l ≠y 2l When , it means that the predicted label of the defense model is not equal to the true label, I = 0; l represents the index variable of the sample, the value range is 1-n, y1 represents the true label, and y2 represents the predicted label;

[0084] It should also be noted that when constructing complete adversarial samples for defense model training, it is necessary to formulate detailed labeling criteria based on the specific task. According to the formulated labeling criteria, the collected adversarial samples are labeled to complete the labeling of the true labels. For example, in image classification, if the image is a picture of a cat, then its true label is the category "cat"; in network security, if the network traffic is normal network access traffic, then the true label may be "normal", if it is malicious attack traffic, the true label may be the corresponding attack type (such as "DDoS attack", etc.).

[0085] S4. Input the multimodal feature cube into the defense model to generate a probability distribution vector and command confidence, perform false touch recognition based on the command confidence, and dynamically generate a locking command based on the recognition result.

[0086] The time feature extraction layer and space feature extraction layer of the defense model extract the time axis features and space axis features of the multimodal feature cube respectively;

[0087] It should be noted that the input multimodal feature cube is divided into time windows of fixed length in the time dimension. Each time window contains multiple continuous time step data. A one-dimensional convolution operation is applied to the time feature data in each time window. Multiple one-dimensional convolution kernels are used, the number of convolution kernels is set to 64, the convolution kernel size is 3, the step size is 1, and the padding method is the same; multiple convolution layers are stacked to form a deep one-dimensional convolution network structure, and the Transformer encoder is applied to the time feature sequence processed by multiple layers of convolution. The output of the Transformer encoder is the time axis feature; the spatial dimension of the multimodal feature cube is divided into fixed-size spatial blocks according to the spatial resolution, and a three-dimensional convolution operation is applied to the spatial feature data in the spatial block. The three-dimensional convolution operation obtains a spatial feature map, and the maximum pooling operation is applied to the spatial feature map. The spatial feature after pooling is the spatial axis feature.

[0088] Through the feature fusion layer, the time axis features and the space axis features are fused to generate fused features, and the fused features are calculated through the multi-head self-attention mechanism to generate weighted fused features;

[0089] It should be noted that the time axis features and the space axis features are connected by channel splicing, and the feature dimension after fusion is the sum of the time axis features and the space axis features; the fused features are input into the multi-head self-attention mechanism, and the number of attention heads is set, for example, set to 8, and the dimension of each attention head is determined according to the fusion feature dimension. The weight matrix and bias vector are calculated using the scaled dot product and weighted summed. The outputs of all attention heads are spliced ​​and the final attention output is generated through linear transformation. The final attention output feature is input into the fully connected layer, and the ReLU activation function is used to introduce the nonlinear transformation output weighted fusion feature.

[0090] Use the Softmax activation function to classify the weighted fusion features, generate a probability distribution vector, and analyze the instruction confidence based on the probability distribution vector;

[0091] It should be noted that the weighted fusion features are mapped to the classification space using the fully connected layer, and the Softmax transformation is performed to generate a probability distribution vector, and the maximum value in the probability distribution vector is taken as the instruction confidence, which is expressed as follows:

[0092]

[0093] Among them, P t Indicates the probability of the target category t, the value range is 0 <P t <1; Z t represents the original score of target category t in the weighted fusion feature Z; max(Z) represents the maximum value of all elements in the weighted fusion feature Z; C represents the total number of categories; r represents the index variable of all categories, and the value range is 1-C; t represents the target category index, which is used to specify the specific category of the current probability calculation, t=0 represents normal instruction, t=1 represents adversarial instruction; Z r Represents the original score of the rth category in the weighted fusion feature Z;

[0094] It should also be noted that each sample corresponds to a probability distribution vector, which represents the probability of belonging to each category, for example [0.78, 0.22] (the probability that the sample belongs to category 0 is 78%, and the probability of category 1 is 22%), where the maximum probability is 0.78 as the instruction confidence.

[0095] Use the confusion matrix method to set a confidence threshold and evaluate whether the instruction confidence exceeds the confidence threshold;

[0096] It should be noted that the rows of the confusion matrix represent the true categories and the columns represent the predicted categories. A C×C zero matrix is ​​created, and all multimodal data are traversed to obtain the true category and predicted category of each multimodal data. The true category and predicted category are then filled into the zero matrix to form the confusion matrix. Each combination of the true category and the predicted category is counted, and performance indicators of the confusion matrix are set, including true positive examples (TP), false positive examples (FP), false negative examples (FN), and true negative examples (TN). An initial confidence threshold of 50% is set for each category, and prediction results below the initial confidence threshold are considered low confidence. The confidence threshold is adjusted based on performance indicators such as TP, FP, and FN in the confusion matrix. If the FP of the confusion matrix is ​​≥20%, the initial confidence threshold is increased; if the FN is ≥20%, the initial confidence threshold is lowered. After each threshold adjustment, the confusion matrix and evaluation indicators (precision, recall, and F1 score) are recalculated until the evaluation indicators no longer improve, and the final confidence threshold for each category is recorded. The instruction confidence of each category is compared with the final confidence threshold to determine whether it exceeds the range.

[0097] It should also be noted that true positives (TP) refer to the number of samples whose true category completely matches the predicted category and is predicted correctly; false positives (FP) refer to the number of samples whose true category is wrong but is predicted to be a certain category (the true category is A but predicted to be B, and B is mistakenly judged to be correct); false negatives (FN) refer to the number of samples whose true category is correct but is mistakenly predicted to be other categories by the defense model; true negatives (TN) refer to the number of samples whose true category and predicted category are both wrong but match (for example, the true category is B and predicted to be B, but it should actually be A).

[0098] According to the evaluation results, set the false touch identification rules and the lock instruction generation rules and execute the lock instruction;

[0099] It should be noted that the final confidence threshold of category A is 65%. If the predicted probability is 70%, it is judged as high confidence; if it is 60%, it is marked as low confidence; according to the confusion matrix, if the false positive (FP) ratio of a category exceeds 15%, it is marked as an "easy to confuse category"; calculate the instruction confidence of each category and the confidence offset of the final confidence threshold, and calculate the risk score based on the confidence offset and the FP ratio; risk score = (FP ratio × confidence offset); when a category is marked as low confidence and belongs to the easy to confuse category, it is judged as an accidental touch operation; when the risk score is <0.4, it is low risk and requires secondary confirmation of the instruction; when 0.4≤risk score<0.7, it is medium risk, then the operation is suspended and feedback is given; when the risk score is ≥0.7, the locking operation is immediately executed and a safety alarm is triggered.

[0100] This embodiment also provides an anti-misoperation locking system based on intelligent recognition, comprising: a data acquisition module, a feature extraction module, a model construction module and an instruction generation module;

[0101] The data acquisition module is used to collect and preprocess multimodal data to generate a multimodal data set. The multimodal data includes voice data, visual data, and device status data.

[0102] A feature extraction module is used to extract multimodal features based on a multimodal dataset through a multimodal feature encoding method, and to generate a multimodal feature cube based on the multimodal features through a three-dimensional feature modeling method;

[0103] A model building module, which is used to input the multimodal feature cube into the two-stream GAN network, generate adversarial samples, build a defense model, and train with adversarial samples;

[0104] The instruction generation module is used to input the multimodal feature cube into the defense model, generate a probability distribution vector and instruction confidence, perform false touch identification based on the instruction confidence, and dynamically generate a locking instruction based on the identification result.

[0105] This embodiment also provides a computer device, which is suitable for the case of an anti-misoperation locking method based on intelligent identification, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the anti-misoperation locking method based on intelligent identification proposed in the above embodiment.

[0106] The computer device may be a terminal, comprising a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse.

[0107] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the anti-misoperation locking method based on intelligent identification as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0108] In summary, the present invention overcomes the problem of reduced recognition accuracy caused by environmental factors in complex scenarios through multimodal data fusion, significantly improves the accuracy and robustness of command recognition, and enhances adaptability and recognition accuracy in complex environments. At the same time, through adversarial training and false touch recognition mechanisms, the generalization ability of the defense model and its ability to resist unknown threats are improved, and the anti-interference ability is enhanced. It can still maintain high accuracy in the face of noise interference or abnormal input, ensuring the safety and reliability of the operation.

[0109] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for preventing misoperation based on intelligent identification, characterized in that: include, Collecting and preprocessing multimodal data to generate a multimodal dataset, wherein the multimodal data includes voice data, visual data, and device status data; Based on the multimodal dataset, multimodal features are extracted through the multimodal feature encoding method, and the multimodal features are generated into a multimodal feature cube through the three-dimensional feature modeling method; Input the multimodal feature cube into the two-stream GAN network to generate adversarial samples, build a defense model, and train it with adversarial samples; The multimodal feature cube is input into the defense model to generate a probability distribution vector and command confidence. False touch recognition is performed based on the command confidence, and a locking command is dynamically generated based on the recognition result.

2. The method for preventing misoperation based on intelligent identification according to claim 1, characterized in that: The multimodal data is collected and preprocessed to generate a multimodal data set. The specific steps are as follows: The WaveNet-based sound source separation method is used to reduce noise in speech data, the ESRGAN-based super-resolution reconstruction method is used to deblur visual data, and the wavelet transform method is used to reduce noise in device status data. Based on the preprocessed multimodal data, a multimodal dataset is generated through a multimodal data integration method.

3. The method for preventing misoperation based on intelligent identification according to claim 2, characterized in that: Based on the multimodal dataset, multimodal features are extracted through a multimodal feature encoding method, and the multimodal features are generated into a multimodal feature cube through a three-dimensional feature modeling method. The specific steps are as follows: Use multimodal feature encoding methods to encode speech data, visual data, and device status data in multimodal datasets; The encoded voice data, visual data, and device status data are concatenated and fused in the feature dimension through a joint encoding method, and mapped to a fixed dimension to generate multimodal features; The multimodal features are temporally and spatially aligned based on the timestamp synchronization interpolation method and the RTK differential positioning method; Through the spatiotemporal coordinate mapping method, the spatiotemporally aligned multimodal features are mapped to the corresponding positions of the three-dimensional cube according to the time axis resolution and the space axis resolution to generate a multimodal feature cube.

4. The anti-misoperation locking method based on intelligent identification according to claim 3, characterized in that: The multimodal feature cube is input into the two-stream GAN network to generate adversarial samples. The specific steps are as follows: Extract temporal and spatial features from the multimodal feature cube through the generator in the two-stream GAN network; The temporal features and spatial features are modally concatenated and input into a multi-layer perceptron to generate the initial features of the adversarial sample. Use the deconvolution network to upsample the initial features of the adversarial sample to generate a complete adversarial sample.

5. The anti-misoperation locking method based on intelligent identification according to claim 4, characterized in that: The defense model is constructed and trained through adversarial samples. The specific steps are as follows: The input layer is defined based on the characteristics of the multimodal feature cube, and the temporal feature extraction layer is defined based on the Transformer architecture; The spatial feature extraction layer is defined based on a 3D convolutional neural network, the feature fusion layer is defined based on a fully connected network, and the output layer is defined based on a classifier and a multi-head self-attention mechanism; Construct a defense model through the input layer, temporal feature extraction layer, spatial feature extraction layer, feature fusion layer and output layer; Using cross entropy loss as the loss function, Softmax as the activation function, and Adam as the optimizer, the complete adversarial sample is input into the defense model for training.

6. The method for preventing misoperation based on intelligent identification according to claim 5, characterized in that: The multimodal feature cube is input into the defense model to generate the probability distribution vector and instruction confidence. The specific steps are as follows: The time feature extraction layer and space feature extraction layer of the defense model extract the time axis features and space axis features of the multimodal feature cube respectively; Through the feature fusion layer, the time axis features and the space axis features are fused to generate fused features, and the fused features are calculated through the multi-head self-attention mechanism to generate weighted fused features; The Softmax activation function is used to classify the weighted fusion features, generate a probability distribution vector, and analyze the instruction confidence based on the probability distribution vector.

7. The method for preventing misoperation based on intelligent identification according to claim 6, characterized in that: The false touch recognition is performed according to the command confidence, and the locking command is dynamically generated according to the recognition result. The specific steps are as follows: Use the confusion matrix method to set a confidence threshold and evaluate whether the instruction confidence exceeds the confidence threshold; According to the evaluation results, the false touch identification rules and the lock instruction generation rules are set and the lock instruction is executed.

8. An anti-misoperation locking system based on intelligent recognition, based on the anti-misoperation locking method based on intelligent recognition according to any one of claims 1 to 7, characterized in that: Including data acquisition module, feature extraction module, model building module and instruction generation module; A data acquisition module, configured to acquire and preprocess multimodal data to generate a multimodal dataset, wherein the multimodal data includes voice data, visual data, and device status data; A feature extraction module is used to extract multimodal features based on a multimodal dataset through a multimodal feature encoding method, and to generate a multimodal feature cube from the multimodal features through a three-dimensional feature modeling method; A model building module, which is used to input the multimodal feature cube into the two-stream GAN network, generate adversarial samples, build a defense model, and train with adversarial samples; The instruction generation module is used to input the multimodal feature cube into the defense model, generate a probability distribution vector and instruction confidence, perform false touch identification based on the instruction confidence, and dynamically generate a locking instruction based on the identification result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the anti-misoperation locking method based on intelligent identification according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the anti-misoperation locking method based on intelligent identification according to any one of claims 1 to 7 are implemented.

Images