An intelligent identification-based anti-misoperation locking system and method

Abstract

The application discloses an anti-misoperation locking system and method based on intelligent identification, and relates to the technical field of industrial automation.The application comprises the following steps: collecting multi-modal data and preprocessing, generating a multi-modal data set, the multi-modal data comprising voice data, visual data and device state data; based on the multi-modal data set, extracting multi-modal features through a multi-modal feature coding method, generating a multi-modal feature cube through a three-dimensional feature modeling method; inputting the multi-modal feature cube into a double-flow GAN network to generate an adversarial sample, constructing a defense model, and training through the adversarial sample; inputting the multi-modal feature cube into the defense model to generate a probability distribution vector and an instruction confidence, and performing mis-touch identification according to the instruction confidence, and dynamically generating a locking instruction according to the identification result.The application improves the generalization ability of the defense model and the resistance ability when facing unknown threats through the adversarial training and mis-touch identification mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial automation, and in particular to an anti-misoperation locking system and method based on intelligent identification. BACKGROUND

[0002] In the field of power networks and industrial automation, anti-misoperation locking technology is a core and key technology to ensure the stable operation of power equipment, prevent electrical misoperation, and protect the personal safety of operators. Misoperation can cause equipment damage, power grid failure, and even major safety accidents, especially in critical scenarios such as high-voltage power transmission, substation switching operation, and factory automation production line control, which require more intelligent and accurate technical means to improve the anti-misoperation capability.

[0003] However, the recognition accuracy of intelligent identification technology for operation instructions is affected by various factors, including environmental noise, the accent and speech rate of operators, etc., resulting in a decrease in recognition accuracy in complex environments. Secondly, the threat of adversarial attacks is also a problem that needs to be solved. Adversarial attacks deceive the recognition mechanism by introducing minor perturbations, causing it to produce false recognition results, which poses a serious threat to the safety of anti-misoperation. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides an anti-misoperation locking method based on intelligent identification to solve the problems of poor recognition accuracy and adversarial attack threats.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides an anti-misoperation locking method based on intelligent identification, which includes collecting and preprocessing multi-modal data to generate a multi-modal data set, the multi-modal data including voice data, visual data, and device state data; based on the multi-modal data set, extracting multi-modal features through a multi-modal feature encoding method, generating a multi-modal feature cube through a three-dimensional feature modeling method; inputting the multi-modal feature cube into a dual-flow GAN network to generate adversarial samples, constructing a defense model, and training through the adversarial samples; inputting the multi-modal feature cube into the defense model to generate a probability distribution vector and an instruction confidence, and performing mis-touch identification according to the instruction confidence, and dynamically generating a locking instruction according to the identification result.

[0008] As a preferred solution of the anti-misoperation locking method based on intelligent identification, wherein:

[0009] The multi-modal data is collected and preprocessed to generate a multi-modal data set, and the specific steps are as follows,

[0010] The WaveNet-based sound source separation method is used for noise reduction processing of speech data, the ESRGAN-based super-resolution reconstruction method is used for deblurring processing of visual data, and the wavelet transform method is used for noise reduction processing of device state data.

[0011] Based on the preprocessed multi-modal data, a multi-modal data set is generated by a multi-modal data integration method.

[0012] As a preferred scheme of the anti-misoperation locking method based on intelligent identification, wherein:

[0013] Based on the multi-modal data set, multi-modal features are extracted by a multi-modal feature encoding method, and the multi-modal features are used to generate a multi-modal feature cube by a three-dimensional feature modeling method, and the specific steps are as follows,

[0014] The multi-modal feature encoding method is used to encode the speech data, visual data and device state data in the multi-modal data set;

[0015] The encoded speech data, visual data and device state data are spliced and fused in the feature dimension by a joint encoding method, and are mapped to a fixed dimension to generate multi-modal features;

[0016] The multi-modal features are spatio-temporally aligned according to a timestamp synchronization interpolation method and an RTK differential positioning method;

[0017] The spatio-temporally aligned multi-modal features are mapped to the corresponding positions of the three-dimensional cube according to the time axis resolution and the space axis resolution by a spatio-temporal coordinate mapping method, to generate a multi-modal feature cube.

[0018] As a preferred scheme of the anti-misoperation locking method based on intelligent identification, wherein:

[0019] The multi-modal feature cube is input into a double-flow GAN network to generate an adversarial sample, and the specific steps are as follows,

[0020] The time features and space features in the multi-modal feature cube are extracted by a generator in the double-flow GAN network;

[0021] The time features and space features are modally concatenated and input into a multi-layer perceptron to generate adversarial sample initial features;

[0022] The adversarial sample initial features are upsampled using a deconvolutional network to generate complete adversarial samples.

[0023] As a preferred scheme of the anti-misoperation locking method based on intelligent identification, wherein:

[0024] The defense model is constructed, and the specific steps are as follows,

[0025] The input layer is defined based on the characteristics of the multi-modal feature cube, the time feature extraction layer is defined based on the Transformer architecture;

[0026] The spatial feature extraction layer is defined based on the 3D convolutional neural network, the feature fusion layer is defined based on the fully connected network, and the output layer is defined based on the classifier and the multi-head self-attention mechanism;

[0027] The defense model is constructed through the input layer, the time feature extraction layer, the spatial feature extraction layer, the feature fusion layer and the output layer;

[0028] The cross-entropy loss is used as the loss function, the Softmax is used as the activation function, the Adam is used as the optimizer, and the complete adversarial sample is input into the defense model for training.

[0029] As a preferred scheme of the intelligent identification based anti-misoperation locking method,

[0030] The multi-modal feature cube is input into the defense model to generate a probability distribution vector and an instruction confidence, and the specific steps are as follows,

[0031] The time feature extraction layer and the spatial feature extraction layer of the defense model extract the time axis feature and the space axis feature of the multi-modal feature cube respectively;

[0032] The time axis feature and the space axis feature are fused through the feature fusion layer to generate a fusion feature, and the fusion feature is calculated through the multi-head self-attention mechanism to generate a weighted fusion feature;

[0033] The weighted fusion feature is classified using the Softmax activation function to generate a probability distribution vector, and the instruction confidence is analyzed according to the probability distribution vector.

[0034] As a preferred scheme of the intelligent identification based anti-misoperation locking method,

[0035] The mis-touch identification is performed according to the instruction confidence, and the locking instruction is dynamically generated according to the identification result, and the specific steps are as follows,

[0036] The confusion matrix method is used to set the confidence threshold, and whether the instruction confidence exceeds the confidence threshold is evaluated;

[0037] According to the evaluation result, the mis-touch identification rule and the locking instruction generation rule are set and the locking instruction is executed.

[0038] In a second aspect, the present application provides an intelligent identification-based anti-misoperation locking system, comprising a data acquisition module, a feature extraction module, a model construction module, and an instruction generation module; the data acquisition module is configured to acquire and preprocess multi-modal data to generate a multi-modal data set, wherein the multi-modal data comprises voice data, visual data, and device state data; the feature extraction module is configured to extract multi-modal features from the multi-modal data set by using a multi-modal feature coding method, and generate a multi-modal feature cube by using a three-dimensional feature modeling method; the model construction module is configured to input the multi-modal feature cube into a dual-stream GAN network to generate an adversarial sample, construct a defense model, and train the defense model by using the adversarial sample; and the instruction generation module is configured to input the multi-modal feature cube into the defense model to generate a probability distribution vector and an instruction confidence, perform mis-touch identification according to the instruction confidence, and dynamically generate a locking instruction according to the identification result.

[0039] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, any step of the intelligent identification-based anti-misoperation locking method according to the first aspect of the present application is implemented.

[0040] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, any step of the intelligent identification-based anti-misoperation locking method according to the first aspect of the present application is implemented.

[0041] The present application has the following advantages: through multi-modal data fusion, the problem of decreased recognition accuracy caused by environmental factors in complex scenes is overcome, the accuracy and robustness of instruction recognition are significantly improved, and the adaptability and recognition precision in complex environments are improved. Meanwhile, through adversarial training and mis-touch identification mechanism, the generalization ability of the defense model and the resistance to unknown threats are improved, the anti-interference ability is enhanced, high accuracy can be maintained when facing noise interference or abnormal input, and the safety and reliability of operation are ensured. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Fig. 1 The flowchart of the intelligent identification-based anti-misoperation locking method.

[0044] Fig. 2 A schematic diagram of an intelligent recognition-based anti-misoperation locking system.

[0045] Fig. 3 A flowchart of data collection and preprocessing.

[0046] Fig. 4 A flowchart of defense model construction. DETAILED DESCRIPTION

[0047] In order to make the above objectives, features and advantages of the present application more apparent, specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0048] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details given herein, that the present application can be practiced with other than the described implementations, and that variations of the described implementations can be made. Thus, the present application is not intended to be limited to the particular implementations described.

[0049] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. The "in one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments.

[0050] Reference Signs List Figs. 1-4 For one embodiment of the present application, the embodiment provides an intelligent recognition-based anti-misoperation locking method, comprising the following steps:

[0051] S1, collect multi-modal data and perform preprocessing to generate a multi-modal data set, the multi-modal data including voice data, visual data and device state data.

[0052] The voice data is denoised by a WaveNet-based sound source separation method, the visual data is deblurred by an ESRGAN-based super-resolution reconstruction method, and the device state data is denoised by a wavelet transform method;

[0053] It should be noted that WaveNet analyzes voice signals through a deep neural network, identifies and separates useful components in the voice, and removes background noise; ESRGAN enhances low-resolution images through a generative adversarial network to restore image details; the wavelet transform method can effectively process high-frequency noise in the device state data by decomposing signals into sub-signals of different frequencies, identifying and removing noise components.

[0054] Based on the preprocessed multi-modal data, a multi-modal data set is generated by a multi-modal data integration method;

[0055] It should be noted that the pre-processed speech data is standardized using Mel-frequency cepstral coefficients and stored as a two-dimensional floating-point array, the visual data is stored as a sequence of three-dimensional integer arrays of RGB images based on the input requirements of the convolutional neural network (CNN), and the device status data is stored as a two-dimensional floating-point array in time series format. After that, a multi-modal data structure containing speech data fields, visual data fields and device status data fields is defined by class object, and the standardized speech data, visual data and device status data are stored according to the defined multi-modal data structure. A plurality of instances storing different modal data are combined into a set to form a multi-modal data set.

[0056] S2, based on the multi-modal data set, multi-modal features are extracted by a multi-modal feature encoding method, and the multi-modal features are generated by a three-dimensional feature modeling method to form a multi-modal feature cube.

[0057] The multi-modal feature encoding method is used to encode the speech data, visual data and device status data in the multi-modal data set.

[0058] It should be noted that in the process of multi-modal data encoding, it usually refers to converting the original data into a format suitable for machine learning model processing; linear predictive coding (LPC) can be used to represent the speech features of speech data, pre-emphasis processing is performed on the speech data, usually using a first-order FIR filter with a coefficient of 0.95 or 0.97, the pre-emphasized speech data is segmented into short-time frames, usually with a frame length of 20-40 ms and a frame shift of 10 ms, a Hamming window function is applied to each frame of speech data to reduce spectral leakage, the linear prediction coefficients (LPC) of each frame of speech data are calculated, usually by solving the Yule-Walker equation to obtain the LPC coefficients, and the first 10-16 order coefficients are extracted as the acoustic features of the speech data; pixel-level encoding (such as RGB values) can be used to represent visual data features, an image processing library (such as OpenCV) is used to read visual data, if the image is not in RGB format, the image is converted to RGB format through color space conversion to ensure that each pixel contains the values of the red, green and blue channels, for each pixel, the RGB values are extracted, and all pixel RGB values of the image are arranged in row-major order to form a one-dimensional feature vector; the device status data is usually time series data, which is encoded by a time series encoding method, the time series data is segmented into fixed-length frames, usually each frame contains a number of sampling points (such as 1024 points), and the frame shift can be set as needed (such as 512 points), and the fast Fourier transform (FFT) is performed on each frame of time series data to convert the time domain signal to the frequency domain signal, and the amplitude spectrum and power spectrum of the frequency spectrum are calculated.

[0059] The coded speech data, visual data and device state data are spliced and fused in feature dimension by a joint coding method, and mapped to a fixed dimension to generate multi-modal features.

[0060] It should be noted that the coded speech data, visual data and device state data are feature extracted, such as extracting a speech feature vector from MFCC coefficients, which is usually 12-13 dimensions, extracting pixel values from RGB images to form a three-dimensional array, and extracting amplitude spectrum features such as spectral peak, spectral centroid and spectral bandwidth from the spectrum obtained by FFT transformation; ensure that the feature vectors of each modality have the same dimension, if the feature dimensions are different, the adjustment can be made by dimension reduction or padding operation; the speech features, visual features and device state features are spliced in the feature dimension, and the spliced feature vectors are mapped to a fixed dimension using a fully connected layer, and in the mapping process, an activation function can be used to enhance the expression ability of multi-modal features; the mapped feature vector is the multi-modal feature.

[0061] The multi-modal features are spatio-temporally aligned according to the timestamp synchronization interpolation method and the RTK differential positioning method.

[0062] It should be noted that for speech data and visual data, first, the respective timestamp information is obtained, and a common time point or a reference time point is found in the two timestamp sequences, the earliest timestamp in the two sequences can be selected as the reference time point, the offset of each timestamp relative to the reference time point is calculated, and the time is corrected through the offset to ensure that the speech data and the visual data are synchronized in time during the time alignment process, and then the value of the speech data at T vk is calculated according to the linear interpolation principle, through such interpolation operation, the timestamps of the speech data and the visual data can be aligned, thereby realizing the alignment of the speech data and the visual data in the time dimension, and the expression is,

[0063]

[0064] Wherein, x sk represents the feature vector of the speech data s at the kth time point of the visual data timestamp; x si represents the feature vector of the speech data s at the i-th time point of the timestamp; T vk represents the kth time point of the timestamp in the visual data v; T si represents the i-th time point of the timestamp in the speech data s, which is the adjacent and earlier timestamp of T vk ; T s(i+1) represents the adjacent and later timestamp of i-th time point in the speech data s; x s(i+1) represents the feature vector of the speech data s with the timestamp T s(i+1)The feature vector of the time instant; s represents the speech data; i represents an index in the speech data timestamp sequence, used to identify a specific timestamp position; v represents the visual data; k represents an index in the visual data timestamp sequence, used to identify a specific timestamp position;

[0065] The RTK differential positioning method improves the positioning accuracy through differential calculation between the reference station and the rover station. If the timestamps in the device state data and the timestamps of the differential correction numbers are inconsistent, a timestamp synchronization interpolation method is used to calculate the differential correction number corresponding to the device state data timestamp T according to the change rule of the differential correction number with time, so as to obtain more accurate and time-aligned device position coordinates.

[0066] Through the space-time coordinate mapping method, the space-time aligned multi-modal feature is mapped to the corresponding position of the three-dimensional cube according to the time axis resolution and the space axis resolution, and a multi-modal feature cube is generated.

[0067] It should be noted that for each space-time aligned multi-modal feature data point, the corresponding timestamp is obtained, and the coordinate position of the data point on the time axis is calculated according to the time axis resolution. The spatial coordinates of each pixel point are obtained, and the coordinate position of the corresponding pixel point on the space axis is calculated according to the space axis resolution. The time axis coordinate position and the space axis coordinate position are combined to determine the position of the multi-modal feature in the three-dimensional cube. According to the range of the time axis and the space axis and the determined resolution, a three-dimensional array is created to represent the multi-modal feature cube, and all elements of the three-dimensional array are initialized to default values. For each space-time aligned multi-modal feature data point, the feature value of the data point is filled into the corresponding position of the three-dimensional cube according to the position mapped into the three-dimensional cube, and finally a multi-modal feature cube is generated.

[0068] S3, input the multi-modal feature cube into the dual-stream GAN network to generate an adversarial sample, construct a defense model, and train through the adversarial sample.

[0069] The time feature and the space feature in the multi-modal feature cube are extracted through the generator in the dual-stream GAN network.

[0070] It should be noted that the generator is composed of multiple layers, including convolutional layers, pooling layers, deconvolutional layers, fully connected layers, etc.; the multi-modal feature cube first enters the convolutional layer in the generator, the convolutional layer performs convolution operation on the multi-modal feature cube by sliding a set of learnable convolutional kernels, the size of the convolutional kernel can be selected according to the characteristics of the time dimension and the space dimension, the convolutional kernel slides on the time dimension and the space dimension of the multi-modal feature cube, and the features in each local time window and space window are convolved and calculated, through the parallel operation of multiple such convolutional kernels, multiple different time feature maps and space feature maps can be obtained, then the time feature map and the space feature map are dimensionally reduced through the pooling layer, after multiple convolution and pooling operations, the obtained time feature maps and space feature maps at different levels are spliced and fused to obtain time features and space features.

[0071] The time features and the space features are modally concatenated and input into a multi-layer perceptron to generate an initial feature of the adversarial sample;

[0072] It should be noted that modal concatenation refers to splicing the features of two different modalities (time and space) in a specific dimension, usually selecting the channel dimension for splicing, arranging all the information of the time features in the channel dimension and all the information of the space features in the channel dimension in turn to form a new feature representation that fuses the time features and the space features; the multi-layer perceptron is composed of multiple fully connected layers, including an input layer, a hidden layer and an output layer; the feature data after modal concatenation is input into the multi-layer perceptron, and forward propagation calculation is performed through the hidden layer in turn, the neurons of each layer will process the input feature data, and this processing process includes weighted summation of the input feature data and nonlinear transformation through an activation function, and finally an initial feature of the adversarial sample is generated.

[0073] The initial feature of the adversarial sample is up-sampled using a deconvolutional network to generate a complete adversarial sample;

[0074] It should be noted that according to the dimension of the initial feature of the adversarial sample, the number of deconvolutional layers of the deconvolutional network is determined, and for each deconvolutional layer, the size of the convolution kernel, the step and the padding mode and the like parameters need to be determined, the size of the convolution kernel determines the size of the input region involved in each convolution operation, the step determines the step of the convolution kernel moving on the input data, and the padding mode is used to control the size of the output data; the initial feature of the adversarial sample is input into the first layer of the deconvolutional network as input data, and the initial feature of the adversarial sample is deconvoluted to increase the spatial dimension of the initial feature of the adversarial sample, after the processing of a deconvolutional layer, the output data obtained will be used as the input data of the next deconvolutional layer, and the deconvolutional operation is continued, until the number of deconvolutional layers is determined in advance, each deconvolutional operation gradually increases the spatial dimension of the initial feature of the adversarial sample, so that the initial feature of the adversarial sample gradually recovers to the size and feature representation close to the complete adversarial sample, and after the processing of all deconvolutional layers, the final output data obtained is the complete adversarial sample.

[0075] The input layer is defined based on the characteristics of the multi-modal feature cube, and the time feature extraction layer is defined based on the Transformer architecture.

[0076] It should be noted that the dimension information of the multi-modal feature cube is determined, including the length of the time dimension, the size of the space dimension and the number of feature channels, and the input layer needs to receive a multi-dimensional tensor as input, and the dimension of the tensor corresponds to the dimension of the multi-modal feature cube; in the time feature extraction layer, the self-attention mechanism of the Transformer architecture is used to extract the time feature in the multi-modal feature cube, the multi-modal feature cube is expanded into a sequence in the time dimension, and then the sequence is input into the Transformer architecture, the key parameters of the Transformer architecture are determined, such as the number of heads, the dimension of the hidden layer and the dimension of the feedforward neural network, the encoder of the Transformer architecture is constructed, and the time feature is extracted by using the encoder part.

[0077] The spatial feature extraction layer is defined based on the 3D convolutional neural network, the feature fusion layer is defined based on the fully connected network, and the output layer is defined based on the classifier and the multi-head self-attention mechanism.

[0078] It should be noted that firstly, the structure parameters of the 3D convolutional neural network are determined, a suitable convolution kernel size is selected, the convolution kernel size determines the range covered by each convolution operation in the spatial dimension; the step of the convolution operation is determined, the step determines the distance of the convolution kernel moving in the spatial dimension; a suitable padding method is selected, such as "same" padding or "valid" padding, "same" padding will add appropriate zero values at the edges of the input data, so that the output size after the convolution operation is the same as the input size; "valid" padding does not perform padding, and the output size after the convolution operation will be reduced according to the convolution kernel size and the step; the channel numbers of the input and the output are determined, the input channel number corresponds to the number of feature channels of the input data; according to the determined structure parameters, the 3D convolutional layer is constructed; the input and output dimensions of the fully connected network are determined, the input dimension should be determined according to the feature dimension output by the spatial feature extraction layer and the time feature dimension output by the time feature extraction layer, and the output dimension is determined according to the requirements of the task; according to the determined input dimension and output dimension, the fully connected layer is constructed, the spatial feature output by the spatial feature extraction layer and the time feature output by the time feature extraction layer are input into the fully connected network for feature splicing and fusion; a suitable classifier is selected according to the type of the task, common classifiers include Softmax classifier and Sigmoid classifier, etc., the parameters of the multi-head self-attention mechanism are determined, such as the number of heads and the hidden layer dimension, the number of heads determines the number of parallel calculations of the attention mechanism, and the hidden layer dimension determines the dimension size of each head, the fusion feature output by the feature fusion layer is input into the multi-head self-attention mechanism layer as input to obtain the fusion feature representation after attention weighting, and the weighted fusion feature output by the multi-head self-attention mechanism layer is input into the classifier for classification.

[0079] A defense model is constructed through an input layer, a time feature extraction layer, a spatial feature extraction layer, a feature fusion layer and an output layer;

[0080] Cross-entropy loss is used as a loss function, Softmax is used as an activation function, and Adam is used as an optimizer, and the complete adversarial sample is input into the defense model for training;

[0081] It should be noted that the complete adversarial samples are divided into a training set, a validation set and a test set; before starting the training, all parameters of the defense model need to be initialized, and the number of training rounds is set, the training set is divided according to the batch size, for example, the batch size is set to 32, a batch of selected adversarial samples is input into the defense model, and each class probability distribution vector is generated through the output layer by forward propagation; the parameters of the defense model are optimized by minimizing the cross-entropy loss, the gradients of the loss with respect to the parameters of each layer of the defense model are calculated by the back propagation algorithm, the gradients of each layer are calculated in turn from the output layer to the input layer, and the gradients of the input layer are calculated. The parameters of the defense model are updated using the Adam optimizer according to the calculated gradients, and the Adam optimizer adjusts the learning rate according to the historical gradient information of each parameter, and updates the weights and biases of the defense model according to certain update rules; after reaching the maximum number of training rounds, the validation set is input into the defense model, and the loss value of the defense model on the validation set is calculated, if the performance of the defense model on the validation set no longer improves or starts to decline, the hyperparameters of the defense model (such as learning rate, batch size, etc.) need to be adjusted; finally, the test set is input into the defense model, and the prediction result of the defense model is obtained, the evaluation index of the defense model on the test set is calculated, such as accuracy, the performance of the defense model in defending the complete adversarial samples is measured by the evaluation index, whether the defense model achieves the expected effect is evaluated, and the expression is,

[0082]

[0083] Wherein, A represents the proportion of the number of complete adversarial samples predicted correctly by the defense model to the total number of complete adversarial samples; n represents the total number of complete adversarial samples participating in the evaluation of the defense model, y 1l represents the true label of the lth complete adversarial sample; y 2l represents the prediction label of the lth complete adversarial sample by the defense model; I(y 1l =y 2l ) is an indicator function, when y 1l =y 2l , it means that the prediction label of the defense model is equal to the true label, I=1, when y 1l ≠y 2l , it means that the prediction label of the defense model is not equal to the true label, I=0; l represents the index variable of the sample, the value range is 1-n, y1 represents the true label, and y2 represents the prediction label.

[0084] It should be noted that when constructing a complete adversarial sample for defense model training, detailed labeling criteria need to be formulated according to specific tasks, and the collected adversarial samples are labeled according to the formulated labeling criteria, so as to complete the labeling of real labels; for example, in image classification, if the image is a picture of a cat, its real label is the "cat" category; in network security, if the network traffic is normal network access traffic, the real label may be "normal", and if it is malicious attack traffic, the real label may be the corresponding attack type (such as "DDoS attack" and the like).

[0085] S4, input the multi-modal feature cube into the defense model to generate a probability distribution vector and an instruction confidence, and perform mis-touch identification according to the instruction confidence, and dynamically generate a lock instruction according to the identification result.

[0086] The time feature extraction layer and the space feature extraction layer of the defense model extract the time axis feature and the space axis feature of the multi-modal feature cube respectively;

[0087] It should be noted that the input multi-modal feature cube is divided into fixed-length time windows in the time dimension, each time window contains continuous multiple time step data, a one-dimensional convolution operation is applied to the time feature data in each time window, multiple one-dimensional convolution kernels are used, the number of convolution kernels is set to 64, the size of the convolution kernel is 3, the step is 1, and the padding mode is same; a plurality of convolution layers are stacked to form a deep one-dimensional convolution network structure, and a Transformer encoder is applied to the time feature sequence processed by the plurality of convolution layers, and the output of the Transformer encoder is the time axis feature; the spatial dimension of the multi-modal feature cube is divided into fixed-size spatial blocks according to the spatial resolution, and a three-dimensional convolution operation is applied to the spatial feature data in the spatial block, and the three-dimensional convolution operation obtains a spatial feature map, and a maximum pooling operation is applied to the spatial feature map, and the spatial axis feature after the pooling is obtained.

[0088] The time axis feature and the space axis feature are fused through the feature fusion layer to generate a fusion feature, and the fusion feature is calculated through a multi-head self-attention mechanism to generate a weighted fusion feature;

[0089] It should be noted that the time axis feature and the space axis feature are connected by channel concatenation, and the dimension of the fused feature is the sum of the time axis feature and the space axis feature; the fused feature is input into a multi-head self-attention mechanism, the number of attention heads is set, for example, 8, the dimension of each attention head is determined according to the dimension of the fused feature, the weight matrix and the bias vector are calculated using the scaled dot product and weighted summation, the outputs of all attention heads are spliced and linearly transformed to generate the final attention output, and the final attention output feature is input into a fully connected layer, and a ReLU activation function is used to introduce a nonlinear transformation to output the weighted fused feature.

[0090] The weighted fused feature is classified using a Softmax activation function to generate a probability distribution vector, and the instruction confidence is analyzed according to the probability distribution vector.

[0091] It should be noted that the weighted fused feature is mapped to a classification space using a fully connected layer, and a Softmax transformation is performed to generate a probability distribution vector, and the maximum value in the probability distribution vector is taken as the instruction confidence, and the expression is,

[0092]

[0093] where P t represents the probability of the target category t, and the value range is 0 t <1; Z t represents the original score of the target category t in the weighted fused feature Z; max(Z) represents the maximum value of all elements in the weighted fused feature Z; C represents the total number of categories; r represents the index variable of all categories, and the value range is 1-C; t represents the target category index, which is used to specify the specific category for which the probability is calculated, t=0, indicating a normal instruction, and t=1, indicating an adversarial instruction; Z r represents the original score of the rth category in the weighted fused feature Z;

[0094] It should also be noted that each sample corresponds to a probability distribution vector, which represents the probability of belonging to each category, for example, [0.78, 0.22] (the probability of the sample belonging to category 0 is 78%, and the probability of category 1 is 22%), and the maximum value of the probability is taken as the instruction confidence, which is 0.78.

[0095] The confidence threshold is set using the confusion matrix method, and whether the instruction confidence exceeds the confidence threshold is evaluated.

[0096] It should be noted that the rows of the confusion matrix represent the true classes, and the columns represent the predicted classes, a Cx C zero matrix is created, all multi-modal data is traversed, the true class and the predicted class of each multi-modal data are obtained, and the true class and the predicted class are filled into the zero matrix, that is, the confusion matrix, the combination of each true class and predicted class is counted, and the performance indicators of the confusion matrix are set, including true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN); an initial confidence threshold of 50% is set for each class, and the prediction result below the initial confidence threshold is regarded as low confidence; according to the performance indicators TP, FP, FN, etc. in the confusion matrix, the confidence threshold is adjusted, if the FP of the confusion matrix is greater than or equal to 20%, the initial confidence threshold is increased; if the FN is greater than or equal to 20%, the initial confidence threshold is decreased; after adjusting the threshold each time, the confusion matrix and the evaluation indicators (accuracy, recall rate, F1 score) are recalculated until the evaluation indicators no longer improve, and the final confidence threshold of each class is recorded; the instruction confidence of each class is compared with the final confidence threshold to determine whether it exceeds the range.

[0097] It should also be noted that true positives (TP) represent the number of samples that are completely matched with the predicted class and correctly predicted; false positives (FP) represent the number of samples that are incorrectly classified but predicted as a certain class (true A but predicted as B, and B is incorrectly determined as correct); false negatives (FN) represent the number of samples that are correctly classified but incorrectly predicted by the defense model as other classes; and true negatives (TN) represent the number of samples that are incorrectly classified but matched (e.g., true B and predicted B, but actually A).

[0098] According to the evaluation results, the mis-touch identification rule and the lock instruction generation rule are set and the lock instruction is executed;

[0099] It should be noted that the final confidence threshold of class A is 65%, if the prediction probability is 70%, it is determined as high confidence; if it is 60%, it is marked as low confidence; according to the confusion matrix, if the false positive (FP) ratio of a certain class exceeds 15%, it is marked as an "easily confused class"; the confidence offset of the instruction confidence and the final confidence threshold of each class is calculated, and the risk score is calculated according to the confidence offset and the FP ratio; risk score = (FP ratio x confidence offset); when a certain class is marked as low confidence and belongs to an easily confused class, it is determined as a mis-touch operation; when the risk score is less than 0.4, it is low risk and needs a secondary confirmation instruction; when the risk score is between 0.4 and 0.7, it is medium risk, and the operation is suspended and feedback is given; when the risk score is greater than or equal to 0.7, the lock operation is immediately executed and a safety alarm is triggered.

[0100] The embodiment also provides an intelligent identification-based anti-misoperation locking system, comprising a data acquisition module, a feature extraction module, a model construction module and an instruction generation module.

[0101] The data acquisition module is configured to acquire and pre-process multi-modal data, generate a multi-modal data set, and the multi-modal data comprises voice data, visual data and device state data.

[0102] The feature extraction module is configured to extract multi-modal features based on the multi-modal data set by a multi-modal feature coding method, and generate a multi-modal feature cube by a three-dimensional feature modeling method.

[0103] The model construction module is configured to input the multi-modal feature cube into a dual-flow GAN network, generate an adversarial sample, construct a defense model, and train the defense model by the adversarial sample.

[0104] The instruction generation module is configured to input the multi-modal feature cube into the defense model, generate a probability distribution vector and an instruction confidence, perform mis-touch identification according to the instruction confidence, and dynamically generate a locking instruction according to the identification result.

[0105] The embodiment also provides a computer device suitable for the intelligent identification-based anti-misoperation locking method, comprising a memory and a processor.

[0106] The computer device can be a terminal, and the computer device comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is configured to perform wired or wireless communication with external terminals. The wireless communication can be achieved through WIFI, a carrier network, NFC (near field communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device. In addition, the input device can be an external keyboard, touchpad or mouse, etc.

[0107] The embodiment also provides a storage medium on which a computer program is stored, the program being executed by a processor to implement the method for preventing misoperation locking based on intelligent identification as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0108] To sum up, the present application overcomes the problem of low recognition accuracy in complex scenes due to environmental factors by multi-modal data fusion, significantly improves the accuracy and robustness of instruction recognition, and enhances the adaptability and recognition accuracy in complex environments. At the same time, through the adversarial training and mis-touch recognition mechanism, the generalization ability and resistance to unknown threats of the defense model are improved, and the anti-interference ability is enhanced, which can still maintain high accuracy when facing noise interference or abnormal input, ensuring the safety and reliability of the operation.

[0109] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. A method for preventing misoperation locking based on intelligent identification, characterized in that: comprising, Collecting multi-modal data and preprocessing to generate a multi-modal dataset, the multi-modal data including speech data, visual data and device status data; comprising, The speech data is denoised by a WaveNet-based sound source separation method, the visual data is deblurred by an ESRGAN-based super-resolution reconstruction method, and the device status data is denoised by a wavelet transform method; based on the preprocessed multi-modal data, a multi-modal data set is generated by a multi-modal data integration method; Based on the multi-modal dataset, multi-modal features are extracted by a multi-modal feature encoding method, and the multi-modal features are generated into a multi-modal feature cube by a three-dimensional feature modeling method; comprising, The speech data, visual data and device status data in the multi-modal dataset are encoded using a multi-modal feature encoding method; the encoded speech data, visual data and device status data are spliced and fused in the feature dimension by a joint encoding method, and mapped to a fixed dimension to generate multi-modal features; the multi-modal features are spatio-temporally aligned according to a timestamp synchronization interpolation method and an RTK differential positioning method; the spatio-temporally aligned multi-modal features are mapped to the corresponding positions of the three-dimensional cube according to the time axis resolution and the space axis resolution by a spatio-temporal coordinate mapping method, to generate a multi-modal feature cube; The multi-modal feature cube is input into a dual-flow GAN network to generate an adversarial sample, construct a defense model, and train through the adversarial sample; the specific steps are as follows, The time features and space features in the multi-modal feature cube are extracted by the generator in the dual-flow GAN network; the time features and space features are modally concatenated and input into a multi-layer perceptron to generate adversarial sample initial features; the adversarial sample initial features are upsampled using a deconvolution network to generate complete adversarial samples; The multi-modal feature cube is input into the defense model to generate a probability distribution vector and an instruction confidence, and to perform mis-touch recognition according to the instruction confidence, and to dynamically generate a lock instruction according to the recognition result; The defense model is constructed and trained through the adversarial sample, and the specific steps are as follows, The input layer is defined based on the characteristics of the multi-modal feature cube, the time feature extraction layer is defined based on the Transformer architecture; The spatial feature extraction layer is defined based on the 3D convolutional neural network, the feature fusion layer is defined based on the fully connected network, and the output layer is defined based on the classifier and the multi-head self-attention mechanism; The defense model is constructed through the input layer, the time feature extraction layer, the spatial feature extraction layer, the feature fusion layer and the output layer; The cross-entropy loss is used as the loss function, the Softmax is used as the activation function, and the Adam is used as the optimizer, and the complete adversarial sample is input into the defense model for training.

2. The intelligent identification-based misoperation prevention locking method of claim 1, wherein: The multi-modal feature cube is input into the defense model to generate a probability distribution vector and an instruction confidence, and the specific steps are as follows, The time axis features and space axis features of the multi-modal feature cube are extracted by the time feature extraction layer and the spatial feature extraction layer of the defense model, respectively; The time axis feature and the space axis feature are fused through a feature fusion layer to generate a fused feature, and the fused feature is calculated through a multi-head self-attention mechanism to generate a weighted fused feature; The weighted fused feature is classified using a Softmax activation function to generate a probability distribution vector, and the instruction confidence is analyzed according to the probability distribution vector.

3. The intelligent identification-based anti-misoperation locking method of claim 1, characterized in that: The instruction confidence is used to identify a misoperation, and a locking instruction is dynamically generated according to an identification result, and the specific steps are as follows, A confusion matrix method is used to set a confidence threshold, and whether the instruction confidence exceeds the confidence threshold is evaluated; According to the evaluation result, a misoperation identification rule and a locking instruction generation rule are set and the locking instruction is executed.

4. A misoperation prevention locking system based on intelligent identification, based on the misoperation prevention locking method based on intelligent identification of any one of claims 1-3, characterized in that: The method comprises a data acquisition module, a feature extraction module, a model construction module, and an instruction generation module; The data acquisition module is used to acquire and pre-process multi-modal data to generate a multi-modal data set, wherein the multi-modal data comprises voice data, visual data, and device state data; The feature extraction module is used to extract multi-modal features based on the multi-modal data set through a multi-modal feature encoding method, and the multi-modal features are used to generate a multi-modal feature cube through a three-dimensional feature modeling method; The model construction module is used to input the multi-modal feature cube into a dual-flow GAN network to generate an adversarial sample, construct a defense model, and train the adversarial sample; The instruction generation module is used to input the multi-modal feature cube into the defense model to generate a probability distribution vector and an instruction confidence, identify a misoperation according to the instruction confidence, and dynamically generate a locking instruction according to an identification result.

5. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that: The processor executes the computer program to implement the steps of the intelligent identification-based anti-misoperation locking method of any one of claims 1-3.

6. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to implement the steps of the intelligent identification-based anti-misoperation locking method of any one of claims 1-3.

Images