Power industry social engineering attack identification method based on multi-modal feature fusion

Abstract

The invention discloses a power industry social engineering attack identification method based on multi-modal feature fusion, and belongs to the technical field of information security, and the method comprises the steps: obtaining multi-source heterogeneous data of a system user; based on the preprocessed multi-source heterogeneous data, extracting a multi-modal state feature vector; carrying out standardization processing on the multi-modal state feature vector, mapping the multi-modal state feature vector to a unified semantic space through linear transformation to obtain feature representation, calculating modal confidence distribution of the feature representation, constructing cross-modal attention weight distribution based on modal confidence, and carrying out cross-modal attention weight distribution based on the cross-modal attention weight distribution; according to the cross-modal attention weight distribution, performing weighted aggregation on the feature representation to generate a multi-modal fusion feature; and inputting the multi-modal fusion features into a neural network classifier to generate an abnormal behavior recognition result. According to the method, a dual extraction mode of static semantic features and time sequence dynamic features is combined, so that the current state information of the user is reserved, the behavior evolution trend is utilized, and the technical problem of simultaneously processing instant state and time sequence changes is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information security, in particular to a power industry social engineering attack recognition method based on multi-modal feature fusion. BACKGROUND

[0002] In the field of information security, there are social engineering attacks, not only using technical vulnerabilities, but also using human behavior, psychological state and emotional fluctuations to bypass system protection, guiding system users to perform unconventional operations by manipulating their emotions, trust or stress.

[0003] In the existing method, the recognition of social engineering attacks usually uses network behavior and operation logs for analysis, but it is difficult to effectively capture the human behavior, psychological state and emotional fluctuations of system users, and usually uses natural language technology to analyze text data, uses language processing technology to identify language data, and uses operation logs for evaluation analysis to identify abnormal behavior recognition of social engineering.

[0004] However, when integrating data, it is not possible to combine static and dynamic timing information, and accurately capture the impact on system optimization.

[0005] Therefore, there is an urgent need for a power industry social engineering attack recognition method based on multi-modal feature fusion to recognize dynamic behavior changes of system users. SUMMARY

[0006] In view of the above problems, the present application is proposed.

[0007] Therefore, the technical problem solved by the present application is: how to use cross-modal attention mechanism and behavior pattern dynamic adjustment to solve the accuracy problem of multi-source heterogeneous data fusion evaluation in the prior art and improve the effectiveness of abnormal behavior recognition.

[0008] To solve the above technical problems, the present application provides the following technical solutions: a power industry social engineering attack recognition method based on multi-modal feature fusion, comprising: Obtain multi-source heterogeneous data of system users, and preprocess the multi-source heterogeneous data based on data modal differences; Extract a multi-modal state feature vector based on the preprocessed multi-source heterogeneous data; Standardize and map the multi-modal state feature vector to a unified semantic space, perform weighted fusion based on an attention mechanism, and generate a multi-modal fusion feature; Input the multi-modal fusion feature into a neural network classifier to generate an abnormal behavior recognition result.

[0009] As a preferred solution of the power industry social engineering attack recognition method based on multi-modal feature fusion, wherein: The preprocessing of the multi-source heterogeneous data based on the data modal difference comprises: performing word segmentation processing on text data based on a dictionary, and constructing a stop word filtering rule according to word frequency statistics; The audio data is subjected to effective signal detection, and background noise is eliminated by using a digital filtering method; The log data is subjected to missing value prediction and completion based on historical records, and data format standardization processing is performed.

[0010] As a preferred scheme of the power industry social engineering attack recognition method based on multi-modal feature fusion, the multi-modal state feature vector is extracted, the semantic representation of the text data is obtained through a language model, the semantic representation is input into a psychological perception attention layer, and a psychological state feature vector is extracted through psychological weight calculation. The frame-level representation of the audio data is obtained through an audio coding model, the frame-level representation is input into a bidirectional sequence network for context modeling, and an emotion state feature vector is extracted through a multi-head attention mechanism to capture key emotional fragments. The log data is encoded into a hidden state sequence through a time sequence network, the hidden state sequence is input into a graph neural network for dynamic correlation modeling, a behavior state feature vector is extracted by using a distance measurement method to identify an abnormal pattern.

[0011] As a preferred scheme of the power industry social engineering attack recognition method based on multi-modal feature fusion, the multi-modal state feature vector is standardized and mapped to a unified semantic space, and weighted fusion is performed based on an attention mechanism, which comprises: projecting and transforming the feature representation to obtain a query vector. The similarity between the query vector and the feature representation is calculated to obtain a confidence score; The confidence score is subjected to probability normalization processing to obtain a modal confidence distribution.

[0012] As a preferred scheme of the power industry social engineering attack recognition method based on multi-modal feature fusion, the multi-modal state feature vector is standardized and mapped to a unified semantic space, and weighted fusion is performed based on an attention mechanism, which further comprises: obtaining a feature attention component by performing linear transformation and nonlinear activation on the feature representation and then performing inner product with a weight vector; The modal confidence is taken as a logarithm and subjected to weighted scaling to obtain a confidence enhancement component; The feature attention component and the confidence enhancement component are added to obtain an attention score, and the attention score is subjected to exponential normalization processing to obtain a cross-modal attention weight distribution.

[0013] As a preferred scheme of the power industry social engineering attack recognition method based on multi-modal feature fusion, the generating multi-modal fusion features comprises: obtaining weighted feature representations by performing element-wise multiplication operation on the attention weights of each modality and the feature representations. The multi-modal fusion features are obtained by performing vector summation operation on the weighted feature representations.

[0014] As a preferred scheme of the power industry social engineering attack recognition method based on multi-modal feature fusion, the inputting the multi-modal fusion features into the neural network classifier to generate abnormal behavior recognition results comprises: organizing the multi-modal fusion features into a time sequence matrix through time window division, calculating the average value of the feature vectors in all time steps in the time sequence matrix to obtain static semantic features. The time sequence matrix is sequentially modeled based on a gating mechanism, the history information is selected by an update gate, the current information is selected by a reset gate, and a time sequence dynamic feature is output. The static semantic features and the time sequence dynamic features are spliced by feature dimension expansion to combine into a comprehensive feature vector. The comprehensive feature vector is transformed layer by layer through forward propagation between layers of the neural network, each layer obtains a transformation result through weight transformation and activation function processing, the transformation result is converted into a numerical score of each abnormal category through probability mapping in the output layer, and the abnormal behavior recognition result is determined according to the maximum value position of the numerical score.

[0015] The application provides an abnormal behavior recognition system based on multi-modal feature fusion.

[0016] To solve the above technical problems, the application provides the following technical scheme: an abnormal behavior recognition system based on multi-modal feature fusion, comprising: a data acquisition module configured to acquire multi-source heterogeneous data of system users, wherein the multi-source heterogeneous data comprises text data, audio data and log data, and the multi-source heterogeneous data is preprocessed based on data modality differences; A feature extraction module is configured to extract multi-modal state feature vectors from behavior dimensions, interaction dimensions and operation dimensions based on the preprocessed multi-source heterogeneous data. A feature fusion module is configured to perform standardization processing on the multi-modal state feature vectors, map the multi-modal state feature vectors to a unified semantic space through linear transformation to obtain feature representations, calculate modality confidence of the feature representations, construct cross-modality attention weight distribution based on the modality confidence, and generate multi-modal fusion features by weighted aggregation of the feature representations according to the cross-modality attention weight distribution. The behavior prediction module is configured to input the multi-modal fusion feature into a neural network classifier, extract a static semantic feature through global average pooling, extract a time sequence dynamic feature through a gated recurrent network, and generate an abnormal behavior recognition result based on feature splicing of the static semantic feature and the time sequence dynamic feature.

[0017] The application provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the steps of the power industry social engineering attack identification method based on multi-modal feature fusion when executing the computer program.

[0018] The application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the power industry social engineering attack identification method based on multi-modal feature fusion when executed by a processor.

[0019] The application has the beneficial effects that the reliability of different data sources is evaluated by introducing a modal confidence mechanism, and a cross-modal attention weight distribution is constructed based on the confidence to optimize the feature fusion stage.

[0020] In the feature extraction stage, a state feature vector system is constructed from multiple dimensions, key information is captured through an attention mechanism, and hidden abnormal behaviors caused by psychological factors can be identified.

[0021] The dual extraction mode of the static semantic feature and the time sequence dynamic feature retains the current state information of the user and utilizes the behavior evolution trend, and solves the technical problem that the traditional method cannot simultaneously process the instant state and the time sequence change. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0023] Figure 1 The application provides a power industry social engineering attack identification method based on multi-modal feature fusion.

[0024] Figure 2 The application provides a power industry social engineering attack identification method based on multi-modal feature fusion.

[0025] Figure 3 The application provides a power industry social engineering attack identification method based on multi-modal feature fusion.

[0026] Figure 4 A system user activity behavior heat map of the power industry social engineering attack recognition method based on multi-modal feature fusion is provided for an embodiment of the present application.

[0027] Figure 5 A key performance indicator diagram of the power industry social engineering attack recognition method based on multi-modal feature fusion is provided for an embodiment of the present application.

[0028] Figure 6 A five-category social engineering attack detection matrix diagram of the power industry social engineering attack recognition method based on multi-modal feature fusion is provided for an embodiment of the present application.

[0029] Figure 7 A single-modal and multi-modal comparison curve diagram of the power industry social engineering attack recognition method based on multi-modal feature fusion is provided for an embodiment of the present application.

[0030] Figure 8 A multi-element data performance evaluation diagram of the power industry social engineering attack recognition method based on multi-modal feature fusion is provided for an embodiment of the present application.

[0031] Figure 9 A loss value change curve diagram of the power industry social engineering attack recognition method based on multi-modal feature fusion is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the above objectives, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0033] Embodiment 1, refer to Figure 1 For an embodiment of the present application, the embodiment provides a power industry social engineering attack recognition method based on multi-modal feature fusion, comprising: Step 1: Obtain multi-source heterogeneous data of system users, the multi-source heterogeneous data including text data, audio data and log data, and pre-process the multi-source heterogeneous data based on data modal differences; Step 2: Based on the pre-processed multi-source heterogeneous data, extract multi-modal state feature vectors from behavior dimensions, interaction dimensions and operation dimensions; Step 3: Standardize the multimodal state feature vectors and map them to a unified semantic space through linear transformation to obtain feature representations. Calculate the modal confidence distribution of the feature representations. Construct a cross-modal attention weight distribution based on the modal confidence. Perform weighted aggregation on the feature representations according to the cross-modal attention weight distribution to generate multimodal fusion features. Step 4: Input the multimodal fusion features into a neural network classifier, extract static semantic features through global average pooling, extract temporal dynamic features through a gated recurrent network, and generate abnormal behavior recognition results based on the static semantic features and the temporal dynamic features.

[0034] Existing abnormal behavior identification technologies primarily rely on single data sources for analysis. However, abnormal user behavior often manifests as multi-layered and multi-dimensional composite signals. Text data reflects user language expression patterns, audio data contains information on emotional and intonation variations, and operation logs record temporal characteristics of behavior; each of these three types of data has its limitations. Current methods generally employ simple feature concatenation or weighted averaging when fusing these heterogeneous data, ignoring the quality differences and varying importance of different modalities. Furthermore, abnormal user behavior exhibits temporal evolutionary characteristics, possessing both static features reflecting the current state and dynamic features demonstrating behavioral trends. Existing technologies lack effective mechanisms for integrating these two types of features.

[0035] Therefore, to address the aforementioned issues of uneven quality in multimodal data and insufficient utilization of temporal features, such as... Figure 1 As shown, through steps 1-4, this method establishes a complete technical framework for abnormal behavior recognition. Step 1 improves the usability of various types of data through differentiated preprocessing. Step 2 constructs a comprehensive feature description of user status from three perspectives: psychology, emotion, and behavior. Step 3 introduces confidence assessment and attention mechanisms to achieve adaptive fusion of data quality perception. Step 4 combines static state analysis and dynamic trend modeling to complete the accurate identification of abnormal behavior, solving the technical shortcomings of traditional methods in multimodal fusion and time series analysis.

[0036] Example 2, refer to Figures 2-4 As an embodiment of the present invention, based on the previous embodiment, a method for identifying social engineering attacks in the power industry based on multimodal feature fusion is provided, including: like Figure 2 As shown, in step 1: acquiring multi-source heterogeneous data from system users, the multi-source heterogeneous data includes text data, audio data, and log data. Preprocessing the multi-source heterogeneous data based on data modality differences includes the following steps A1-A3: A1: Perform word segmentation on the text data based on a dictionary, and construct stop word filtering rules based on word frequency statistics; A2: Perform effective signal detection on the audio data, and eliminate background noise by using a digital filtering method; A3: Perform missing value prediction and completion on the log data based on historical records, and perform data format standardization processing.

[0037] Further, perform word segmentation processing on the text data based on a dictionary, and construct a stop word filtering rule based on word frequency statistics, including the following steps A1.1-A1.3: A1.1: Clean and standardize the obtained text data. This embodiment selects the power industry for specific description. In the power industry, a power professional dictionary is introduced to ensure the accuracy of word segmentation. Among them, represents a set of words containing power industry professional terms, device names, and technical concepts, such as substation, power distribution network, relay protection, load forecasting, and other professional words.

[0038] jieba is a Chinese text segmentation library that supports custom dictionary functions. The power professional dictionary is loaded into the jieba segmentation tool as a custom dictionary to ensure that power professional terms are not incorrectly segmented during the segmentation process. Count the frequency of each word in the segmented text to establish a correspondence between the word and the frequency, and obtain the initial word vector .

[0039] A1.2: Due to the particularity of the power industry, it is necessary to filter out strong field high-frequency words with extremely high frequency but low psychological information, such as inspection and operation. These words, although an inevitable part of the work process, can overwhelm personalized psychological characteristic signals if they appear in large quantities. This embodiment filters high-frequency words from the initial word vector by constructing a field stop word candidate set.

[0040] Construct a field stop word candidate set, denoted as: Among them, represents the field stop word candidate set, represents the words in the power professional dictionary , represents the number of times the word appears in the power professional dictionary , represents the sum of the number of times all words in the power professional dictionary appear, represents each word in the power professional dictionary , represents the frequency threshold, represents the calculation of the frequency of the word In the power professional dictionary The relative frequency; By constructing a field stopword candidate set, all words with a relative frequency exceeding the frequency threshold are included in the stopword candidate set It should be noted that the stopword candidate set is usually composed of procedural words such as check, operation, execution, etc.

[0041] A1.3: According to the content of the field stopword candidate set , the words contained in the initial word vector are deleted to obtain a word frequency vector containing high-discrimination psychological information.

[0042] To ensure that important psychological expression words are not mistakenly deleted during the frequency filtering process, the word frequency vector is merged with the general psychological state basic word list to obtain the cleaned text features : wherein represents the general psychological state basic word list, which contains psychological feature words related to anxiety, trust, and alertness.

[0043] The cleaned text features not only retain the field-related words in the power professional dictionary , but also contain the psychological expression words in the general psychological state basic word list .

[0044] It should be noted that the frequency threshold is determined by statistical analysis method. The specific steps are as follows: first, calculate the relative frequency distribution of all words in the corpus, and sort the words by frequency from high to low; then calculate the cumulative probability of the frequency distribution, and identify the frequency value when the cumulative probability reaches 95% as the baseline threshold; finally, multiply the baseline threshold by the adjustment coefficient to obtain the actual threshold, when the corpus size is less than 50,000 words, the adjustment coefficient is 1.5, when the size is between 50,000-200,000 words, the adjustment coefficient is 1.2, and when the size is greater than 200,000 words, the adjustment coefficient is 1.0. In this way, the threshold setting can adapt to the features of different sizes of corpus.

[0045] Further, the general psychological state basic word list The construction adopts a technical scheme of vocabulary classification and screening. First, an emotional vocabulary classification system is established, and the vocabulary is divided into three categories: positive emotions (such as happy, satisfied, confident, and calm), negative emotions (such as worried, nervous, uneasy, and doubtful), and neutral states (such as focused, thinking, observing, and waiting). Then, through the word frequency statistical method, the synonyms and synonyms of the above-mentioned classified vocabulary are extracted from the general Chinese corpus to expand the vocabulary coverage. Finally, through semantic similarity calculation, the repeated vocabulary with a similarity of more than 0.8 is deleted to form the basic vocabulary table of the psychological state vocabulary.

[0046] Further, the electric power professional vocabulary is constructed by using the technical method of field document analysis and term extraction. First, the technical documents of the electric power industry are collected as source data, including DL / T standard series documents, GB / T electric power standard documents, substation operation regulations, and relay protection device instruction manuals; then the collected documents are preprocessed, including format conversion, encoding unification, and special symbol cleaning; then the TF-IDF algorithm is used to calculate the field-specific score of the vocabulary, and the vocabulary with a score exceeding 0.3 is selected as a professional term candidate; finally, through manual verification, the general vocabulary and the incorrectly identified terms are deleted to form a vocabulary dictionary containing electric power professional vocabulary such as circuit breaker, disconnector, current transformer, voltage transformer, and main transformer.

[0047] Due to the particularity of speech data, it may contain a large number of silent segments, which need to be detected and removed first. In the speech signal, the short-time energy can reflect the strength of the speech signal in the local time, and the energy of the voiced segment is usually higher, and the energy of the unvoiced segment is usually lower. The short-time zero-crossing rate is used to reflect the number of zero-crossings in a unit of time, which can reflect the change frequency and intensity of the signal. Therefore, by calculating the short-time energy and the short-time zero-crossing rate, it can be judged whether the current frame contains valid speech signal, and the accuracy of the silence detection is improved.

[0048] In the embodiment of the application, the digital filtering method is used to eliminate background noise by: According to the sampling rate to obtain the original speech signal , the short-time energy and the short-time zero-crossing rate are calculated, which are represented as: Among them, is the short-time energy of the t th frame, is the amplitude of the t th sampling point, N represents the frame length, represents the short-time zero-crossing rate of the t is an amplitude threshold value for excluding small perturbations caused by noise, is an indicator function, which takes 1 if the condition is true, otherwise 0, j represents the sample point index in the short-time zero-crossing rate calculation, which is used to traverse the sample points in the frame.

[0049] Specifically, the amplitude threshold value is set using a signal energy adaptive method. The standard deviation of the amplitude values of all sample points in the current frame is calculated, and 0.1 times the standard deviation is taken as the initial threshold value. The threshold value coefficient is adjusted according to the signal-to-noise ratio of the signal. When the estimated signal-to-noise ratio is high (greater than 20 dB), the threshold value coefficient is set to 0.05 times the standard deviation; when the signal-to-noise ratio is medium (10-20 dB), the threshold value coefficient is set to 0.1 times the standard deviation; and when the signal-to-noise ratio is low (less than 10 dB), the threshold value coefficient is set to 0.2 times the standard deviation.

[0050] It should be noted that, when the short-time energy and the short-time zero-crossing rate are less than the threshold values ,

[0051] Finally, the non-silence segment or is obtained, wherein, represents the i-th audio segment, represents the short-time energy of the i-th audio segment, represents the energy threshold value, represents the short-time zero-crossing rate value of the i-th audio segment, represents the zero-crossing rate threshold value.

[0052] In the non-silence speech data segment, there is also a problem of high background noise (such as a substation environment), so the interference of high background noise cannot be ignored when processing speech data. The background noise suppression is performed using a Wiener filter-based method, which is represented as: wherein, is the frequency domain response of the filter, is the estimated pure speech spectrum, is the noise template of the 1s before the silence segment.

[0053] Each non-silence segment is subjected to Fourier transform to obtain the frequency at the i-th moment, is substituted into the formula to obtain the frequency domain response of the period, and finally the inverse Fourier transform is performed to obtain the noise-reduced signal .

[0054] ​After the noise reduction processing, there are still damaged speech segments. A missing value segment completion method based on a generative adversarial network is used to complete the damaged speech segments into the WSGAIN-GP model for generator training, denoted as: wherein, is the damaged speech segment; is a mask matrix, with 1 representing damage and 0 representing integrity; is the repaired signal generated by the generator, denotes an element-wise multiplication operator, denotes the complement of the code matrix.

[0055] The repaired speech signal is finally obtained .

[0056] After the silence segment removal, noise reduction and damaged speech segment repair, the final high-quality complete segment of the speech signal containing the power industry system users is obtained, denoted as: In an optional embodiment, the background noise can be eliminated by using a digital filtering method, which can be implemented by: Adaptive filtering algorithm is adopted to perform frame processing on the speech signal, and the power spectral density of each frame is calculated; Based on the minimum mean square error criterion, the filter coefficients are adaptively adjusted; The statistical property difference between speech and noise is used to dynamically estimate the noise power spectrum; The noise speech signal is filtered by the adaptive filter, and the noise-reduced speech signal is output.

[0057] In another optional embodiment, the background noise can be eliminated by using a digital filtering method, which can be implemented by: The spectral subtraction method is used to estimate the noise power spectrum in the silence segment of the speech signal, and a noise spectrum template is established; The noise speech signal is subjected to short-time Fourier transform to obtain a frequency domain representation; The enhanced speech power spectrum is obtained by subtracting the estimated value of the noise power spectrum from the power spectrum of the noise speech; The time domain speech signal is reconstructed by inverse short-time Fourier transform combined with the phase information of the original signal.

[0058] The power industry operation log contains many different contents, but due to improper operation or different real situations, some important contents such as time stamp and operation behavior may be missing.

[0059] In the embodiments of the present application, the data format standardization processing is performed by: Linear interpolation is used for missing timestamps, and historical behavior pattern prediction is used for missing operation type processing. By analyzing the historical behavior pattern, the most likely missing operation type is predicted, denoted as: wherein, denotes the set of all possible operation types (e.g., query = 0, modify = 1), denotes the operation type, denotes the set of all possible operation types , and is the user based historical operation type, maximizes. Finally, a new operation log dataset containing user ID, timestamp, and operation type is obtained.

[0060] Due to the large number of power system devices, the timestamps or operation type codes generated by the operation log will also differ, so standardization processing is needed before multi-modal feature extraction. The timestamps of the above preliminary cleaned are unified into UTC format and discretized time windows, and the operation types of the operation log are one-hot encoded and mapped, denoted as: Finally, a standardized matrix is obtained.

[0061] wherein, is the function, = number of time windows + number of operation types, denotes the i-th operation type, denotes all possible operation type numbers, denotes the set of real numbers, n denotes the number of rows of the matrix, and d denotes the number of columns of the matrix.

[0062] According to the operation log, Z-Score threshold method is used for abnormal operation detection, and operation frequency statistics are performed. The number of operations of a user in a window is calculated , denoted as: wherein, is the mean and standard deviation of the window , and denotes the standardized score.

[0063] Abnormality determination is performed, denoted as: Anomaly operation label vector can be obtained .

[0064] Because a single operation in the original log may contain random fluctuations (such as accidental misoperation), noise can be smoothed by aggregation calculation, long-term trends can be highlighted, and attack behavior tends to show statistical anomalies in continuous windows (such as continuous high-frequency access), and such patterns are easier to capture after aggregation. Therefore, time series segmentation data is obtained by time series segmentation, a conventional sliding window method is adopted, and the window is defined as , ΔT represents the length of the time window, represents the end time of the kth time window, represents the start time of the kth time window.

[0065] Operation intra-window aggregation is performed, which is represented as: wherein, represents the operation weight, represents the operation index belonging to the kth time window, represents the one-hot encoding vector of the ith operation.

[0066] The original fine-grained data is converted into high information density features by statistical calculation, and the original operation series is converted into aggregated statistical indicators. Finally, time series segmentation data windows containing the number of times of accessing sensitive data by users and the operation frequency variance features are obtained, and m represents the total number of windows.

[0067] It should be noted that the operation weight The calculation of the score adopts a multi-factor weighted scoring method. First, a basic weight is assigned according to the security sensitivity of the operation type, and the operation type is divided into three levels of high-sensitive operation, medium-sensitive operation and low-sensitive operation, and the basic weight values are 3, 2 and 1 respectively. High-sensitive operations include system configuration modification, user permission change and sensitive data access, medium-sensitive operations include data export, batch query and file transfer, and low-sensitive operations include ordinary query, state check and log viewing. Then, weight adjustment is made according to the time characteristics of the operation, and 24 hours of a day are divided into working hours, delayed working hours and non-working hours, and the corresponding time adjustment coefficients are 1.0, 1.5 and 2.0 respectively, and the operation in non-working hours is considered to have a higher abnormal risk. Then, the operation frequency factor is considered, and the frequency of each user performing a specific operation type in the current time window is calculated, and when the frequency is more than twice the historical average frequency of the user, the frequency adjustment coefficient is set to 1.8, and when it is more than three times, the frequency adjustment coefficient is set to 2.5, otherwise the frequency adjustment coefficient is kept at 1.0. Finally, personalized adjustment is made in combination with the historical behavior pattern of the user, the user behavior baseline is established by analyzing the operation records of the user in the past 30 days, and when the current operation deviates greatly from the historical behavior pattern of the user, the weight coefficient is increased. The final operation weight calculation formula is the product of the basic weight, the time adjustment coefficient, the frequency adjustment coefficient and the personalized adjustment coefficient.

[0068] In an optional implementation, the data format standardization processing can be performed by: The maximum and minimum values of each feature dimension are calculated by using the maximum-minimum normalization method, and the original numerical values are mapped to the [0, 1] interval.

[0069] For the timestamp feature, it is converted into the number of seconds from the reference time, and then normalized.

[0070] For the category type feature, the label encoding method is used to convert the category into a numerical identifier.

[0071] In another optional implementation, the data format standardization processing can also be performed by: The Z-Score standardization method is used to process the continuous type features, the mean and standard deviation of each feature are calculated, the original numerical value is subtracted from the mean and divided by the standard deviation, so that the feature conforms to the standard normal distribution.

[0072] For the time feature, the periodic components of hours, weeks and months are extracted, and the standardization processing is performed respectively.

[0073] For sparse category type features, the hash encoding method is used for feature representation to reduce the dimension and keep the semantic information of the features.

[0074] In step 2: based on the pre-processed multi-source heterogeneous data, multi-modal state feature vectors are extracted from the behavior dimension, interaction dimension, and operation dimension, including the following steps B1-B3: B1: Obtain the semantic representation of the text data through the language model, input the semantic representation into the psychological perception attention layer, and extract the psychological state feature vector through psychological weight calculation; B2: Obtain the frame-level representation of the audio data through the audio coding model, input the frame-level representation into the bidirectional sequence network for context modeling, capture key emotional fragments through the multi-head attention mechanism, and extract the emotional state feature vector; B3: Encode the log data into a hidden state sequence through the time sequence network, input the hidden state sequence into the graph neural network to model dynamic association, and extract the behavior state feature vector by identifying abnormal patterns using distance measurement methods.

[0075] Specifically, in step B1, based on natural language processing technology, the BERT pre-training model is used to obtain the deep semantic representation of the text data, and the psychological features (such as anxiety, trust) of the system user in the text data are extracted.

[0076] The BERT model is based on the Transformer architecture and deeply understands the context relationship of the text through bidirectional encoding. Each message is converted into a token sequence, and the BERT model outputs the context representation of each token sequence , as the global representation of the text, which contains the overall semantic information of the text. And the psychological perception attention layer (PAL) is introduced to extract the psychological feature vector.

[0077] The processed text data , pre-trained BERT model parameters and learnable psychological weight matrix are input into the psychological perception attention layer (PAL) to extract psychological related features.

[0078] First, psychological projection is performed, and the hidden state of the first layer of BERT is multiplied by the psychological weight matrix to obtain the psychological perception query vector, represented as: where, is the hidden state matrix of the first layer of BERT, representing the context semantic information of the text, is the psychological perception query vector, used to locate key semantics in the power psychological feature space, represents the learnable psychological weight matrix.

[0079] This maps BERT's general semantics to the psychological feature space of the power industry. A psychological prior constraint is introduced to calculate the attention score, expressed as: in, It is the element in the i-th row and j-th column of the attention weight matrix, representing the perceived attention weight of the i-th token to the j-th token in the text. Represents the normalization function. Represents the first in the query vector 1 element, from the query matrix The Line, representing the first The query representation of each token Represents the first key in the key vector elements, from the key matrix The Line, representing the first The key representation of each token Indicates the scaling factor. It is a dimension of psychological characteristic space. This represents the mental prior weighting coefficient, which controls the influence of mental lexicon similarity on attention scores. The default value is 0.3. This represents a semantic similarity function, calculated based on a power psychology dictionary. Harmony Semantic similarity between them Indicates the first in the text The words corresponding to each position Indicates the first in the text The words corresponding to each position.

[0080] The obtained attention weight matrix value vector Weighted summation is performed to aggregate contextual information related to psychological characteristics, generating a psychological context vector. , Represents a value matrix, It is a value transformation matrix.

[0081] Using a multi-head fusion mechanism to integrate mental context vectors It is divided into 8 heads, each head corresponding to a specific psychological characteristic dimension.

[0082] The mental context vector is split according to a specific index range, with each head corresponding to mental feature information in a subspace.

[0083] Multi-head attention mechanisms can capture the correlation between words in parallel in different subspaces, thereby extracting psychological features more comprehensively.

[0084] Dynamically assign psychological importance and calculate gating weights, expressed as follows: in, Indicates the first Gating weights for each attention head, This represents the Sigmoid activation function. Indicates the first The output vector of each attention head. Represents the learnable weight matrix, with dimension . Used to map the head vector to a scalar. This represents the hidden layer dimension size of each attention head. Represents the learnable bias parameters, with dimension . .

[0085] The final psychological feature vector is obtained, represented as: in, This represents the final psychological feature vector, which includes indicators such as anxiety index, trust level, and alertness. h This indicates the total number of heads.

[0086] By introducing a multi-head attention gating mechanism enhanced with industry knowledge, not only is the core problem of insufficient capture of psychological characteristics of the power industry by general BERT solved, but it can also extract potential psychological characteristics of system users from multiple dimensions (such as a sharp drop in trust and a surge in anxiety).

[0087] Furthermore, in step B2, by analyzing the voice data of system users, the emotional state characteristics of system users are revealed to determine whether they are being manipulated by emotions.

[0088] In social engineering attacks, attackers often manipulate the target's emotions to carry out their attacks. This study uses a Wav2Vec pre-trained model and a bidirectional LSTM to analyze changes in emotional expression in audio data.

[0089] The methods for extracting audio emotion features are as follows: Preprocessed system user voice signal data The data is input into the Wav2Vec model. The Wav2Vec model uses a deep neural network to analyze features such as tone, rate of speech, and pitch in the audio signal, capturing emotional fluctuations in the user's voice. Secondly, the Wav2Vec model outputs a general feature representation for each frame of audio data. ,in Indicates the first Emotional feature vector of frame.

[0090] Temporal context modeling is performed, and bidirectional LSTM is used to capture the context information in the audio sequence. The hidden state of bidirectional LSTM is spliced and represented as: wherein, represents the forward LSTM; represents the backward LSTM; is the input at the current time; and represent the hidden states of the forward and backward, respectively; represents the spliced hidden state of bidirectional LSTM at time step t, which contains the forward and backward context information.

[0091] The multi-head attention mechanism is used to capture key emotional segments, and the obtained key emotional segments are time-pooled and classified, represented as: wherein, represents the key emotional segments captured by the multi-head attention mechanism, T represents the number of time steps, and t represents the index of the time step. The multi-head attention mechanism can capture different emotional features in parallel, improving the expression ability and accuracy of the model.

[0092] In order to convert the emotional features into a form suitable for subsequent feature fusion, the pooled features are further mapped to a fixed-dimensional emotional feature vector containing indicators such as happy and lost, represented as: wherein, is a learnable weight matrix, is a learnable bias vector.

[0093] Further, in step B3, the behavior state feature is mainly captured by analyzing the operation log data of the system user to capture the dynamic changes of the behavior. In social engineering attacks, attackers often induce system users to perform certain abnormal behaviors, such as frequent access to sensitive information, abnormal login behavior, etc. These abnormal behaviors are often precursors to attacks. Therefore, a time series modeling method is used to analyze the behavior patterns in the operation log to identify abnormal access patterns and behaviors.

[0094] The cleaned operation log dataset is modeled for time series relationship to capture the time series dependence of the operation behavior. First, the discrete operation types are embedded into continuous vectors, and then LSTM time series modeling is used to obtain the time series feature matrix wherein is the LSTM hidden layer dimension, is the number of time steps, is the hidden state of each time step.

[0095] To obtain the dynamic features of the time series , the time series feature matrix of the LSTM output is averaged and pooled: where, is the hidden state of each time step.

[0096] The time series graph network modeling is performed to obtain the time series interaction graph of user-device-operation , the nodes , represent the user nodes, represent the operation nodes, and the edge E represents the association strength between the user and the operation (based on frequency or time proximity).

[0097] First, the timestamp is mapped to the periodic feature: where, is the period length, usually taking one day of time (e.g., in hours, P=24).

[0098] The dynamic node embedding is obtained using the time series graph attention network (TGAT) , reflecting the dynamic association between the user and the operation, and the specific formula is as follows: where, is the neighbor set of node , is the time encoding of the edge, is the embedding vector of node i in the l−1 layer, is the information of all neighbor nodes, is the time series graph attention network function.

[0099] Finally, the graph structure feature based on TGAT is obtained.

[0100] In step 1 data cleaning, the abnormal operation label vector has been obtained, and the Fourier transform is used to extract the main period, converting the abnormal operation label vector into a time series signal , and then performing Fourier transform, represented as: ​ wherein the main period Fourier transform can convert the signal in time domain to the signal in frequency domain, so as to extract the main period in the signal, which is very useful for analyzing the periodic mode of operation behavior.

[0101] According to the frequency characteristics obtained by spectrum analysis, the behavior graph structure characteristics are combined The Mahalanobis distance calculation method is used to distinguish each time Abnormal operation, expressed as: wherein, The current operation feature vector is represented as , The mean vector and covariance matrix of historical data are represented as. The Mahalanobis distance considers the covariance structure of the data, accurately measures the distance between the sample and the population, and effectively detects abnormal operation. Finally, the abnormal operations at different times are combined into a vector to obtain the feature statistics .

[0102] The , , Three feature vectors are dimensionally aligned and spliced Finally, a fixed-dimensional behavior feature vector is obtained, which integrates the operation events and other indicators of system users.

[0103] As shown in Figure 4 , the behavior patterns and operation type frequency distribution of system users on different dates within a week are displayed in the form of a heat map. The horizontal axis marks the operation type (such as normal login, frequent access to sensitive information, file transfer, and non-normal request time), the vertical axis is the date, and the cell value represents the number of occurrences or frequency score of the corresponding behavior (positive value for normal frequency, negative value for abnormal degree).

[0104] In step 3: standardizing the multi-modal state feature vector, mapping to a unified semantic space through linear transformation to obtain a feature representation, calculating the modal confidence distribution of the feature representation, constructing a cross-modal attention weight distribution based on the modal confidence, and generating a multi-modal fusion feature by weighted aggregation of the feature representation according to the cross-modal attention weight distribution, including steps C1-C3: Step C1: projecting the feature representation to obtain a query vector; calculating the confidence score based on the similarity between the query vector and the feature representation; and performing probability normalization on the confidence score to obtain the modal confidence distribution.

[0105] Step C2: obtain a feature attention component by performing a linear transformation and a nonlinear activation on the feature representation and then performing an inner product with a weight vector; obtain a confidence enhancement component by taking a logarithm of the modality confidence and performing a weighted scaling; add the feature attention component and the confidence enhancement component to obtain an attention score, and perform an exponential normalization on the attention score to obtain a cross-modal attention weight distribution.

[0106] Step C3: obtain a weighted feature representation by performing an element-wise multiplication operation on the attention weight of each modality and the feature representation; and obtain a multi-modal fusion feature by performing a vector summation operation on the weighted feature representation.

[0107] In the embodiment of the present application, the multi-modal fusion feature is generated in step 3 by: performing normalization processing on the input psychological feature vector, emotional feature vector and behavioral feature vector respectively to eliminate dimensional differences, to obtain normalized features, denoted as: wherein, is the mean, is the standard deviation, is a constant to prevent the denominator from being zero, i represents the modality index, j represents the feature type index, p represents the psychological feature, e represents the emotional feature, and b represents the behavioral feature.

[0108] The indicators between different modalities are different, which may be biased towards the behavioral modality with large numerical values during training. For example, the “anxiety degree” indicator in the psychological modality feature may fluctuate between 0.2-0.9, while the “access frequency” in the behavioral modality feature may be between 10-200 times, with a large difference in order of magnitude. Thus, the importance of other modalities is ignored. Through the normalization processing, it can be ensured that different modalities are on the same scale in numerical value.

[0109] Map the normalized features of each modality to the same dimension shared latent space to obtain a representation vector: wherein, represents the scientific system weight matrix of the i-th modality, represents the bias vector of the scientific system of the i-th modality, is a nonlinear activation function.

[0110] In practical applications, the psychological feature vector can strengthen the feature dimensions of "anxiety degree" and "trust degree" through the mapping; the emotional feature vector can highlight typical abnormal emotions such as "sudden increase in speech speed" and "trembling tone"; and the behavior feature vector emphasizes behavior patterns such as "frequent operation during non-working hours" and "batch access to sensitive data". Through the projection, the features of different modalities are mapped into the same dimensional space, ensuring that different modalities can be compared and fused in the same semantic space.

[0111] The query vector is obtained by projecting and transforming the feature representation; the confidence score is calculated based on the similarity between the query vector and the feature representation; the modal confidence distribution confidence distribution is obtained by probability normalization processing of the confidence score, and is represented as: wherein, is a Sigmoid function, represents the transpose of the weight vector for confidence calculation, represents the bias scalar for confidence calculation.

[0112] The confidence is normalized, and is represented as: wherein, represents the normalized confidence of the i th modality.

[0113] The feature attention component is obtained by performing linear transformation and nonlinear activation on the feature representation and then taking the inner product with the weight vector; the confidence enhancement component is obtained by taking the logarithm of the modal confidence and performing weighted scaling; the attention score is obtained by adding the feature attention component and the confidence enhancement component; and the cross-modal attention weight distribution is obtained by exponential normalization processing of the attention score; The cross-modal attention weight distribution is represented as: wherein, represents the transpose of the weight vector for attention calculation, the weight matrix of the attention mechanism, represents the bias vector of the attention mechanism, represents the weight coefficient of the confidence enhancement term.

[0114] The attention weights of each modality are obtained by Softmax, and are represented as: When the emotional modality detects that "there is an angry emotion in the voice", the attention mechanism will automatically increase the weight of this modality; if the behavior log only shows an occasional abnormal click, its importance will be reduced. Dynamically adjust the weight of the modality, pay more attention to the modality features related to the attack.

[0115] The weighted feature representation is obtained by element-wise multiplication of the attention weight of each modality and the feature representation; and the multi-modal fusion feature is obtained by vector sum operation on the weighted feature representation. The multi-modal fusion feature is represented as: wherein, represents the weight of each modality. According to the different weights between modalities, it is reflected that the fluctuation of user state at this moment may be the key trigger point of social engineering attack.

[0116] In an optional embodiment, the multi-modal fusion feature generated in step 3 can be obtained by: Cross-correlation analysis is performed on each modality representation vector to calculate the Pearson correlation coefficient matrix between modalities; A modality similarity graph is constructed based on the correlation coefficient matrix, and a graph convolution network is used to propagate the feature of the similarity graph; The feature representation of each node is updated through multi-layer graph convolution operation; and the updated node feature is globally pooled to obtain a graph-level feature representation; The graph-level feature representation is linearly transformed to output the multi-modal fusion feature.

[0117] In another optional embodiment, the multi-modal fusion feature generated in step 3 can also be obtained by: A multi-modal contrast learning framework is constructed, and different modalities of the same sample are regarded as positive sample pairs; A temperature parameter controlled contrast loss function is used to learn the consistency representation between modalities; Feature alignment is performed by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs; and The aligned features of each modality obtained by contrast learning are simply spliced; The spliced feature vector is dimensionally projected to obtain the final fusion feature.

[0118] In step 4: the multi-modal fusion feature is input into a neural network classifier, a static semantic feature is extracted through global average pooling, a timing dynamic feature is extracted through a gated recurrent network, and based on the static semantic feature and the timing dynamic feature, an abnormal behavior recognition result is generated, including the following steps D1-D4: D1: Organize the multi-modal fusion features into a time sequence matrix through time window division, and calculate the average value of the feature vectors at all time steps in the time sequence matrix to obtain static semantic features; D2: Sequence modeling of the time sequence matrix based on a gating mechanism, the history information is selected by the update gate, the current information is selected by the reset gate, and the time sequence dynamic features are output; D3: The static semantic features and the time sequence dynamic features are spliced through feature dimension expansion to form a comprehensive feature vector; D4: The comprehensive feature vector is transformed layer by layer through forward propagation between layers of the neural network, each layer is processed through weight transformation and activation function to obtain a transformation result, and the transformation result is converted into a numerical score of each abnormal class through probability mapping in the output layer, and the abnormal behavior recognition result is determined according to the maximum value position of the numerical score.

[0119] In the embodiment of the application, the abnormal behavior recognition result generated in step 4 is obtained by: As shown in Figure 3 Based on the obtained feature vector, the fusion feature is generated after the cross-modal attention mechanism, which is represented as: Wherein, is the length of the time window, is the multi-modal fusion feature vector of time step t (which contains psychological state, emotional features, behavior features, etc.), and d represents the dimension size of the fusion feature vector.

[0120] In order to support the combination of static features (reflecting the state of the system user "at this moment") and dynamic features (referring to the evolution trajectory of the system user state in time sequence) of attack mode, the dynamic features extracted by the static global average feature extraction and the gating recurrent unit are combined with the dynamic features and the static global average feature, which is represented as: Wherein, is the static global average feature, is the dynamic feature extracted by the gating recurrent unit, is the vector splicing operation, is the final fusion feature vector.

[0121] The input feature vector enters the hidden layer through the input layer of the MLP model, each hidden layer is nonlinearly transformed through the full connection layer and the ReLU activation function, and the high-level representation of the feature is gradually extracted. Finally, the output layer maps the feature to the probability distribution of the attack type through the Softmax activation function, and determines the final prediction result.

[0122] First, the input feature vector The input layer of the MLP model enters the first hidden layer. The output of the input layer is the input of the first hidden layer: Then assume that the MLP model has L hidden layers, and the output of each hidden layer is nonlinearly transformed by a fully connected layer and a ReLU activation function. The input of the hidden state of the layer is calculated as: where, is the weight matrix of the layer, is the output dimension of the layer, is the output dimension of the layer, is the bias vector of the layer, =max(0,x) is an activation function used to introduce nonlinearity.

[0123] The output of the last hidden layer serves as the input of the output layer. The number of neurons in the output layer is equal to the number of attack types C, and the Softmax activation function is used to convert the output into a probability distribution: where, is the weight matrix of the layer, is the bias vector of the layer, is the Softmax activation function that normalizes the output values into a probability distribution. represents the probability of each attack sample belonging to each class, and C represents the total dimension of the attack type.

[0124] For each input sample, the model outputs the predicted probability of each attack type (such as phishing attacks, impersonation attacks, information collection attacks, etc.).

[0125] In an optional implementation, the abnormal behavior recognition result generated in step 4 can be generated by: inputting the fusion feature vector into a convolutional neural network for feature extraction; using a one-dimensional convolution kernel to perform local pattern recognition on the feature sequence; capture feature patterns of different granularities through multi-scale convolution kernels; concatenate the multi-scale convolution results to form a feature map; perform a global max-pooling operation on the concatenated feature map; The pooling result is input into a fully connected layer for classification, and the prediction probability of each category is output.

[0126] In another optional embodiment, the abnormal behavior recognition result generated in step 4 can also be generated by: using a distance metric-based classification method for abnormal recognition; pre-establishing a prototype feature vector of each abnormal category as a reference template; calculating the Euclidean distance between the input fusion feature and each prototype feature; converting the distance value into a similarity score through a negative exponential function; normalizing the similarity scores of each category to obtain a probability distribution; selecting the category with the highest similarity score as the final recognition result.

[0127] Embodiment 3, refer to Figures 5-9 For an embodiment of the present application, a multi-modal feature fusion-based power industry social engineering attack recognition method is provided. In order to verify the beneficial effects of the present application, scientific demonstration is carried out through experiments.

[0128] The experimental setup includes a data set of 10,000 records, which covers five types of social engineering attacks including phishing attacks, impersonation attacks, privilege escalation attacks, information collection attacks, and identity spoofing attacks.

[0129] The precision, recall rate and F1 score are used as evaluation indicators of the multi-modal model to evaluate different attack types, such as Figure 5 As shown in the figure, the horizontal axis represents the five types of attacks, and the vertical axis shows the indicator values, with a range of 0.2 to 0.8. Phishing attacks and identity spoofing attacks perform best on all indicators, with values close to 0.8, followed by privilege escalation attacks; while information collection attacks still have room for improvement in recall rate and F1 score. The evaluation indicators of all attack types exceed the baseline value. Through algorithm-level optimization, the multi-modal model achieves an effective balance between high precision and high recall rate, further verifying its good generalization ability in heterogeneous social engineering attack scenarios.

[0130] The confusion matrix is used to verify the performance of the multi-modal model, as shown in Figure 6As shown, the differences in classification performance of the five types of social engineering attacks are shown by the confusion matrix. The left figure represents detection based on single log behavior only, and the latter represents multi-modal attack detection. The rows represent the true attack types, and the columns represent the multi-modal model prediction results. The diagonal values are the number of correctly classified samples. In this embodiment, the number of correctly classified samples for all categories is improved. The number of correctly classified phishing attacks increases from 120 to 130, the number of impersonation attacks increases from 115 to 120, the number of privilege escalation attacks increases from 110 to 115, the number of information collection attacks increases from 100 to 105, and the number of identity spoofing attacks increases from 110 to 115. The overall performance of the multi-modal model in identifying different types of attacks is enhanced. At the same time, the number of misclassified samples is reduced. The number of phishing attacks misclassified as impersonation attacks decreases from 10 to 5, and the number of information collection attacks misclassified as privilege escalation attacks decreases from 6 to 4.

[0131] Figure 7 The attack recognition effect of data under the single-modal model and the multi-modal model is shown. Although the performance of the single-modal model is similar to that of the multi-modal model proposed in this application in the early stage of the first 5 rounds, the accuracy of the multi-modal model is better than that of the single-modal model after the 6th round.

[0132] Figure 8 The performance evaluation of the five types of network attack detection models on text, sound, and log data is shown in polar form. The angle corresponds to the attack type, and the radial value ranges from 0.1 to 0.6, representing the detection score. The overall data shows that the model achieves balanced performance across attack types through multi-modal feature fusion in heterogeneous data scenarios.

[0133] Figure 9 The loss function is used to optimize and train the model in the power industry social engineering attack recognition experiment based on multi-modal behavior feature analysis. The horizontal axis represents the training steps, and the vertical axis represents the loss value. From the curve trend, as the training steps increase, the loss value starts to decrease rapidly from about 2.3 initially, and then continues to decrease in fluctuations. When the training step reaches about 8000, the loss value stabilizes at a low level. The multi-modal model gradually reduces the error between the predicted value and the true value by continuously learning and adjusting parameters during the training process. The loss function can effectively guide the model optimization, and the accuracy of the model in identifying power industry social engineering attacks is continuously improved.

[0134] Embodiment 4 is an embodiment of the present application, which provides an abnormal behavior recognition system based on multi-modal feature fusion, comprising: The data collection module is configured to acquire multi-source heterogeneous data of a system user, the multi-source heterogeneous data including text data, audio data, and log data, and pre-process the multi-source heterogeneous data based on data modal difference; The feature extraction module is configured to extract a multi-modal state feature vector from a behavior dimension, an interaction dimension, and an operation dimension based on the pre-processed multi-source heterogeneous data. The feature fusion module is configured to perform standardization processing on the multi-modal state feature vector, map the multi-modal state feature vector to a unified semantic space through linear transformation to obtain a feature representation, calculate a modal confidence of the feature representation, construct a cross-modal attention weight distribution based on the modal confidence, and generate a multi-modal fusion feature by weighting and aggregating the feature representation according to the cross-modal attention weight distribution. The behavior prediction module is configured to input the multi-modal fusion feature into a neural network classifier, extract a static semantic feature through global average pooling, extract a timing dynamic feature through a gated recurrent network, and generate an abnormal behavior recognition result based on feature splicing of the static semantic feature and the timing dynamic feature.

[0135] The embodiment also provides an electronic device suitable for the power industry social engineering attack recognition method based on multi-modal feature fusion, which includes a memory and a processor.

[0136] The embodiment also provides a storage medium having a computer program stored thereon, the program being executed by a processor to implement the power industry social engineering attack recognition method based on multi-modal feature fusion proposed in the above embodiment.

[0137] The storage medium proposed in the embodiment and the power industry social engineering attack recognition method based on multi-modal feature fusion proposed in the above embodiment belong to the same inventive concept, and the technical details not described in the embodiment can be referred to the above embodiment, and the embodiment has the same beneficial effects as the above embodiment.

[0138] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary universal hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a floppy disk, a read-only memory (ROM), a random access memory (RAM), a FLASH, a hard disk, or an optical disc, and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of various embodiments of the present application.

[0139] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application, and although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application, and all of them should be covered in the scope of the claims of the present application.

Claims

1. A method for identifying social engineering attacks in the power industry based on multimodal feature fusion, characterized by: include, Acquire multi-source heterogeneous data from system users, including text data, audio data, and log data, and preprocess the multi-source heterogeneous data based on data modality differences; Based on the preprocessed multi-source heterogeneous data, extract multimodal state feature vectors; The multimodal state feature vectors are standardized and mapped to a unified semantic space, and weighted fusion is performed based on an attention mechanism to generate multimodal fusion features; The multimodal fusion features are input into a neural network classifier to generate abnormal behavior recognition results.

2. The method for identifying social engineering attacks in the power industry based on multimodal feature fusion as described in claim 1, characterized in that: The preprocessing of the multi-source heterogeneous data based on data modality differences includes: performing word segmentation on text data based on a dictionary, and constructing stop word filtering rules based on word frequency statistics; Effective signal detection is performed on the audio data, and background noise is eliminated using digital filtering methods; The system performs missing value prediction and completion on log data based on historical records, and performs data format standardization processing.

3. The method for identifying social engineering attacks in the power industry based on multimodal feature fusion as described in claim 2, characterized in that: The extraction of multimodal state feature vectors includes obtaining the semantic representation of the text data through a language model, inputting the semantic representation into a psychological perception attention layer, and extracting psychological state feature vectors through psychological weight calculation. The frame-level representation of the audio data is obtained through an audio coding model. The frame-level representation is then input into a bidirectional sequence network for context modeling. Key emotional segments are captured and emotional state feature vectors are extracted through a multi-head attention mechanism. The log data is encoded into a hidden state sequence using a temporal network. The hidden state sequence is then input into a graph neural network for dynamic association. A distance metric method is used to identify abnormal patterns and extract behavioral state feature vectors.

4. The method for identifying social engineering attacks in the power industry based on multimodal feature fusion as described in claim 3, characterized in that: The step of standardizing the multimodal state feature vector and mapping it to a unified semantic space, and performing weighted fusion based on an attention mechanism, includes projecting the feature representation to obtain a query vector. A confidence score is calculated based on the similarity between the query vector and the feature representation. The confidence scores are subjected to probability normalization to obtain the modal confidence distribution.

5. The method for identifying social engineering attacks in the power industry based on multimodal feature fusion as described in claim 4, characterized in that: The standardization of the multimodal state feature vector and its mapping to a unified semantic space, along with the weighted fusion based on the attention mechanism, further includes obtaining the feature attention component by performing a linear transformation and nonlinear activation on the feature representation and then taking the inner product with the weight vector. The confidence enhancement component is obtained by taking the logarithm of the modal confidence and then performing a weighted scaling. The attention score is obtained by adding the feature attention component and the confidence enhancement component, and the attention score is then subjected to exponential normalization to obtain the cross-modal attention weight distribution.

6. The method for identifying social engineering attacks in the power industry based on multimodal feature fusion as described in claim 5, characterized in that: The generated multimodal fusion features include, Weighted feature representations are obtained by performing element-wise multiplication of the attention weights of each modality with the feature representations; Multimodal fusion features are obtained by performing vector summation on the weighted feature representations.

7. The method for identifying social engineering attacks in the power industry based on multimodal feature fusion as described in claim 6, characterized in that: The step of inputting the multimodal fusion features into a neural network classifier to generate abnormal behavior recognition results includes organizing the multimodal fusion features into a time-series matrix by dividing the time window, and calculating the average value of the feature vectors of all time steps in the time-series matrix to obtain static semantic features. The time series matrix is ​​modeled based on a gating mechanism. By updating the historical information retained by the gate selection and resetting the current information fused by the gate selection, the dynamic features of the time series are output. The static semantic features and the temporal dynamic features are concatenated by expanding the feature dimension to form a comprehensive feature vector; The comprehensive feature vector is transformed layer by layer through forward propagation between layers of the neural network. Each layer is processed by weight transformation and activation function to obtain the transformation result. In the output layer, the transformation result is converted into a numerical score for each anomaly category through probability mapping. The abnormal behavior recognition result is determined according to the position of the maximum value of the numerical score.

8. An abnormal behavior recognition system based on multimodal feature fusion, applying the power industry social engineering attack recognition method based on multimodal feature fusion as described in any one of claims 1 to 7, characterized in that, include: The data acquisition module is used to acquire multi-source heterogeneous data from system users, including text data, audio data, and log data, and to preprocess the multi-source heterogeneous data based on data modality differences. The feature extraction module is used to extract multimodal state feature vectors from the behavioral dimension, interaction dimension, and operation dimension based on the preprocessed multi-source heterogeneous data. The feature fusion module is used to standardize the multimodal state feature vectors, map them to a unified semantic space through linear transformation to obtain feature representations, calculate the modality confidence of the feature representations, construct a cross-modal attention weight distribution based on the modality confidence, and perform weighted aggregation on the feature representations according to the cross-modal attention weight distribution to generate multimodal fusion features. The behavior prediction module is used to input the multimodal fusion features into a neural network classifier, extract static semantic features through global average pooling, extract temporal dynamic features through a gated recurrent network, and generate abnormal behavior recognition results based on the feature concatenation of the static semantic features and the temporal dynamic features.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the power industry social engineering attack identification method based on multimodal feature fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the power industry social engineering attack identification method based on multimodal feature fusion as described in any one of claims 1 to 7.

Images