Multi-modal behavior identification method and system for security and protection of transformer substation

Abstract

The invention discloses a substation security-oriented multi-modal behavior identification method and system, and the method comprises the steps: obtaining the motion data, including WiFi channel state information image data and distance-Doppler image data, of personnel inside and outside a substation, obtaining the classification prediction result of the motion of the personnel inside and outside the substation based on a pre-trained behavior identification model, and carrying out the classification prediction of the motion of the personnel inside and outside the substation. Replacing a VGG19 network with a MobileNetv4 network to improve an OpenPose algorithm, and extracting personnel action postures in personnel action video frames inside and outside the transformer substation as a tag training behavior recognition model; and extracting distance-Doppler image data by using a VGG16 network with a convolution attention module, and carrying out feature fusion on the WiFi channel state information image data and the distance-Doppler image data. According to the method, the privacy and data security of the transformer substation are protected to the greatest extent while the personnel behaviors are effectively identified, and the leakage and abuse of sensitive information are effectively avoided.

Description

Technical Field

[0001] The present invention relates to a multimodal behavior recognition method and system for substation security, belonging to the technical field of power system safety monitoring and multimodal data analysis. Background Art

[0002] Substations are a vital link in the power system for energy conversion. Their safe and stable operation is the basis for ensuring the smooth operation of the entire power system. At present, the security monitoring of substations mainly relies on a series of traditional technical means, among which video monitoring occupies a core position. This monitoring method captures real-time images in the substation through cameras, providing an intuitive monitoring perspective for operation and maintenance personnel. However, the existing security monitoring system is often limited to a single video data source. Although it can monitor the substation environment to a certain extent, it is still insufficient in comprehensiveness and accuracy. Video monitoring depends on light conditions. At night or in low-light environments, the monitoring effect will be greatly reduced, and it is difficult to capture all key information. In addition, existing monitoring technologies also try to combine some basic image recognition algorithms to achieve preliminary early warning of abnormal behavior, but this recognition capability is relatively limited and cannot deeply analyze and understand the complex behavior patterns of people in substations.

[0003] Although the existing substation security monitoring technology has improved the monitoring capability of the substation environment to a certain extent, its limitations and challenges cannot be ignored. First, the limitation of a single data source makes it difficult for the monitoring system to fully and accurately capture and analyze the behavior information of personnel in the substation. Although video monitoring is intuitive, it is often unable to provide complete and continuous monitoring records due to factors such as viewing angle and light, resulting in potential security risks being ignored. Secondly, traditional monitoring methods have obvious security risks in the data processing and storage links. The substation contains a large amount of sensitive data, such as equipment operating status, personnel operation records, etc., which are crucial for the safe and stable operation of the power system. However, the existing data processing and storage mechanisms often lack effective encryption and protection measures, making these data extremely easy to leak or be maliciously used. In particular, when image data is directly used for real-time recognition, it may not only expose the privacy information of the substation, but also greatly increase the risk of data leakage, posing a serious threat to the security of the power system. Therefore, the existing substation security monitoring technology needs to be improved urgently to overcome these defects and improve the comprehensiveness and security of monitoring. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal behavior recognition method and system for substation security, which can more accurately identify human behavior by obtaining distance-Doppler image data to determine the distance and speed of a human body relative to a radar and WiFi channel status information image data. In addition, the video frames of human actions inside and outside the substation obtained in the process of the present invention only do not directly participate in the real-time recognition process of the behavior recognition model. While effectively identifying human behavior, the present invention protects the privacy and data security of the substation to the maximum extent, and effectively avoids the leakage and abuse of sensitive information.

[0005] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0006] In a first aspect, the present invention provides a multimodal behavior recognition method for substation security, comprising:

[0007] Obtain the motion data of personnel inside and outside the substation, including WiFi channel status information image data and range-Doppler image data;

[0008] According to the action data of the personnel inside and outside the substation, based on the pre-trained behavior recognition model, abnormal behavior recognition is performed to obtain the classification prediction results of the actions of the personnel inside and outside the substation;

[0009] The behavior recognition model uses the improved OpenPose algorithm and VideoPose3D algorithm to extract the action postures of people in the video frames of the actions of people inside and outside the substation as labels for training;

[0010] The abnormal behavior recognition includes using a VGG16 network with a convolutional attention module to extract features from the range-Doppler image data, and then fusing the features of the WiFi channel status information image data with the extracted features of the range-Doppler image data.

[0011] Furthermore, before obtaining the motion data of personnel inside and outside the substation, the method further includes:

[0012] An optical camera for acquiring video frames of movements of personnel inside and outside the substation and a WIFI signal receiver for acquiring image data of WiFi channel status information are connected through the TCP protocol, and a hardware clock is synchronized using the network time protocol to synchronize the video frames of movements of personnel inside and outside the substation and the image data of WiFi channel status information;

[0013] The millimeter wave radar of the distance-Doppler image data is connected to the WIFI signal receiver for obtaining the WiFi channel status information image data via Ethernet, and the PTP protocol is used to adjust the hardware clock according to the timestamp difference to synchronize the WiFi channel status information image data and the distance-Doppler image data.

[0014] Furthermore, the method for acquiring the WiFi channel status information image data includes:

[0015] Before acquiring the WiFi channel status information image data, the frequency of the WiFi channel status signal is set to be much higher than the electromagnetic environment frequency of the substation for frequency isolation to avoid environmental interference;

[0016] By using the improved independent component analysis algorithm, the WiFi channel status signal is separated from the mixed signal of the WiFi channel status signal and the substation electromagnetic field interference signal received by the WiFi signal receiver, and the substation electromagnetic field interference signal is suppressed to obtain the WiFi channel status information image data.

[0017] Furthermore, the improved independent component analysis algorithm is used to separate the WiFi channel status signal from the mixed signal of the WiFi channel status signal and the substation electromagnetic field interference signal received by the WiFi signal receiver, and suppress the substation electromagnetic field interference signal, so as to obtain the WiFi channel status information image data, including:

[0018] Negative entropy is used as the objective function to measure independence;

[0019] By using the improved Newton iterative correction formula, the separation matrix is ​​calculated based on the objective function by finding the maximum non-Gaussian direction to maximize the statistical independence of the WiFi channel state signal and the substation electromagnetic field interference signal, thereby obtaining the separation matrix;

[0020] The separation matrix is ​​used to separate the WiFi channel status signal from the mixed signal, while suppressing the influence of the electromagnetic field interference signal of the substation.

[0021] Furthermore, the expression of the WiFi channel status signal is:

[0022] ;

[0023] In the formula, Indicates the WiFi channel status signal. represents the separation matrix, The mixed signal representing the WiFi channel status signal and the substation electromagnetic field interference signal is processed by whitening;

[0024] The approximate formula of the negative entropy is:

[0025] J ( W ) ≈ [ E { G ( W T S )} − E { G ( V )}] 2 ;

[0026] In the formula, Represents the separation matrix The negative entropy, , Respectively and expectations, In order to evaluate the deviation of the WiFi channel state signal after separation matrix transformation from the Gaussian distribution in the negative entropy formula, a non-quadratic function is used to assist in determining the independence of the WiFi channel state signal. is the separation matrix, is the mixed signal of WiFi channel status signal and substation electromagnetic field interference signal after whitening processing. It means that after the separation matrix The transposed matrix of After transformation Performing non-quadratic functions The operation of , represents the hyperbolic cosine function, represents the logarithmic function, represents a special Gaussian random vector with mean 0 and The covariance matrix of is the same as Represents the separation matrix The transposed matrix of .

[0027] Furthermore, the improved Newton iterative correction formula is used to calculate the separation matrix based on the objective function by finding the maximum non-Gaussian direction to maximize the statistical independence of the WiFi channel state signal and the substation electromagnetic field interference signal, and the separation matrix is ​​obtained, including:

[0028] Performing a decentralization operation on the mixed signal of the WiFi channel status signal and the substation electromagnetic field interference signal so that the mixed signal has a zero mean characteristic, thereby obtaining a mixed signal with a zero mean characteristic;

[0029] Performing a whitening operation on the mixed signal with the zero mean characteristic, converting the covariance matrix of the mixed signal with the zero mean characteristic into a unit matrix, and obtaining a whitened mixed signal;

[0030] Divide the whitened mixed signal into several independent components that need to be estimated, and divide the independent components into four parts based on the human behavior and the electromagnetic field interference factors of the substation, wherein the first part of the independent components is used to describe the basic human actions, the second part of the independent components is used for complex and dangerous actions, the third part of the independent components is used for action combination and transition, and the fourth part of the independent components is used to capture the basic characteristics of electromagnetic interference, and a convergence threshold is preset to determine whether the iterative process converges;

[0031] The improved Newton iterative correction formula is used to continuously iterate and update the separation matrix based on the objective function and a number of independent components that need to be estimated until a convergence condition is met, thereby obtaining a separation matrix.

[0032] Furthermore, the expression of the improved Newton iterative correction formula is:

[0033] ;

[0034] In the formula, It represents the estimated value of the separation matrix at the m+1th iteration, which is used to describe the variable state of the improved Newton iterative correction formula at different stages of the iteration process. express Negative entropy function A derivation of express Negative entropy function A derivative of The square of express Negative entropy function The second derivative of , where , is the first intermediate variable introduced in the calculation process of the improved Newton iterative correction formula, express Negative entropy function A derivative of , where , is the second intermediate variable introduced in the calculation process of the improved Newton iterative correction formula, Represents the estimated value of the separation matrix at the mth iteration Negative entropy function A derivative of ; , is the third intermediate variable introduced in the calculation process of the improved Newton iterative correction formula, Indicates the number of iterations.

[0035] Furthermore, the improved OpenPose algorithm and VideoPose3D algorithm are used to extract the action postures of personnel in the video frames of personnel actions inside and outside the substation as labels to train the behavior recognition model, including:

[0036] Use MobileNetv4 network to replace VGG19 network to improve OpenPose algorithm;

[0037] Using the improved OpenPose algorithm to detect the positions of the human action postures in the human action video frames inside and outside the substation, and obtain two-dimensional posture information and time dimension information, wherein the two-dimensional posture information includes the two-dimensional coordinates of the two-dimensional joints of the human body;

[0038] The VideoPose3D algorithm is used to detect the two-dimensional posture information and use the dilated time domain convolution method to simultaneously process the two-dimensional posture information and the time dimension information to obtain the position coordinates of the 3D key skeleton points of the human body, where the 3D key skeleton points of the human body include the head, left and right shoulders, left and right elbows, left and right wrists, hips, left and right knees, and left and right ankles of the personnel inside and outside the substation;

[0039] According to the position coordinates of the 3D key skeleton points of the human body, sequence labels of 3D key skeleton points of the human body with different action postures are constructed to train the behavior recognition model. The different action postures include walking, standing upright, observing and recording, squatting, climbing, bending over, falling down and crossing the line.

[0040] Further, the range-Doppler image data is extracted using a VGG16 network with a convolutional attention module, and feature fusion is performed on the WiFi channel state information image data and the range-Doppler image data, including:

[0041] Insert a convolutional attention module at the end of the first convolutional module to the fourth convolutional module of the VGG16 network to obtain a VGG16 network with a convolutional attention module;

[0042] Extracting the range-Doppler image data using a VGG16 network with a convolutional attention module to obtain range-Doppler image features;

[0043] Extract the WiFi channel status information image data using a VGG16 network to obtain WiFi channel status information image data features;

[0044] The WiFi channel state information image data features and the range-Doppler image features are spliced ​​and fused using two fully connected layers to obtain fused features, and the fused features are converted into position coordinates of 3D key bone points of the human body.

[0045] In a second aspect, the present invention provides a multimodal behavior recognition system for substation security, comprising:

[0046] A data acquisition module is used to acquire the motion data of personnel inside and outside the substation, including WiFi channel status information image data and range-Doppler image data;

[0047] A posture estimation module, used to identify abnormal behaviors based on the motion data of personnel inside and outside the substation and based on a pre-trained behavior recognition model;

[0048] Data output module, used to obtain the classification prediction results of the actions of personnel inside and outside the substation;

[0049] The behavior recognition model uses the improved OpenPose algorithm and VideoPose3D algorithm to extract the action postures of people in the video frames of the actions of people inside and outside the substation as labels for training;

[0050] The abnormal behavior recognition includes using a VGG16 network with a convolutional attention module to extract features from the range-Doppler image data, and then fusing the features of the WiFi channel status information image data with the extracted features of the range-Doppler image data.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. The present invention obtains WiFi channel status information image data and distance-Doppler image data of personnel inside and outside the substation comprehensively, and performs abnormal behavior recognition based on a pre-trained behavior recognition model. It can achieve high-precision classification prediction of the actions of personnel inside and outside the substation, including walking, standing upright, observing and recording, squatting, climbing, bending over, falling down, and crossing the line to intrude, etc., which effectively improves the security monitoring level of the substation. While effectively identifying the behavior of personnel, it maximizes the protection of the privacy and data security of the substation, and effectively avoids the leakage and abuse of sensitive information.

[0053] 2. Before acquiring the action data of personnel inside and outside the substation, the present invention synchronizes the hardware clocks of the optical camera and the WIFI signal receiver, the millimeter-wave radar and the WIFI signal receiver respectively through the TCP protocol and the PTP protocol, thereby ensuring the precise time alignment between the action video frames of personnel inside and outside the substation, the WiFi channel status information image data and the distance-Doppler image data. This efficient data synchronization mechanism effectively avoids the recognition error caused by time asynchrony, and provides a more accurate and reliable data basis for subsequent behavior recognition. At the same time, the feature fusion of the WiFi channel status information image data and the distance-Doppler image data further enhances the recognition ability of the behavior recognition model for complex scenes and actions, and improves the accuracy and robustness of recognition.

[0054] 3. When acquiring the WiFi channel status information image data, the present invention effectively suppresses the interference of the substation electromagnetic environment on the WiFi signal through frequency isolation and improved independent component analysis algorithm. The frequency isolation setting ensures the purity of the WiFi channel status signal, and the improved independent component analysis algorithm can accurately separate the WiFi channel status signal from the mixed signal, further enhancing the anti-interference ability of the system. This optimization process not only improves the accuracy and reliability of the WiFi channel status information image data, but also provides clearer and more effective data support for subsequent behavior recognition.

[0055] 4. The present invention uses the improved OpenPose algorithm and VideoPose3D algorithm to accurately extract the postures of people in the video frames of people's actions inside and outside the substation, and use them as labels to train the behavior recognition model. At the same time, the distance-Doppler image data is extracted through the VGG16 network with a convolutional attention module, and the feature fusion is performed with the WiFi channel status information image data, realizing the effective integration and utilization of multimodal data. This precise human motion posture estimation and multimodal data fusion method not only improves the accuracy and efficiency of behavior recognition, but also provides a more comprehensive and detailed behavior monitoring and analysis capability for substation security, which helps to timely discover and deal with potential safety hazards. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 The figure is a flow chart of a multi-modal behavior recognition method for substation security provided by an embodiment of the present invention;

[0057] Figure 2 The figure is a schematic diagram of the prediction results of 3D key skeleton points of a human body provided by an embodiment of the present invention;

[0058] Figure 3 The figure shows a schematic diagram of actual results of 3D key skeleton points of a human body provided by an embodiment of the present invention;

[0059] Figure 4 The figure is a schematic flow chart of iterative separation matrix estimation using the improved Newton iterative correction formula provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0061] The term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " generally indicates that the related objects are in an "or" relationship.

[0062] Example 1

[0063] This embodiment introduces a multimodal behavior recognition method for substation security, including:

[0064] Obtain the motion data of personnel inside and outside the substation, including WiFi channel status information image data and range-Doppler image data.

[0065] The present invention can infer information such as the movement and position of an object by analyzing the changes in WiFi signals during propagation, including phase and amplitude, etc. It has the advantages of being non-contact and having a wide coverage range, and is suitable for monitoring personnel activities inside and outside substations.

[0066] The present invention utilizes radar technology to analyze the speed, direction and distance of an object by transmitting and receiving electromagnetic waves according to the Doppler effect of the echo. Radar technology can penetrate obstacles and is suitable for monitoring in complex environments.

[0067] The present invention can comprehensively and accurately capture the motion data of personnel inside and outside the substation by acquiring WiFi channel status information image data and distance-Doppler image data.

[0068] According to the action data of the personnel inside and outside the substation, abnormal behavior recognition is performed based on a pre-trained behavior recognition model to obtain classification prediction results of the actions of the personnel inside and outside the substation.

[0069] Using deep learning algorithms, such as convolutional neural networks (CNNs), we can build a behavior recognition model that can recognize multiple behaviors. The behavior recognition model can learn to map the input action data to specific behavior categories through training with a large amount of labeled data.

[0070] The present invention utilizes a behavior recognition model to automatically and quickly determine whether the behavior of personnel inside and outside the substation is abnormal, thereby improving monitoring efficiency.

[0071] The classification prediction results include walking, standing upright, observing and recording, squatting, climbing over, climbing, bending over, falling down and crossing the line intrusion, which helps to clarify the behavioral status of people inside and outside the substation and facilitates monitoring personnel to quickly respond to abnormal situations.

[0072] Among them, the MobileNetv4 network is used to replace the VGG19 network to improve the OpenPose algorithm. The improved OpenPose algorithm and the VideoPose3D algorithm are used to extract the movement postures of people in the video frames of the movements of people inside and outside the substation as labels to train the behavior recognition model.

[0073] MobileNetv4 is a lightweight convolutional neural network with fewer parameters and computational complexity while maintaining a high accuracy. The OpenPose algorithm is an algorithm for human posture estimation that can accurately identify key points of the human body, such as joints. The present invention can reduce the computational complexity of the model and improve the operating efficiency of the algorithm by replacing the VGG19 network used in the OpenPose algorithm with MobileNetv4. The improved OpenPose algorithm can increase the operating speed while maintaining accuracy, and is suitable for real-time monitoring scenarios.

[0074] The VideoPose3D algorithm is an algorithm for extracting three-dimensional human posture from video. It uses the results of two-dimensional human posture estimation, combined with time information and deep learning algorithms, to reconstruct three-dimensional human posture. By extracting three-dimensional human posture, the present invention can more accurately describe the movements of people and improve the accuracy of behavior recognition.

[0075] The present invention extracts the personnel action postures in the video frames of personnel actions inside and outside the substation only as labels to train the behavior recognition model. While effectively identifying personnel behavior, it maximizes the protection of the privacy and data security of the substation and effectively avoids the leakage and abuse of sensitive information.

[0076] The range-Doppler image data is extracted using a VGG16 network with a convolutional attention module, and feature fusion is performed on the WiFi channel state information image data and the range-Doppler image data.

[0077] The VGG16 network is a classic convolutional neural network architecture that can extract multi-level features from images. The convolutional attention module allows the model to focus on more important parts when processing input data. The convolutional attention module introduces an attention mechanism, which enables the VGG16 network to focus more on key information in the image, such as human posture and motion trajectory, when extracting range-Doppler image data.

[0078] The present invention uses a VGG16 network with a convolutional attention module to more accurately extract key information in range-Doppler images, such as human posture, motion trajectory, etc., which helps to more accurately identify human movements.

[0079] The present invention fuses the features of WiFi channel status information image data and range-Doppler image data, so that the behavior recognition model can simultaneously utilize the movement and position information of the object as well as the visual information, learn a more comprehensive and rich feature representation, and thus more accurately identify the behavior of people inside and outside the substation, thereby improving the safety and stability of the substation.

[0080] Example 2

[0081] Based on the same inventive concept as in Example 1, Figure 1 As shown, this embodiment introduces a multimodal behavior recognition method for substation security, which specifically includes the following steps:

[0082] Step 1: Data Collection

[0083] Use WIFI signal transmitters and WIFI signal receivers to collect WiFi channel status information data to obtain amplitude and phase information including human movements; use millimeter wave radar to collect millimeter wave radar signal data to obtain distance and speed information of human movements; and use optical cameras to extract video frame data of human movements inside and outside the substation.

[0084] The WiFi channel status information data is processed to obtain WiFi channel status information image data; the millimeter wave radar signal data is processed to obtain distance-Doppler image data and the movement postures of personnel in the video frames of personnel movements inside and outside the substation are extracted.

[0085] In this embodiment, the WIFI signal transmitter uses three antennas, and the WIFI signal receiver uses three antennas to form a 3×3 mode transmission-reception link. The WIFI signal receiver can be a laptop computer with a network card installed, running an operating system such as Linux. The data synchronization of the WIFI signal receiver is achieved through the network time protocol, and the receiver is connected to the wireless network created by the transmitter.

[0086] In this embodiment, the millimeter-wave radar uses an industrial millimeter-wave radar as a sensor to obtain the distance and speed information of human movements. The collected original echo data is subjected to a two-dimensional fast Fourier transform to obtain millimeter-wave radar signal data. The millimeter-wave radar signal data can display the information on the distance and speed dimensions of the movements of people inside and outside the substation, forming dynamic feature data.

[0087] Step 2: Data Processing

[0088] Acquiring the motion data of the personnel inside and outside the substation, including the WiFi channel status information image data and the distance-Doppler image data. In this embodiment, before acquiring the motion data of the personnel inside and outside the substation, the following steps are included:

[0089] The optical camera for obtaining the video frames of the movements of people inside and outside the substation and the WIFI signal receiver for obtaining the image data of the WiFi channel status information are connected through the TCP protocol, and the network time protocol is used to synchronize the hardware clock to synchronize the video frames of the movements of people inside and outside the substation and the image data of the WiFi channel status information, ensuring that the synchronization error is within milliseconds.

[0090] The millimeter wave radar of the distance-Doppler image data is connected to the WIFI signal receiver for obtaining the WiFi channel status information image data via Ethernet, and the PTP protocol is used to adjust the hardware clock according to the timestamp difference to synchronize the WiFi channel status information image data and the distance-Doppler image data.

[0091] Before acquiring the WiFi channel status information image data, the method further includes:

[0092] The frequency of the WiFi channel status signal is set to be much higher than the electromagnetic environment frequency of the substation for frequency isolation to avoid environmental interference. In this embodiment, the frequency of the WiFi channel status signal is set to 5.8 GHz, and the electromagnetic environment frequency of the substation is lower than 50 Hz, which can effectively avoid environmental interference.

[0093] By using the improved independent component analysis algorithm, the WiFi channel status signal is separated from the mixed signal of the WiFi channel status signal and the substation electromagnetic field interference signal received by the WiFi signal receiver, and the substation electromagnetic field interference signal is suppressed to obtain the WiFi channel status information image data.

[0094] In this embodiment, the expression of the WiFi channel status signal is:

[0095] ;

[0096] In the formula, Indicates the WiFi channel status signal. represents the separation matrix, The mixed signal representing the WiFi channel status signal and the substation electromagnetic field interference signal is processed by whitening;

[0097] The approximate formula of the negative entropy is:

[0098] J ( W ) ≈ [ E { G ( W T S )} − E { G ( V )}] 2 ;

[0099] In the formula, Represents the separation matrix The negative entropy, , Respectively and expectations, In order to evaluate the deviation of the WiFi channel state signal after separation matrix transformation from the Gaussian distribution in the negative entropy formula, a non-quadratic function is used to assist in determining the independence of the WiFi channel state signal. is the separation matrix, is the mixed signal of WiFi channel status signal and substation electromagnetic field interference signal after whitening processing. It means that after the separation matrix The transposed matrix of After transformation Performing non-quadratic functions The operation of , represents the hyperbolic cosine function, represents the logarithmic function, represents a special Gaussian random vector with mean 0 and The covariance matrix of is the same as Represents the separation matrix The transposed matrix of .

[0100] is a non-quadratic function of any form, used to evaluate the degree of deviation of the WiFi channel state signal after separation matrix transformation from the Gaussian distribution, and to assist in determining the independence of the WiFi channel state signal. , represents the hyperbolic cosine function, represents the logarithmic function, represents a special Gaussian random vector with mean 0 and The covariance matrix of is the same as Represents the separation matrix The transposed matrix of .

[0101] is a non-quadratic function of any form, express , that is, the signal after whitening of the transpose of the separation matrix W and the mixed signal of the WiFi channel status signal and the substation electromagnetic field interference signal The product of After the separation matrix is ​​transformed, a new signal representation is obtained, namely the WiFi channel status signal, so that it can be subsequently used through the function To evaluate the deviation of the WiFi channel status signal from the Gaussian distribution, thus assisting in determining the independence of the WiFi channel status signal. The function value of is large, indicating It deviates far from the Gaussian distribution, and we can infer that After separation matrix After the transformation, it has a strong non-Gaussian property, which helps to determine whether the separation matrix W at this time makes the statistical independence of the WiFi channel status signal and the substation electromagnetic field interference signal reach a large degree, that is, whether it is a better separation matrix.

[0102] In this embodiment, an improved independent component analysis algorithm is used to separate the WiFi channel status signal from the mixed signal of the WiFi channel status signal and the substation electromagnetic field interference signal received by the WiFi signal receiver, and suppress the substation electromagnetic field interference signal to obtain WiFi channel status information image data, including:

[0103] Negative entropy is used as the objective function to measure independence;

[0104] By using the improved Newton iterative correction formula, the separation matrix is ​​calculated based on the objective function by finding the maximum non-Gaussian direction to maximize the statistical independence of the WiFi channel state signal and the substation electromagnetic field interference signal, thereby obtaining the separation matrix;

[0105] The separation matrix is ​​used to separate the WiFi channel status signal from the mixed signal, while suppressing the influence of the electromagnetic field interference signal of the substation.

[0106] like Figure 4 As shown, in this embodiment, the improved Newton iterative correction formula is used to calculate the separation matrix based on the objective function by finding the maximum non-Gaussian direction to maximize the statistical independence of the WiFi channel state signal and the substation electromagnetic field interference signal, and obtain the separation matrix, including:

[0107] A decentralized operation is performed on a mixed signal of the WiFi channel status signal and the transformer substation electromagnetic field interference signal so that the mixed signal has a zero mean characteristic, thereby obtaining a mixed signal with a zero mean characteristic.

[0108] In this embodiment, the observation signal is a mixed signal of the WiFi channel status signal and the substation electromagnetic field interference signal. First, we need to decentralize the mixed signal X so that it has zero mean. Suppose the total number of observation vectors is , then the mean of the observed data is ,in is the i-th observation vector, the decentralized data , and a mixed signal with zero mean characteristics is obtained.

[0109] Computing decentralized data The covariance matrix of , where E represents the mathematical expectation, where Represents the transposed matrix of decentralized data, represents the transpose; then the covariance matrix Perform eigenvalue decomposition and get ,in is a matrix consisting of eigenvectors, It is a diagonal matrix, the elements on the diagonal are the eigenvalues, and then the whitening matrix is ​​obtained The whitening process is to decentralize the mixed signal Perform linear transformation to obtain the whitened mixed signal ,Right now .

[0110] The whitened mixed signal is divided into several independent components that need to be estimated. The independent components are divided into four parts based on personnel behavior and substation electromagnetic field interference factors. The first part of independent components is used to describe basic personnel actions, the second part of independent components is used for complex and dangerous actions, the third part of independent components is used for action combination and transition, and the fourth part of independent components is used to capture the basic characteristics of electromagnetic interference. At the same time, a convergence threshold is preset to determine whether the iterative process converges.

[0111] In this embodiment, the whitened mixed signal is divided into 12 independent components that need to be estimated, of which 3 independent components are used to describe basic personnel actions, 5 independent components are used for complex and dangerous actions, 2 independent components are used for action combinations and transitions, and 2 independent components are used to capture the basic characteristics of electromagnetic interference. This can ensure a balance between accurate description of personnel behavior and effective suppression of electromagnetic interference to a certain extent, and the convergence threshold is preset to 0.001.

[0112] The improved Newton iterative correction formula is used to continuously iterate and update the separation matrix based on the objective function and a number of independent components that need to be estimated until a convergence condition is met, thereby obtaining a separation matrix.

[0113] In this embodiment, the expression of the improved Newton iteration correction formula is:

[0114] ;

[0115] In the formula, It represents the estimated value of the separation matrix at the m+1th iteration, which is used to describe the variable state of the improved Newton iterative correction formula at different stages of the iteration process. express Negative entropy function A derivation of express Negative entropy function A derivative of The square of express Negative entropy function The second derivative of , where , is the first intermediate variable introduced in the calculation process of the improved Newton iterative correction formula, express Negative entropy function A derivative of , where , is the second intermediate variable introduced in the calculation process of the improved Newton iterative correction formula, Represents the estimated value of the separation matrix at the mth iteration Negative entropy function A derivative of ; , is the third intermediate variable introduced in the calculation process of the improved Newton iterative correction formula, Indicates the number of iterations.

[0116] According to the negative entropy approximation formula and the improved Newton iterative correction formula, we can know that:

[0117] Negative Entropy The maximum value of The extreme point of is obtained because is a constant, then finding the maximum value of J(W) is equivalent to finding Find the extreme value, set the constraints through the Lagrange multiplier method , introduce auxiliary functions, ,Will The extreme value problem of is transformed into solving =0 problem.

[0118] Helper Functions ,in for The derivative of , is the initial value of W, and They are The first and second derivatives of are calculated as follows:

[0119] (1) Based on the improved Newton iterative correction formula, the updated weight vector is obtained ,calculate , Indicates the number of independent components currently estimated. Indicates the number of iterations, and then Normalization is performed, that is, , so that the modulus of the weight vector is 1, controlling the size of the weight vector and avoiding numerical instability during the iteration process.

[0120] (2) Judgment Whether the preset convergence conditions are met If the convergence condition is met, the iteration is stopped; otherwise, the iterative calculation step is returned to continue updating the weight vector until the convergence condition is met. The judgment of the convergence condition is to ensure that the algorithm can stop the iteration at a suitable accuracy and obtain a more accurate unmixing matrix W.

[0121] (3) Order ,if , return to the step of selecting the initial weight vector in the initialization step, and continue to estimate the next independent component. Through this cyclic process, all independent components that need to be estimated are gradually extracted, and finally a complete unmixing matrix is ​​obtained, which can separate the original transmission signal from the mixed signal containing electromagnetic interference in the substation scenario.

[0122] According to the motion data of the personnel inside and outside the substation, abnormal behavior recognition is performed based on a pre-trained behavior recognition model to obtain classification prediction results of the movements of the personnel inside and outside the substation. The classification prediction results include walking, standing upright, observing and recording, squatting, climbing over, climbing, bending over, falling down and crossing the line.

[0123] The improved OpenPose algorithm and VideoPose3D algorithm are used to extract the action postures of personnel in the video frames of personnel actions inside and outside the substation as labels to train the behavior recognition model, including:

[0124] Use MobileNetv4 network to replace VGG19 network to improve OpenPose algorithm;

[0125] Using the improved OpenPose algorithm to detect the positions of the human action postures in the human action video frames inside and outside the substation, and obtain two-dimensional posture information and time dimension information, wherein the two-dimensional posture information includes the two-dimensional coordinates of the two-dimensional joints of the human body;

[0126] The VideoPose3D algorithm is used to detect the two-dimensional posture information and use the dilated time domain convolution method to simultaneously process the two-dimensional posture information and the time dimension information to obtain the position coordinates of the 3D key skeleton points of the human body, where the 3D key skeleton points of the human body include the head, left and right shoulders, left and right elbows, left and right wrists, hips, left and right knees, and left and right ankles of the personnel inside and outside the substation;

[0127] According to the position coordinates of the 3D key skeleton points of the human body, sequence labels of 3D key skeleton points of the human body with different action postures are constructed to train the behavior recognition model. The different action postures include walking, standing upright, observing and recording, squatting, climbing, bending over, falling down and crossing the line.

[0128] In this embodiment, the behavior recognition model is constructed using a neural network based on the TensorFlow framework, and after multiple rounds of training, each round batches the data and randomly shuffles the data before input. The loss function for training the behavior recognition model is a mean square error loss function, expressed as:

[0129] ;

[0130] In the formula, represents the loss value of the behavior recognition model, represents the total number of training samples, Indicates The true value of the position coordinates of the 3D key skeleton points of the human body of the training sample, Indicates The predicted values ​​of the position coordinates of the 3D key skeleton points of the human body of the training sample. The schematic diagram of the predicted results and the actual results of the 3D key skeleton points of the human body provided in this embodiment are as follows: Figure 2 and Figure 3 shown.

[0131] The method comprises: extracting the range-Doppler image data using a VGG16 network with a convolutional attention module, and performing feature fusion on the WiFi channel state information image data and the range-Doppler image data, including:

[0132] Insert a convolutional attention module at the end of the first convolutional module to the fourth convolutional module of the VGG16 network to obtain a VGG16 network with a convolutional attention module;

[0133] Extracting the range-Doppler image data using a VGG16 network with a convolutional attention module to obtain range-Doppler image features;

[0134] Extract the WiFi channel status information image data using a VGG16 network to obtain WiFi channel status information image data features;

[0135] The WiFi channel state information image data features and the range-Doppler image features are spliced ​​and fused using two fully connected layers to obtain fused features, and the fused features are converted into position coordinates of 3D key bone points of the human body.

[0136] Example 3

[0137] Based on the same inventive concept as other embodiments, this embodiment introduces a multimodal behavior system for substation security, including:

[0138] The data acquisition module is used to obtain the motion data of personnel inside and outside the substation, including WiFi channel status information image data and distance-Doppler image data.

[0139] The posture estimation module is used to identify abnormal behaviors based on the motion data of personnel inside and outside the substation and based on a pre-trained behavior recognition model.

[0140] Data output module, used to obtain the classification prediction results of the actions of personnel inside and outside the substation;

[0141] The behavior recognition model uses the improved OpenPose algorithm and VideoPose3D algorithm to extract the action postures of people in the video frames of the actions of people inside and outside the substation as labels for training;

[0142] The abnormal behavior recognition includes using a VGG16 network with a convolutional attention module to extract features from the range-Doppler image data, and then fusing the features of the WiFi channel status information image data with the extracted features of the range-Doppler image data.

[0143] The specific functions of the above modules can be found in the method of Example 1.

[0144] Example 3

[0145] Based on the same inventive concept as other embodiments, this embodiment introduces a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the steps of the method in the above-mentioned embodiment 1 or 2 are implemented.

[0146] Example 4

[0147] Based on the same inventive concept as other embodiments, the present invention further provides a computer program product, including computer instructions, which implement the steps of the method in the above-mentioned embodiment 1 or 2 when executed by a processor.

[0148] In summary, the present invention can achieve high-precision classification prediction of the actions of personnel inside and outside the substation, including walking, standing upright, observing and recording, squatting, climbing, bending over, falling down, and crossing the line, etc., by comprehensively acquiring WiFi channel status information image data and distance-Doppler image data of personnel inside and outside the substation, and performing abnormal behavior recognition based on a pre-trained behavior recognition model. The various behaviors effectively improve the security monitoring level of the substation, and while effectively identifying personnel behavior, it maximizes the protection of the privacy and data security of the substation, and effectively avoids the leakage and abuse of sensitive information.

[0149] Before acquiring the action data of personnel inside and outside the substation, the present invention synchronizes the hardware clocks of the optical camera and the WIFI signal receiver, the millimeter-wave radar and the WIFI signal receiver respectively through the TCP protocol and the PTP protocol, thereby ensuring the precise time alignment between the action video frames of personnel inside and outside the substation, the WiFi channel status information image data and the distance-Doppler image data. This efficient data synchronization mechanism effectively avoids the recognition errors caused by time asynchrony, and provides a more accurate and reliable data basis for subsequent behavior recognition. At the same time, the feature fusion of the WiFi channel status information image data and the distance-Doppler image data further enhances the recognition ability of the behavior recognition model for complex scenes and actions, and improves the accuracy and robustness of recognition.

[0150] When acquiring WiFi channel status information image data, the present invention effectively suppresses the interference of the substation electromagnetic environment on the WiFi signal through frequency isolation and improved independent component analysis algorithm. The frequency isolation setting ensures the purity of the WiFi channel status signal, and the improved independent component analysis algorithm can accurately separate the WiFi channel status signal from the mixed signal, further enhancing the anti-interference ability of the system. This optimization process not only improves the accuracy and reliability of the WiFi channel status information image data, but also provides clearer and more effective data support for subsequent behavior recognition.

[0151] The present invention uses the improved OpenPose algorithm and VideoPose3D algorithm to accurately extract the postures of people in the video frames of people's actions inside and outside the substation, and use them as labels to train the behavior recognition model. At the same time, the distance-Doppler image data is extracted through the VGG16 network with a convolutional attention module, and the feature fusion is performed with the WiFi channel status information image data, realizing the effective integration and utilization of multimodal data. This precise human motion posture estimation and multimodal data fusion method not only improves the accuracy and efficiency of behavior recognition, but also provides a more comprehensive and detailed behavior monitoring and analysis capability for substation security, which helps to timely discover and deal with potential safety hazards.

[0152] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0153] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0154] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0156] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A multimodal behavior recognition method for substation security, characterized in that: include: Obtain the motion data of personnel inside and outside the substation, including WiFi channel status information image data and range-Doppler image data; According to the action data of the personnel inside and outside the substation, based on the pre-trained behavior recognition model, abnormal behavior recognition is performed to obtain the classification prediction results of the actions of the personnel inside and outside the substation; The behavior recognition model uses the improved OpenPose algorithm and VideoPose3D algorithm to extract the action postures of people in the video frames of the actions of people inside and outside the substation as labels for training; The abnormal behavior recognition includes using a VGG16 network with a convolutional attention module to extract features from the range-Doppler image data, and then fusing the features of the WiFi channel status information image data with the extracted features of the range-Doppler image data.

2. The multimodal behavior recognition method for substation security according to claim 1 is characterized in that: Before obtaining the motion data of personnel inside and outside the substation, the method further includes: An optical camera for acquiring video frames of movements of personnel inside and outside the substation and a WIFI signal receiver for acquiring image data of WiFi channel status information are connected through the TCP protocol, and a hardware clock is synchronized using the network time protocol to synchronize the video frames of movements of personnel inside and outside the substation and the image data of WiFi channel status information; The millimeter wave radar of the distance-Doppler image data is connected to the WIFI signal receiver for obtaining the WiFi channel status information image data via Ethernet, and the PTP protocol is used to adjust the hardware clock according to the timestamp difference to synchronize the WiFi channel status information image data and the distance-Doppler image data.

3. The multimodal behavior recognition method for substation security according to claim 1 is characterized in that: The method for acquiring the WiFi channel status information image data comprises: Before acquiring the WiFi channel status information image data, the frequency of the WiFi channel status signal is set to be much higher than the electromagnetic environment frequency of the substation for frequency isolation to avoid environmental interference; By using the improved independent component analysis algorithm, the WiFi channel status signal is separated from the mixed signal of the WiFi channel status signal and the substation electromagnetic field interference signal received by the WiFi signal receiver, and the substation electromagnetic field interference signal is suppressed to obtain the WiFi channel status information image data.

4. The multimodal behavior recognition method for substation security according to claim 3 is characterized in that: Using the improved independent component analysis algorithm, the WiFi channel status signal is separated from the mixed signal of the WiFi channel status signal and the substation electromagnetic field interference signal received by the WiFi signal receiver, and the substation electromagnetic field interference signal is suppressed to obtain the WiFi channel status information image data, including: Negative entropy is used as the objective function to measure independence; By using the improved Newton iterative correction formula, the separation matrix is ​​calculated based on the objective function by finding the maximum non-Gaussian direction to maximize the statistical independence of the WiFi channel state signal and the substation electromagnetic field interference signal, thereby obtaining the separation matrix; The separation matrix is ​​used to separate the WiFi channel status signal from the mixed signal, while suppressing the influence of the electromagnetic field interference signal of the substation.

5. The multimodal behavior recognition method for substation security according to claim 4 is characterized in that: The expression of the WiFi channel status signal is: ； In the formula, Indicates the WiFi channel status signal. represents the separation matrix, The mixed signal representing the WiFi channel status signal and the substation electromagnetic field interference signal is processed by whitening; The approximate formula of the negative entropy is: ； In the formula, Represents the separation matrix The negative entropy, , Respectively and expectations, In order to evaluate the deviation of the WiFi channel state signal after separation matrix transformation from the Gaussian distribution in the negative entropy formula, a non-quadratic function is used to assist in determining the independence of the WiFi channel state signal. is the separation matrix, is the mixed signal of WiFi channel status signal and substation electromagnetic field interference signal after whitening processing. It means that after the separation matrix The transposed matrix of After transformation Performing non-quadratic functions The operation of , represents the hyperbolic cosine function, represents the logarithmic function, represents a special Gaussian random vector with mean 0 and The covariance matrix of is the same as Represents the separation matrix The transposed matrix of .

6. The multimodal behavior recognition method for substation security according to claim 4 is characterized in that: By using the improved Newton iterative correction formula, the separation matrix is ​​calculated based on the objective function by finding the maximum non-Gaussian direction to maximize the statistical independence of the WiFi channel state signal and the substation electromagnetic field interference signal, and the separation matrix is ​​obtained, including: Performing a decentralization operation on the mixed signal of the WiFi channel status signal and the substation electromagnetic field interference signal so that the mixed signal has a zero mean characteristic, thereby obtaining a mixed signal with a zero mean characteristic; Performing a whitening operation on the mixed signal with the zero mean characteristic, converting the covariance matrix of the mixed signal with the zero mean characteristic into a unit matrix, and obtaining a whitened mixed signal; Divide the whitened mixed signal into several independent components that need to be estimated, and divide the independent components into four parts based on the human behavior and the electromagnetic field interference factors of the substation, wherein the first part of the independent components is used to describe the basic human actions, the second part of the independent components is used for complex and dangerous actions, the third part of the independent components is used for action combination and transition, and the fourth part of the independent components is used to capture the basic characteristics of electromagnetic interference, and a convergence threshold is preset to determine whether the iterative process converges; The improved Newton iterative correction formula is used to continuously iterate and update the separation matrix based on the objective function and a number of independent components that need to be estimated until a convergence condition is met, thereby obtaining a separation matrix.

7. The multimodal behavior recognition method for substation security according to claim 6 is characterized in that: The expression of the improved Newton iteration correction formula is: ； In the formula, It represents the estimated value of the separation matrix at the m+1th iteration, which is used to describe the variable state of the improved Newton iterative correction formula at different stages of the iteration process. express Negative entropy function A derivation of express Negative entropy function A derivative of The square of express Negative entropy function The second derivative of , where , is the first intermediate variable introduced in the calculation process of the improved Newton iterative correction formula, express Negative entropy function A derivative of , where , is the second intermediate variable introduced in the calculation process of the improved Newton iterative correction formula, Represents the estimated value of the separation matrix at the mth iteration Negative entropy function A derivative of ; , is the third intermediate variable introduced in the calculation process of the improved Newton iterative correction formula, Indicates the number of iterations.

8. The multimodal behavior recognition method for substation security according to claim 1 is characterized in that: The improved OpenPose algorithm and VideoPose3D algorithm are used to extract the postures of personnel in the video frames of personnel movements inside and outside the substation as labels to train the behavior recognition model, including: Use MobileNetv4 network to replace VGG19 network to improve OpenPose algorithm; Using the improved OpenPose algorithm to detect the positions of the human action postures in the human action video frames inside and outside the substation, and obtain two-dimensional posture information and time dimension information, wherein the two-dimensional posture information includes the two-dimensional coordinates of the two-dimensional joints of the human body; The VideoPose3D algorithm is used to detect the two-dimensional posture information and use the dilated time domain convolution method to simultaneously process the two-dimensional posture information and the time dimension information to obtain the position coordinates of the 3D key skeleton points of the human body, where the 3D key skeleton points of the human body include the head, left and right shoulders, left and right elbows, left and right wrists, hips, left and right knees, and left and right ankles of the personnel inside and outside the substation; According to the position coordinates of the 3D key skeleton points of the human body, sequence labels of 3D key skeleton points of the human body with different action postures are constructed to train the behavior recognition model. The different action postures include walking, standing upright, observing and recording, squatting, climbing, bending over, falling down and crossing the line.

9. The multimodal behavior recognition method for substation security according to claim 8 is characterized in that: The range-Doppler image data is extracted using a VGG16 network with a convolutional attention module, and feature fusion is performed on the WiFi channel state information image data and the range-Doppler image data, including: Insert a convolutional attention module at the end of the first convolutional module to the fourth convolutional module of the VGG16 network to obtain a VGG16 network with a convolutional attention module; Extracting the range-Doppler image data using a VGG16 network with a convolutional attention module to obtain range-Doppler image features; Extract the WiFi channel status information image data using a VGG16 network to obtain WiFi channel status information image data features; The WiFi channel state information image data features and the range-Doppler image features are spliced ​​and fused using two fully connected layers to obtain fused features, and the fused features are converted into position coordinates of 3D key bone points of the human body.

10. A multimodal behavior recognition system for substation security, characterized in that: include: A data acquisition module is used to acquire the motion data of personnel inside and outside the substation, including WiFi channel status information image data and range-Doppler image data; A posture estimation module, used to identify abnormal behaviors based on the motion data of personnel inside and outside the substation and based on a pre-trained behavior recognition model; Data output module, used to obtain the classification prediction results of the actions of personnel inside and outside the substation; The behavior recognition model uses the improved OpenPose algorithm and VideoPose3D algorithm to extract the action postures of people in the video frames of the actions of people inside and outside the substation as labels for training; The abnormal behavior recognition includes using a VGG16 network with a convolutional attention module to extract features from the range-Doppler image data, and then fusing the features of the WiFi channel status information image data with the extracted features of the range-Doppler image data.

Images