Substation multi-view scene intrusion tracking method based on visual sense and sound sense

Abstract

The invention discloses a visual sense and sound sense-based intrusion tracking method in a multi-view scene of a transformer substation, and the method comprises the steps: carrying out the feature extraction of a processed image, and obtaining the feature information of a target; identifying and classifying the target by adopting an identification and classification model; an improved DeepSort target tracking algorithm is adopted, similar features and target areas are found in subsequent moving images, and a visual movement track of an intrusion target is obtained; acquiring sound source information of a recognition target area, obtaining a specific position of a recognition target through an improved sound source localization algorithm, and obtaining a sound sense motion track of the target; fusing the visual motion trail and the sound motion trail by adopting a weighted average fusion method to obtain an intrusion target motion trail; according to the method, the motion trail of the invasive organism is obtained according to the image information, the target motion trail is recognized by adopting the improved sound source localization algorithm when the monitoring equipment is shielded, and then the two motion trails are fused to obtain the complete motion trail, so that the accuracy of the motion trails is ensured.

Description

Technical Field

[0001] This invention relates to the field of substation security monitoring technology, and in particular to an intrusion tracking method for multi-view substation scenarios based on vision and sound. Background Technology

[0002] In recent years, the scale of power grid construction has been continuously expanding, and substations, as important power conversion facilities in the power system, play a crucial role. Currently, the number of substations is growing rapidly, and more and more substations are being built in remote areas with complex environments, often inhabited by small animals such as rats, snakes, and birds. Animals entering and moving within equipment can easily cause electrical short circuits, leading to power outages, grid collapses, and even equipment explosions and other accidents with extremely serious consequences. Furthermore, substation secondary equipment carries DC power; animals entering these areas may cause short circuits or DC system grounding, resulting in equipment malfunctions or failures to operate, escalating minor power outages into large-scale incidents and increasing economic losses. Therefore, research on biological intrusion repellent technologies for substations is of great significance for ensuring the safe operation of the power grid and the reliability of power supply. Summary of the Invention

[0003] This invention addresses the shortcomings of existing technologies by providing an intrusion tracking method for substations in multi-view scenarios based on vision and sound. By monitoring and tracking intruding animals based on surveillance images, the method can obtain the movement of intruding animals in real time and ensure the safe operation of the power grid.

[0004] To solve the above-mentioned technical problems, the present invention provides a technical solution: an intrusion tracking method for substations in multi-view scenarios based on vision and sound, the steps of which are:

[0005] S1. Preprocess the substation monitoring images and extract features from the processed images to obtain target feature information;

[0006] S2. Based on the obtained target feature information, the target is identified and classified using a recognition and classification model to obtain the intrusion target feature information;

[0007] S3. Based on the obtained feature information of the intrusion target, the improved DeepSort target tracking algorithm is used to find similar features and target regions in subsequent moving images, perform visual tracking of the intrusion target, and obtain the visual motion trajectory of the intrusion target.

[0008] S4. Based on the acquired intrusion target feature information, obtain the sound source information of the target area, and obtain the specific location of the target through the improved sound source localization algorithm, perform acoustic tracking of the intrusion target, and obtain the target's acoustic motion trajectory.

[0009] S5. A weighted average fusion method is used to fuse visual motion trajectories and auditory motion trajectories to obtain the motion trajectory of the intruding target.

[0010] Furthermore, in step S1, the process of obtaining target feature information is as follows: obtaining frame image information from the video information acquired by the camera; scaling the processed image using nearest neighbor interpolation; denoising the scaled image using mean filtering; enhancing the image using histogram equalization; and extracting image features using support vector machines or binary trees.

[0011] Furthermore, in step S2, the process of identifying and classifying the target is as follows:

[0012] 1) Use web crawlers to obtain images of invasive organisms of different species, postures, and backgrounds, and then manually label them to form a database;

[0013] 2) Divide the labeled database into a training set and a dataset;

[0014] 3) Train the deep convolutional neural network with transfer learning introduced using the training set;

[0015] 4) Validate the trained deep convolutional neural network using a dataset to obtain a recognition and classification model;

[0016] 5) Use the obtained identification and classification model to identify invasive organisms.

[0017] Furthermore, in step S3, the process of improving the DeepSort target tracking algorithm is as follows: an LSTM motion model is introduced into DeepSort, and GIoU matching is used to optimize IoU matching in cascaded matching.

[0018] Furthermore, in cascaded matching, cosine similarity is first calculated, then the average Mahalanobis distance between the target position state and the detection box state is predicted by LSTM to construct a cost matrix, and the corresponding values ​​in the cost matrix with Mahalanobis distance greater than the threshold are set to infinity. Finally, the corresponding values ​​in the cost matrix with cosine distance greater than the threshold are also set to infinity.

[0019] Furthermore, in step S4, the step of obtaining the specific location of the target through the improved sound source localization algorithm is as follows:

[0020] 1) Build an acoustic sensor array receiving model and obtain sound source information through the acoustic sensor array;

[0021] 2) Based on the sound source localization algorithm, the generalized cross-correlation algorithm is integrated to improve the sound source localization algorithm, forming a sound source localization algorithm based on arrival time delay; among them, the complex background noise interference of substations is eliminated by the deep learning-based speech enhancement algorithm, thereby improving the localization and tracking accuracy.

[0022] 3) When the target sound source is not blocked in the scene, the sound wave will directly reach the acoustic sensor array. Finally, the time delay information of each array element is used to solve the acoustic sensor array receiving model to obtain the sound source location.

[0023] 4) When the target sound source is in a non-line-of-sight scene, the transmitted frequency-modulated continuous wave is selected as the sound source signal. Through corresponding signal processing, the time delay information of the reflection path is calculated, and the non-line-of-sight target is located based on the time delay information.

[0024] Furthermore, the sound enhancement process is as follows: within the short-time Fourier spectrum estimation speech enhancement framework, the MMSE-LSA speech enhancement algorithm is combined with a noise spectrum perception method. Based on the prior signal-to-noise ratio estimation of the clean speech signal, a deep learning-based prior signal-to-noise ratio estimation model is combined to ensure that the enhanced speech signal has less noise residue.

[0025] To solve the above-mentioned technical problems, the present invention provides a technical solution: an intrusion tracking system for substations in multi-view scenarios based on vision and sound, characterized by comprising:

[0026] Feature extraction module: preprocesses the substation monitoring images and extracts features from the processed images to obtain target feature information;

[0027] Target recognition module: Based on the obtained target feature information, it uses a recognition and classification model to identify and classify the target, thereby obtaining intrusion target feature information;

[0028] Target tracking module: includes visual trajectory module, acoustic trajectory module and fusion module, wherein:

[0029] Visual trajectory module: Based on the acquired feature information of the intrusion target, the improved DeepSort target tracking algorithm is used to find similar features and target regions in subsequent moving images, perform visual tracking of the intrusion target, and obtain the visual motion trajectory of the intrusion target;

[0030] Acoustic trajectory module: Based on the acquired feature information of the intrusion target, it obtains the sound source information of the target area and uses an improved sound source localization algorithm to obtain the specific location of the target, performs acoustic tracking of the intrusion target, and obtains the acoustic motion trajectory of the target;

[0031] Fusion module: The weighted average fusion method is used to fuse visual motion trajectory and auditory motion trajectory to obtain the motion trajectory of the intruding target.

[0032] To solve the above-mentioned technical problems, the present invention provides a technical solution: an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the method described above.

[0033] To solve the above-mentioned technical problems, the present invention provides a technical solution: a computer-readable storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the steps of the method described above.

[0034] The beneficial effects of this invention are as follows:

[0035] 1. This application utilizes existing monitoring equipment in substations to acquire substation videos and extracts image information from the video information to identify invasive organisms. Then, it obtains the movement trajectory of the invasive organisms based on the image information to track them. However, when the monitoring equipment is obstructed, it is impossible to obtain image information of the invasive organisms. Prediction algorithms also have many uncertainties. In this case, an improved sound source localization algorithm is used to obtain the specific location of the target and acquire the movement trajectory of the invasive organisms when the monitoring equipment is obstructed. Then, the two movement trajectories are fused to obtain a complete movement trajectory, ensuring the accuracy of the movement trajectory.

[0036] 2. This application introduces transfer learning, transferring knowledge learned in the source model to the target model, and using information provided in the source domain to assist the target model in completing the classification task, saving annotation costs and overcoming the dependence of deep learning on large-scale data samples. In terms of target tracking algorithms, to address the problem of insufficient robustness of the DeepSort algorithm under nonlinear conditions, the DeepSort algorithm is improved (LG-DeepSort). In trajectory prediction, a bidirectional LSTM network model is adopted, and contextual information is used to reduce the prediction error generated by the DeepSort algorithm in nonlinear environments. Generalized intersection-over-union (GIoU) ​​matching is used to optimize the intersection-over-union (IoU) matching, reducing the false negative rate of the algorithm and improving the detection accuracy.

[0037] 3. When performing location detection by sound source, this application uses an improved sound source localization algorithm for calculation, and incorporates a sound enhancement algorithm to address the complex background noise in substations. This solves the problem of residual noise in traditional speech enhancement algorithms based on statistical models when the signal-to-noise ratio is low, such as 0dB, thereby improving the positioning accuracy.

[0038] To make the above and other objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only twelve of the drawings in this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 This is a flowchart of the method described in this invention;

[0041] Figure 2 A flowchart for identifying and classifying targets;

[0042] Figure 3 This is a diagram of the LeNet-5 convolutional neural network model structure.

[0043] Figure 4 A flowchart illustrating the transfer learning process;

[0044] Figure 5 A flowchart for target tracking;

[0045] Figure 6 Here is a diagram of the LSTM network structure;

[0046] Figure 7 This is a diagram of the bidirectional LSTM trajectory prediction structure.

[0047] Figure 8 Here is a diagram of the LG-DeepSort algorithm framework;

[0048] Figure 9 This is a schematic diagram for time delay estimation.

[0049] Figure 10 This is a block diagram of a speech enhancement algorithm based on a deep learning-based temporal convolutional network.

[0050] Figure 11 This is a diagram of the TCN network structure.

[0051] Figure 12 This is a ResLSTM network structure. Detailed Implementation

[0052] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0053] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0054] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0055] Example

[0056] like Figure 1-12 As shown, an intrusion tracking method for multi-view substation scenarios based on vision and sound is described, and its steps are as follows:

[0057] S1. Preprocess the substation monitoring images and extract features from the processed images to obtain target feature information;

[0058] S2. Based on the obtained target feature information, the target is identified and classified using a recognition and classification model to obtain the intrusion target feature information;

[0059] S3. Based on the obtained feature information of the intrusion target, the improved DeepSort target tracking algorithm is used to find similar features and target regions in subsequent moving images, perform visual tracking of the intrusion target, and obtain the visual motion trajectory of the intrusion target.

[0060] S4. Based on the acquired intrusion target feature information, obtain the sound source information of the target area, and obtain the specific location of the target through the improved sound source localization algorithm, perform acoustic tracking of the intrusion target, and obtain the target's acoustic motion trajectory.

[0061] S5. A weighted average fusion method is used to fuse visual motion trajectories and auditory motion trajectories to obtain the motion trajectory of the intruding target.

[0062] In step S1, the process of obtaining target feature information is as follows: obtain frame image information from the video information acquired by the camera; scale the processed image using nearest neighbor interpolation, denoise the scaled image using mean filtering, enhance the image using histogram equalization, and extract image features using support vector machine or binary tree.

[0063] Substation on-site remote monitoring cameras are typically located outdoors, and the images captured by these cameras are affected by weather changes, resulting in inconsistent image brightness and noise. Furthermore, due to differences in camera models and the distance at which invasive organisms are captured, the resolution and size of the invasive organisms in the images are not uniform. To improve the accuracy and efficiency of substation invasive organism image recognition, preprocessing of substation images captured by high-definition cameras is usually required.

[0064] Image scaling refers to the process of adjusting the size of a digital image, specifically divided into image reduction and enlargement techniques. Image scaling algorithms use nearest neighbor interpolation to reduce the computational load and improve real-time performance.

[0065] When a camera captures images, variations in lighting direction and intensity, as well as the sensor itself, can introduce noise into the image. Furthermore, noise can also be generated during image digitization and transmission. Noise degrades image quality, especially in images captured in low-light environments, where detailed features can become blurred or even lost due to noise, severely impacting the accuracy of target detection. Therefore, it is necessary to filter and process noise, or to select a detection algorithm that is adaptive and robust to noise. This application employs a mean filtering algorithm.

[0066] Image enhancement refers to the purposeful emphasis of certain overall or local features in an image, such as edges, contours, and contrast, while accepting a certain degree of distortion. Image enhancement can highlight useful features and information for a given image's application scenario, amplify the differences between different target features in the image, and suppress unimportant features, thereby improving detection and recognition performance. In substation intrusion biometrics identification, enhancing the features of substation equipment areas, especially the contour features of various invasive species such as birds, while weakening features of other areas and the background, such as equipment, sky, and walls, can help the detection model more accurately and quickly locate the detection area and learn the features of invasive species.

[0067] Preprocessed images still need feature extraction before they can be detected by a classifier. Support vector machines or binary trees are used for image feature extraction. The main features of an image include edge (shape) features, texture features, and color features.

[0068] Edge features are distinct edges or discontinuous regions in a digital image, typically representing the boundaries between different objects. In invasive species detection, edge features can quickly extract the outlines of substation equipment and invasive species, providing crucial feature information for subsequent classification and detection. Distinctly varying regions in an image manifest as depth discontinuities, illumination gradient discontinuities, and texture variations. Edge detection algorithms identify these distinct areas and generate oriented gradient histograms to construct the image's edge features.

[0069] Texture features are the spatial color and light intensity distribution features of an image or a small area within it. Texture features can help detect partially occluded invasive creatures, while excluding objects with similar shapes, outlines, and invasive creatures (such as lampshades). This application can use the local binary mode algorithm.

[0070] Color features are the feature components obtained by analyzing the colors presented by image pixels using various methods. For example, a color histogram reflects the composition and distribution of colors in an image, that is, the probability of each color appearing; color moments use the color mean, color variance, and color skew to describe the color distribution, which only requires converting the pixel values ​​in the digital image into numerical values.

[0071] In step S2, the classification of invasive targets first requires acquiring high-quality, fine-grained images of invasive organisms, thus web scraping is employed. Compared to on-site photography, web scraping has the advantage of acquiring a large number of fine-grained images of invasive organisms in a short time, quickly compensating for the insufficient number of samples in the dataset. However, the quality of data samples obtained through web scraping is difficult to guarantee, and duplicate or erroneous samples may occur, requiring manual post-processing and screening. When creating the dataset, attention must also be paid to data balance and diversity. To ensure that the model can accurately identify animals of different species and postures (humans, dogs, cats, mice, snakes, etc.), images of invasive organisms of different species, postures, and backgrounds are selected as much as possible during image collection.

[0072] Deep convolutional neural network (CNN) models that achieve excellent classification accuracy in the field of fine-grained biological image recognition are often strongly supervised learning models. However, the prerequisites for strong supervised learning are expensive manual annotation information and a large amount of training data. Therefore, small and medium-sized datasets cannot guarantee the data requirements of deep CNNs. To fully utilize fine-grained biological images to improve the training accuracy of network models, save annotation costs, and overcome the dependence of deep learning on large-scale data samples, a deep learning method based on small-scale datasets—transfer learning—is introduced to improve the classification accuracy of fine-grained biological images. When the target domain dataset lacks labeled samples, it is difficult for the target model with randomly initialized weight parameters to relearn to convergence. Transfer learning can transfer the knowledge learned in the source model to the target model, using the information provided in the source domain to assist the target model in completing the classification task.

[0073] The process of building a classification model is as follows:

[0074] 1) Use web crawlers to obtain images of invasive organisms of different species, postures, and backgrounds, and then manually label them to form a database;

[0075] 2) Divide the labeled database into a training set and a dataset;

[0076] 3) Train the deep convolutional neural network with transfer learning introduced using the training set;

[0077] 4) Validate the trained deep convolutional neural network using a dataset to obtain a recognition and classification model;

[0078] 5) Use the obtained identification and classification model to identify invasive organisms.

[0079] In step S2, the deep convolutional neural network (CNN) is defined as a multi-level feedforward network structure model, mainly composed of five parts: input layer, convolutional layer, downsampling layer, fully connected layer, and output layer. The CNN model adopts the LeNet-5 network structure, and the process of object recognition and classification is as follows:

[0080] 1) The input layer of a CNN receives the original image and performs data augmentation and size normalization on the original image;

[0081] 2) The convolutional layers of CNN extract local feature information from the input image. The feature information is processed by a non-linear activation function and then passed to the next layer in the form of a feature map.

[0082] 3) The downsampling layers of CNN are periodically interspersed between various convolutional layers to achieve dimensionality reduction and compression of feature maps;

[0083] 4) The fully connected layer of CNN uses a normalization model to reduce the dimensionality of high-dimensional local feature maps and integrate them. The resulting low-dimensional global feature vectors are then classified, and the classification results are output.

[0084] In step S3, target tracking is based on target detection, which means that after classifying the target, the category of target tracking is determined, features are extracted and analyzed in the area where the target is located in the changing image, and similar features and target areas are found in subsequent moving images.

[0085] DeepSort employs a cascaded matching algorithm to determine the priority of matching predicted trajectories with targets. However, occlusion in the image can affect the accuracy of this algorithm. Kalman filtering tracks targets in linear environments; however, it struggles to adapt well to nonlinear conditions when the target's motion direction is irregular. To address the prediction error caused by Kalman filtering, DeepSort introduces an LSTM motion model and uses GIoU matching in the cascaded matching process to optimize IoU matching.

[0086] LSTM Trajectory Prediction Model: The LSTM model is used to solve long-term dependency problems, and its key element is the cell state. In the LSTM algorithm, there are only a few linear interactions between cell states, which allows information to remain stable during processing. The LSTM network structure employs a control gate mechanism, consisting of memory units, input gates, output gates, and forget gates.

[0087] The input gate primarily determines the information stored in the cell state, and the calculation formula is as follows:

[0088]

[0089] Among them, i t C represents the value of the input gate. t X represents the candidate vector that is added to the cell state, α represents the feedforward network layer with the sigmoid activation function, and X represents the candidate vector that is added to the cell state. t Represents the input at time t, ℎ t W represents the cell state value at time t, tanh represents the feedforward network layer with activation function tanh, and W represents the cell state value at time t. i With b i These are parameters that determine how information is stored in the cell. The main function of the forget gate is to discard information from the cell state; the calculation formula is as follows:

[0090]

[0091] Among them, W f With b f I decided to forget the message left behind in the door.

[0092] The state of the memory cell is obtained through two steps: the input gate and the forget gate. The calculation formula is as follows:

[0093]

[0094] Finally, after obtaining the updated values ​​of the memory cells, LSTM calculates the values ​​of the output gates using the following formula:

[0095]

[0096] Increasing the number of layers in an LSTM network can enhance input feature extraction and improve model accuracy; therefore, a two-layer LSTM can be used for trajectory prediction. A unidirectional LSTM typically requires inputs from multiple time steps to determine the predicted output. A bidirectional LSTM model adds a backward-facing layer to a single-layer LSTM model, combining the hidden vectors from both the forward and backward layers to output a combined output for each time step. Therefore, a bidirectional LSTM can fully utilize information from each time step in trajectory prediction to improve prediction accuracy.

[0097] In bidirectional LSTM, the following is continued: Figure 6 The center coordinates and bounding box height of the tracked target are shown. The target's state information from time t-5 to time t is used as the input to a bidirectional LSTM to predict the target's state information at time t+1. The aspect ratio of the predicted target's bounding box is optimized to a bounding box width w for better target localization and training dataset analysis. Specifically, the target's position information (cx, cy, h, w) is used as the input to the bidirectional LSTM. The hyperparameter settings of the bidirectional LSTM are shown in Table 1.

[0098] Table 1 LSTM Hyperparameter Settings

[0099] parameter Value parameter Value Number of hidden layers 2 Batchsize 16 Hidden layer dimensions 128 loss function Mean square error Time step 20 Optimization Algorithm Adam Initial learning rate 0.0001 Dropout 0.2

[0100] GIoU-based matching algorithm: In cascaded matching, the detection box and the predicted box are matched using the Hungarian algorithm and IoU to obtain the matching result. IoU can well reflect the detection performance between the detection box and the predicted box and has scale invariance, but it cannot accurately reflect the degree of overlap between them. The formula for calculating IoU is as follows:

[0101]

[0102] As shown in the above equation, if the detection box does not overlap with the ground truth box, the IoU calculation result will never be 0, causing the loss function to lose its differentiability. Furthermore, IoU is insensitive to the alignment of objects. Figure 5The detection and prediction boxes shown have an IoU value of 0.33 calculated using the formula. However, the overlap between the three is inconsistent, with the left side showing the best regression performance and the right side the worst. Therefore, using IoU matching in target tracking can negatively impact the matching performance of the tracking algorithm. To address the poor IoU loss performance in regression tasks, GIoU association matching is introduced to optimize the IoU matching algorithm. The formula for calculating GIoU is as follows:

[0103]

[0104] Among them, A c This represents the minimum closure area of ​​the detection box and the predicted box. GIoU calculates the area of ​​the region that does not belong to either box and the closure area A. c The ratio between IoU and GIoU is calculated, and finally, this ratio is subtracted from IoU to obtain the value of GIoU.

[0105] Based on the above analysis of the algorithm flow, the LG-DeepSort algorithm flow is as follows: Figure 8 As shown, the target tracking in the LG-DeepSort algorithm is divided into three modules: position prediction, data update, and target matching.

[0106] In position prediction, a bidirectional LSTM algorithm is used to replace Kalman filtering for target position prediction under nonlinear and linear environmental conditions. Mahalanobis distance is calculated using covariance to measure the distance between the actual and predicted positions. Time stamp updates are used to remember the order in which detected targets appeared in the system. The data update module consists of two parts. The first part is tracker updates, where trackers track newly appearing targets and assign them a new ID to update the feature set and record the target. Trackers that fail to track a target are downgraded in priority. The second part is feature set updates, which record the features of newly appearing targets. The recorded target features are stored in the feature set along with their IDs.

[0107] The target matching module is divided into three parts. The first part is cascaded matching, which continuously matches historical trajectories and newly detected targets within the specified loop parameter range until a successful match is achieved. Priority is used in cascaded matching to track targets, giving higher-priority targets higher priority for matching predicted trajectories. The loop parameter in this application is set to 30. The second part is GIoU matching, which is mainly used to calculate the degree of matching between two targets. The third part is the matching result, which has three possible outcomes: trackers that failed to match, targets that failed to match, and targets and trackers that successfully matched. The information of successfully matched targets and trackers is sent to the update module.

[0108] In cascaded matching, cosine similarity is first calculated. Then, a cost matrix is ​​constructed using the average Mahalanobis distance between the target position state and the detection box state predicted by LSTM. Values ​​in the cost matrix where the Mahalanobis distance is greater than a threshold are set to infinity. Finally, values ​​in the cost matrix where the cosine distance is greater than a threshold are also set to infinity. This application sets the Mahalanobis and cosine distance values ​​to infinity to facilitate the subsequent deletion of detection boxes that do not meet the requirements.

[0109] In step S4, the step of obtaining the specific location of the target through the improved sound source localization algorithm is as follows:

[0110] 1) Build an acoustic sensor array receiving model and obtain sound source information through the acoustic sensor array;

[0111] 2) Based on the sound source localization algorithm, the generalized cross-correlation algorithm is integrated to improve the sound source localization algorithm, forming a sound source localization algorithm based on arrival time delay; among them, the complex background noise interference of substations is eliminated by the deep learning-based speech enhancement algorithm, thereby improving the localization and tracking accuracy.

[0112] 3) When the target sound source is not blocked in the scene, the sound wave will directly reach the acoustic sensor array. Finally, the time delay information of each array element is used to solve the acoustic sensor array receiving model to obtain the sound source location.

[0113] 4) When the target sound source is in a non-line-of-sight scene, the transmitted frequency-modulated continuous wave is selected as the sound source signal. Through corresponding signal processing, the time delay information of the reflection path is calculated, and the non-line-of-sight target is located based on the time delay information.

[0114] In scenarios where the target sound source is not obstructed, the sound source localization algorithm based on arrival delay (ARD) follows these steps: Each acoustic sensor acquires signals with a certain time difference. Combining this with the array topology of the acoustic sensors, the angle between the sound source signal and the array is calculated through geometric analysis. This involves two steps: direction of arrival (DOA) estimation and sound source localization. DOA estimation calculates the angle between the sensor and the target sound source based on the time difference in signal acquisition. Because the acoustic sensors are located in different spatial positions, the sound source propagation paths are also different. Combined with reverberation and reflection within the space, there is a time difference in the sound source's propagation to different sensors. Therefore, time delay estimation is required to estimate the relative time delay difference between the signals received by each sensor, and then the spatial location of the sound source in the scene is calculated.

[0115] The Generalized Cross-Correlation (GCC) algorithm is one of the most commonly used time difference estimation methods. The sound source localization process of the GCC algorithm is as follows: the sound source travels along different paths to each array element, so the signals collected by the array will differ in time. The power spectrum function is weighted in the frequency domain. The expression in the time domain can be obtained through the fast inverse Fourier transform, thus obtaining the cross power spectrum function between the signals received by each pair of array elements. The extreme point is the time difference between the received signals of each pair of array elements.

[0116] Suppose there are two microphones, and the signals they receive are denoted by x1(t) and x2(t) respectively. In the presence of noise, this can be represented by the following mathematical model:

[0117]

[0118] In the formula, n1(t) and n2(t) are the noise signals received by sensor 1 and sensor 2, s(t) is the signal emitted by the target sound source, a1 and a2 are the attenuation coefficients of the sound signal from the sound source to sensor 1 and sensor 2, respectively, and ∆t is the time difference between the sound pressure signals received by sensor 1 and sensor 2. The correlation function of the two signals can be expressed as:

[0119]

[0120] In the formula, This is the cross-correlation power spectrum of the signals acquired by sensor 1 and sensor 2. X1(w) and X2(w) are the frequency domain signals obtained by Fourier transform of x1(t) and x2(t). It is the conjugate of X2(w); For the generalized cross-correlation power spectrum, The weight function is used. Finally, [the following is...] The time-domain signal is obtained after inverse Fourier transform. The time corresponding to the peak of the time-domain signal is the time delay between the signals of sensor 1 and sensor 2.

[0121] In scenarios where the target sound source is obstructed, the steps of a sound source localization algorithm based on arrival delay are as follows:

[0122] 1) Sound source emission model:

[0123] In complex scenarios where the target is obscured, assuming the sound emitted by the target is a pulse signal, due to the presence of obstructions and reflective surfaces, the microphone array ultimately receives the superposition of all signals in the scene (including reflected signals, transmitted signals, etc.), making it difficult to distinguish between signals reflected back from various directions and the time information of their arrival at the microphone array.

[0124] The sound source signal generator sends a modulated sound signal and preprocesses the received signal to simulate the scene's response to short pulses. Frequency-modulated continuous wave (FMCW) is a frequency-modulated signal whose frequency change follows a sine wave waveform. The sensor array receives the signal, making signal processing easy, with low hardware complexity and high distance resolution. The signal g(t) emitted by the sound source increases linearly from the initial frequency f0 to f1 within time T, as shown in the following equation:

[0125] .

[0126] 2) Delay estimation;

[0127] Measurement values ​​of each element in the microphone array This includes the delayed signal g(t) reflected from reflective surfaces in the scene. To simulate the impulse response in the scene, the received signal is multiplied by the original transmitted signal g(t) in the time domain, filtered, and then subjected to a Fourier transform. For a fixed sound source emission position and a microphone array receiving position, each element of the microphone array will produce a peak at the frequency bf given by the following equation:

[0128]

[0129] In the formula: B = f1 - f0 is the bandwidth of the modulation signal g(t), T is the duration of frequency modulation, r is the sound path between the sound source and the array element, and c is the speed of sound.

[0130] like Figure 9 As shown in (a), it is assumed that the sound source transmits a modulation signal at t=0s, the starting frequency of the modulation signal is F0=0Hz, the ending frequency is f1=20kHz, T=0.0625s, and the array element starts to receive the signal transmitted by the sound source at t=0.0125s. Figure 9 (b) The peak image obtained after performing a Fourier transform on the acquired signal (starting at t=0s) and the original transmitted signal g(t) in the time domain by multiplying the acquired signal by the original transmitted signal g(t). b =kHz, from the above formula we get:

[0131]

[0132] The array element receives the sound wave signal sent by the sound source at t=0s at t=0.0125s, which is consistent with the timing result of the received signal in a.

[0133] After completing the time delay estimation step, the available conditions for solving the sound source location are: sound velocity c, array coordinates. And the time t for the sound source to reach the array element n Taking a three-element acoustic sensor array as an example, the formula for calculating the sound source location is as follows:

[0134]

[0135] in, This represents the sound path from the sound source to each array element.

[0136] The sound enhancement process of the deep learning-based speech enhancement algorithm is as follows:

[0137] 1) Model Structure: The short-time Fourier spectrum estimation (AMS) speech enhancement framework does not perform well at low signal-to-noise levels, such as 0dB. This paper proposes a deep learning-based temporal convolutional network speech enhancement algorithm by combining the MMSE-LSA speech enhancement algorithm within the AMS framework. Figure 10 As shown.

[0138] The temporal convolutional network speech enhancement algorithm based on deep learning uses the minimum logarithmic amplitude spectrum mean square error gain function used in the MMSE-LSA speech enhancement algorithm. Therefore, the amplitude spectrum representation of the enhanced speech signal can be shown below.

[0139]

[0140] In the formula, This represents the prior signal-to-noise ratio, while Let represent the posterior signal-to-noise ratio. This refers to the gain function based on the minimum logarithmic magnitude spectrum mean square error used in the MMSE-LSA speech enhancement algorithm. Furthermore, prior knowledge indicates that when the speech signal is stationary and the noise signal is uncorrelated with the speech signal, the posterior signal-to-noise ratio... This can be simplified to the prior signal-to-noise ratio minus 1. -1.

[0141] After modifying the amplitude spectrum information of the existing noisy speech signal to the estimated amplitude spectrum information of the clean speech signal, the system synthesizes the phase information of the noisy speech signal, thereby generating an enhanced speech signal in the frequency domain. Its polar coordinate representation is as follows:

[0142]

[0143] The enhanced speech signal in the frequency domain is then converted to the time domain by inverse short-time fourier transform (ISTFT). Finally, the window functions of the segmented speech signals are overlapped and added to output the enhanced speech signal.

[0144] 2) Prior SNR Estimation: Compared to maximum likelihood estimation and direct decision estimation methods that rely on operator experience to determine the prior SNR, deep learning, after learning from a large amount of noisy speech data and its corresponding clean speech data, undoubtedly provides a more accurate estimation of the prior SNR of the enhanced speech data. Currently, TCN temporal convolutional neural network models, such as... Figure 11 The diagram shows a TCN temporal convolutional neural network structure containing 6 residual blocks.

[0145] A simple TCN (Temporal Convolutional Neural Network) mainly consists of three convolutional units and residual connections. Each convolutional layer includes activation units and normalization operations. This study investigates the ResLSTM network, which boasts superior prediction performance among temporal convolutional neural networks, aiming to accelerate network convergence, improve prediction accuracy, and effectively mitigate the degradation problem of deep neural networks, thereby enhancing network performance. The ResLSTM prior signal-to-noise ratio prediction network model structure diagram is shown below. Figure 12 As shown.

[0146] The input signal is the amplitude spectrum function |(n,k)| after transforming the noisy speech signal to the frequency domain (for k=0,1,…,Nl / 2), where Nl is the frame length in the discrete-time sample. The FC module in the diagram is a fully connected layer that outputs a 512*8 dimensional feature vector, with a normalization operation before the activation function. The F module in the diagram is a residual block; each F module contains a 512-unit Long Short-Term Memory (LSTM) neural network, which facilitates processing sequential speech vectors. The final O module is a fully connected layer that connects to all nodes of the preceding neural network, combining the extracted feature vector parameters to output a 1*1 one-dimensional vector. .

[0147] The instantaneous prior signal-to-noise ratio (SNR) of a speech signal can be obtained by calculating the instantaneous power spectra of the clean speech signal and the noisy signal, respectively. However, since the instantaneous prior SNR of a speech signal has a wide numerical range, directly using it as the training target of a neural network will lead to difficulties in network convergence. This application uses the cumulative distribution function of the instantaneous prior SNR of noisy speech signals as the target mapping, mapping the estimated instantaneous prior SNR (dB) of noisy speech signals to [0,1] to improve the convergence speed when using the stochastic gradient descent algorithm. Furthermore, it is assumed that the instantaneous prior SNR (dB) of the noisy speech signal follows a mean of 1 / 2. and variance Given the distribution of the signal-to-noise ratio target mapping, the cumulative distribution function of the prior signal-to-noise ratio target mapping can be expressed by the following formula:

[0148]

[0149] To obtain the target mapping for the instantaneous prior signal-to-noise ratio (SNR) of the speech signal, it is necessary to calculate the mean and variance of the instantaneous prior SNR. These statistical values ​​are first obtained by randomly selecting 1000 clean speech signals from the training set and dividing them into five equal parts. Subsequently, these speech signals are mixed with noise signals at five different SNRs within the range of -5dB to 5dB. Finally, the mean and variance of the instantaneous prior SNR of the mixed noisy speech signal are calculated.

[0150] The output value of the ResLSTM prior signal-to-noise ratio prediction network model in this application is obtained. That is, after substituting the target mapping estimate of the instantaneous prior signal-to-noise ratio of the speech signal into the above equation and taking the inverse, we obtain the following equation:

[0151]

[0152] In the above formula This refers to the instantaneous prior signal-to-noise ratio (SNR) estimate of the speech signal. Since the unit of the instantaneous prior SNR of the speech signal is dB, while the prior SNR required in the MMSE-LSA speech enhancement algorithm is in numerical units, a conversion is necessary.

[0153]

[0154] The above formula yields This is the desired prior signal-to-noise ratio estimate of the speech signal, thereby completing the prediction and modification of the amplitude spectrum information of the noisy speech signal, achieving the effect of speech enhancement.

[0155] Based on the same inventive concept, this invention also provides an intrusion tracking system for substations in multi-view scenarios based on vision and sound, characterized by comprising:

[0156] Feature extraction module: preprocesses the substation monitoring images and extracts features from the processed images to obtain target feature information;

[0157] Target recognition module: Based on the obtained target feature information, it uses a recognition and classification model to identify and classify the target, thereby obtaining intrusion target feature information;

[0158] Target tracking module: includes visual trajectory module, acoustic trajectory module and fusion module, wherein:

[0159] Visual trajectory module: Based on the acquired feature information of the intrusion target, the improved DeepSort target tracking algorithm is used to find similar features and target regions in subsequent moving images, perform visual tracking of the intrusion target, and obtain the visual motion trajectory of the intrusion target;

[0160] Acoustic trajectory module: Based on the acquired feature information of the intrusion target, it obtains the sound source information of the target area and uses an improved sound source localization algorithm to obtain the specific location of the target, performs acoustic tracking of the intrusion target, and obtains the acoustic motion trajectory of the target;

[0161] Fusion module: The weighted average fusion method is used to fuse visual motion trajectory and auditory motion trajectory to obtain the motion trajectory of the intruding target.

[0162] In this application, the other technical features of the intrusion tracking system in the multi-view scenario of substation based on vision and sound are the same as those disclosed in the above method embodiments, and will not be repeated here.

[0163] Based on the same inventive concept, the present invention also provides an electronic device, which may include a processor, a memory, a display screen, an input device, and a peripheral terminal port.

[0164] Furthermore, the processor can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0165] Furthermore, memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. The memory in the embodiments of this application can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0166] Furthermore, the peripheral terminal interface can be used to connect at least one I / O (Input / Output) related peripheral terminal to the processor and memory. In some embodiments, the processor, memory, and peripheral terminal interface are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor, memory, and peripheral terminal interface can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0167] Furthermore, the display screen is used to display a UI (User Interface). This UI can include graphics, text, icons, videos, and any combination thereof. When the display screen is a touch screen, it also has the ability to collect touch signals on or above the surface of the display. These touch signals can be input as control signals to the processor for processing. In this case, the display screen can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be one display screen, located on the front panel of the electronic device; in other embodiments, there can be at least two display screens, respectively located on different surfaces of the electronic device or in a folded design; in still other embodiments, the display screen can be a flexible display screen, located on a curved or folded surface of the electronic device. Furthermore, the display screen can even be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0168] Furthermore, input devices are used to input data and information into a computer, serving as a bridge for communication between the computer and the user or other devices. Input devices are one of the main devices for information exchange between the user and the computer system. These include keyboards, mice, cameras, scanners, light pens, handwriting input tablets, joysticks, and voice input devices. An input device is a device used by a person or external device to interact with a computer, inputting raw data and programs that process that data into the computer. Computers can receive various types of data, including both numerical and non-numerical data such as graphics, images, and sounds, all of which can be input into the computer through different types of input devices for storage, processing, and output.

[0169] Based on the same inventive concept, this application also provides a computer-readable storage medium storing a computer program that can be executed by a processor to implement the above-described method.

[0170] The technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable computer storage medium and includes several instructions / computer programs to cause an Internet of Things device (which may be a personal computer, server, or network terminal, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as electronic terminals such as computers, mobile phones, laptops, tablets, and cameras that have the aforementioned storage media.

[0171] The description of the execution process of program data in a computer-readable storage medium can be found in the descriptions in the various method embodiments of this application above, and will not be repeated here.

[0172] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A visual and acoustic-based intrusion tracking method in a substation multi-view scene, comprising the following steps: S1. Preprocessing the substation monitoring image and extracting features from the processed image to obtain target feature information; S2. Identifying and classifying the target based on the obtained target feature information using a recognition classification model to obtain intrusion target feature information; S3. Using an improved DeepSort target tracking algorithm to find similar features and target regions in subsequent moving images based on the obtained intrusion target feature information, and performing visual tracking of the intrusion target to obtain a visual motion trajectory of the intrusion target; S4. Obtaining sound source information of the identified target region based on the obtained intrusion target feature information, and obtaining the specific location of the identified target through an improved sound source positioning algorithm to perform acoustic tracking of the intrusion target and obtain an acoustic motion trajectory of the target; S5. Using a weighted average fusion method to fuse the visual motion trajectory and the acoustic motion trajectory to obtain the motion trajectory of the intrusion target.

2. The method for tracking intrusion in substation multi-visual field scene based on vision and acoustic sense according to claim 1, characterized in that In step S1, the process of obtaining target feature information is as follows: obtaining frame image information from video information obtained by a camera; using nearest neighbor interpolation to scale the processed image, using mean filter method to denoise the scaled image, and using histogram equalization method to enhance the image; using support vector machine or binary tree to extract image features.

3. The method of claim 1, wherein the method further comprises: in response to the detection of the intruder, determining whether the intruder is in the first visual field of view or the second visual field of view; and in response to determining that the intruder is in the first visual field of view, activating the first camera and the first microphone. In step S2, the process of identifying and classifying the target is as follows: 1) Using a crawler to obtain images of intrusion organisms in different categories, different postures, and different backgrounds, and manually marking them to form a database; 2) Dividing the marked database into a training set and a data set; 3) Training a deep convolutional neural network with transfer learning through the training set; 4) Verifying the trained deep convolutional neural network through the data set to obtain a recognition classification model; 5) Using the obtained recognition classification model to identify the intrusion organism.

4. The method of claim 1, wherein the method further comprises: in response to the detection of the intruder, determining whether the intruder is in the first visual field of view or the second visual field of view; and in response to determining that the intruder is in the first visual field of view, activating the first camera and the first microphone. In step S3, the process of improving the DeepSort target tracking algorithm is as follows: introducing an LSTM motion model in DeepSort, and using GIoU matching to optimize IoU matching in cascade matching.

5. The method for tracking intrusion in substation multi-visual field scene based on vision and acoustic sense according to claim 4, characterized in that In cascade matching, first, the cosine similarity is calculated, then the average Mahalanobis distance between the target position state and the detection box state is predicted through LSTM to construct a cost matrix, and the corresponding values in the cost matrix whose Mahalanobis distance is greater than the threshold are set to infinity, finally, the corresponding values in the cost matrix whose cosine distance is greater than the threshold are also set to infinity.

6. The method of claim 1, wherein the method is a visual and acoustic based substation multi-view scene intrusion tracking method. The steps of obtaining the specific location of the identified target through the improved sound source positioning algorithm are as follows: 1) Building an acoustic sensor array reception model and obtaining sound source information through the acoustic sensor array; 2) Improving the sound source positioning algorithm by integrating a generalized cross-correlation algorithm based on the sound source positioning algorithm to form a sound source positioning algorithm based on time delay; wherein the background noise interference in the substation is eliminated through a deep learning-based speech enhancement algorithm to improve the positioning and tracking accuracy; 3) When the target sound source is not blocked in the scene, the sound wave will directly reach the acoustic sensor array, and finally the sound source position is obtained by using the time delay information of each array element in the array to solve the acoustic sensor array receiving model; 4) When the target sound source is in the non-visual field scene, the transmitting frequency-modulated continuous wave is selected as the sound source signal, and the time delay information of the reflection path is calculated through corresponding signal processing, so as to position the non-visual field target according to the time delay information.

7. The method of claim 6, wherein the method is a visual and acoustic based substation intrusion tracking method in a multi-view scene, characterized in that: The process of sound enhancement is: in the short-time Fourier spectrum estimation speech enhancement framework, combining the MMSE-LSA speech enhancement algorithm, using the noise spectrum perception method, combining the prior SNR estimation model based on deep learning on the prior SNR estimation of pure speech signal, so that the enhanced speech signal has less noise residue.

8. A visual and acoustic based substation multi- view scene intrusion tracking system, characterized in that: It comprises: a feature extraction module: pre-processing the substation monitoring image, and extracting features from the processed image to obtain target feature information; a target recognition module: according to the obtained target feature information, using a recognition classification model to recognize and classify the target to obtain the intrusion target feature information; a target tracking module: including a visual trajectory module, an acoustic trajectory module and a fusion module, wherein: the visual trajectory module: according to the obtained intrusion target feature information, using an improved DeepSort target tracking algorithm to find similar features and target areas in subsequent moving images, and performing visual tracking of the intrusion target to obtain the visual motion trajectory of the intrusion target; the acoustic trajectory module: according to the obtained intrusion target feature information, obtaining the sound source information of the recognized target area, and obtaining the specific position of the recognized target through an improved sound source positioning algorithm to perform acoustic tracking of the intrusion target and obtain the acoustic motion trajectory of the target; the fusion module: using a weighted average fusion method to fuse the visual motion trajectory and the acoustic motion trajectory to obtain the motion trajectory of the intrusion target.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 7.

Images