Vision and acoustics fused transformer substation intrusion target cross-modal tracking system

Abstract

The invention relates to the field of image processing, in particular to a transformer substation intrusion target cross-modal tracking system fusing vision and acoustics, which comprises a vision monitoring module, an acoustics monitoring module and a filtering tracking module. And the vision module obtains a target azimuth angle by using a camera image and Kalman filtering. The acoustic module collects sound through a microphone array, obtains an observation azimuth angle and a denoising azimuth angle by adopting a random denoising and time delay estimation positioning method, and records the difference between the observation azimuth angle and the visual filtering angle and between the denoising azimuth angle and the visual filtering angle. The filtering tracking module adaptively judges the fusion opportunity by analyzing the difference between the distribution characteristic of the recorded difference and the visual tracking error characteristic; and then taking average noise extracted by historical effective denoising operation as reference noise and removing the reference noise, and finally fusing an acoustic azimuth angle and a visual azimuth angle to realize accurate tracking. According to the invention, the problem of unreliable tracking of a single sensor in a complex environment is solved, and robust, continuous and automatic tracking of an intrusion target is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a substation intrusion target cross-modal tracking system fusing vision and acoustics. BACKGROUND

[0002] As the core hub of the power system, the safe and stable operation of the substation is directly related to the reliable power supply of the regional power grid. However, the substation usually occupies a wide area and is mostly in an open environment, which is easy to be invaded by illegal personnel, animals, or unmanned aerial vehicles, and other invasion target security threats.

[0003] At present, one of the mainstream intrusion target tracking methods is to rely on visual monitoring technology. By deploying high-definition cameras, using computer vision algorithms (such as YOLO target detection model) to identify intrusion targets, and combining Kalman filtering and other prediction algorithms to achieve tracking. However, the pure visual tracking method has inherent limitations: in the case of severe changes in light, bad weather such as rain, snow, fog, and target being partially blocked or being too far away resulting in too low pixels, the recognition accuracy of the visual model will decrease significantly, and even the target will be lost.

[0004] Acoustic monitoring technology is also one of the intrusion target tracking methods. By arranging a microphone array, using sound source positioning (such as time delay estimation TDOA) methods, the acoustic azimuth information generated by the intrusion target can be obtained. This acoustic tracking is not affected by visual conditions, and is particularly advantageous in scenarios where the target is not visible but audible (such as behind the obstruction). However, the substation site environment is complex, filled with continuous and strong background electromagnetic noise such as transformer humming and fan operation, while the sound signals of the intrusion targets (such as unmanned aerial vehicles and human footsteps) are weak and variable, resulting in a very low signal-to-noise ratio of the acoustic signal, and a large error in the observed azimuth angle of the direct positioning, which is insufficient in reliability.

[0005] In summary, the prior art cannot achieve stable and accurate long-time tracking. SUMMARY

[0006] To solve the above problems, the present application provides a substation intrusion target cross-modal tracking system fusing vision and acoustics.

[0007] The substation intrusion target cross-modal tracking system fusing vision and acoustics of the present application adopts the following technical solutions: One embodiment of the present application provides a substation intrusion target cross-modal tracking system fusing vision and acoustics, which comprises the following modules: A visual monitoring module for monitoring the azimuth angle of the intrusion target at each time using images captured by a camera, filtering the azimuth angle using a Kalman filtering algorithm, and obtaining a filtered angle at each time; The acoustic monitoring module is used to collect the observed sound at each moment based on the microphone array, and randomly remove noise from the observed sound to obtain the filtered sound. The time delay estimation positioning method is used to obtain the observed azimuth angle and the noise-reduced azimuth angle at each moment based on the observed sound and the filtered sound respectively. When the first difference between the observed azimuth angle and the filtered angle is greater than the second difference between the noise-reduced azimuth angle and the filtered angle, the second difference is recorded. The filtering and tracking module is used to, at the current moment, for all recorded second difference distribution features A1 and the error distribution features of the filter angle obtained when recording the second differences, when the difference between A1 and the error distribution features is less than a first preset threshold, use the average noise of the noise removed when recording all second differences as reference noise; the denoised azimuth angle obtained after removing the reference noise from the observed sound collected after the current moment, and the filter angle obtained after the current moment, are used to realize the tracking of the intrusion target after the current moment.

[0008] Preferably, the specific steps for obtaining the difference between A1 and the error distribution characteristics are as follows: For all the filter angles obtained when recording the second difference, the mean of the error distribution characteristics of all filter angles is denoted as a2; the distribution difference between the error distribution characteristics of all filter angles is denoted as tracking error stability; the difference between A1 and the error distribution characteristics is positively correlated with A1-a2 and tracking error stability, respectively.

[0009] Preferably, the specific steps for randomly removing noise from the observed sound to obtain the filtered sound are as follows: For the observed sound collected by each microphone in the microphone array, the spectrum of the observed sound is obtained using the Fast Fourier Transform algorithm. The spectrum contains the response values ​​of the observed sound at different frequencies. Several frequencies are randomly selected in the spectrum. In the spectrum of the observed sound collected by all microphones, the response values ​​corresponding to all selected frequencies are first set to 0. Then, the spectrum is transformed into filtered sound using the Inverse Fourier Transform algorithm. In this process, for the spectrum of the observed sound collected by each microphone, the response value of the selected frequency remains unchanged, while the response value of the frequencies other than the selected frequency is set to 0, thus obtaining the noise spectrum of each microphone, which is recorded as the noise randomly removed from the observed sound.

[0010] Preferably, the step of using the average noise removed when recording all second differences as reference noise includes the following specific steps: For the noise that was randomly removed when recording all the second differences, the average noise spectrum corresponding to all the noise was calculated, and the resulting average noise spectrum was denoted as the reference noise.

[0011] Preferably, the denoised azimuth angle obtained by removing reference noise from the observed sound collected after the current moment includes the following specific steps: For any time after the current time, and for the spectrum of the observed sound collected by each microphone in the microphone array at that time, the response value of each frequency in the spectrum is subtracted from the response value of each frequency in the reference noise to obtain the difference spectrum. The difference spectrum is transformed using the inverse Fourier transform algorithm to obtain the filtered sound. The denoised azimuth angle is obtained based on all the filtered sounds obtained by the microphone array using the time delay estimation positioning method.

[0012] Preferably, the denoised azimuth angle obtained by removing reference noise from the observed sound collected after the current moment, and the filtered angle obtained after the current moment, are used to track the intrusion target after the current moment, including the following specific steps: The denoised azimuth angle and the filtered angle obtained at each time after the current time are averaged to obtain the estimated azimuth angle of the intrusion target at each time. The estimated azimuth angle of the intrusion target at each time is used as the observation value of the Kalman filter algorithm. The azimuth angle of the next time at each time is predicted by the Kalman filter algorithm and recorded as the prediction angle. The camera's viewing angle is controlled to face the prediction angle to realize the tracking of the intrusion target.

[0013] Preferably, the specific steps for randomly selecting several frequencies in the spectrum are as follows: For any observed noise spectrum, all frequencies with a response value greater than 0 in the spectrum are denoted as candidate frequencies. The Otsu threshold segmentation algorithm is used to divide the response values ​​of all candidate frequencies into two parts. The frequency of the part with the largest average response value is denoted as the first frequency, and the frequency of the part with the smallest average response value is denoted as the second frequency. Frequencies are selected from the first frequency and the second frequency respectively, wherein the percentage of the number of frequencies selected from the first frequency is greater than the percentage of the number of frequencies selected from the second frequency.

[0014] Preferably, the distribution feature A1 is the variance of all recorded second differences.

[0015] Preferably, the error distribution characteristics of each filtering angle are determined by the state covariance matrix of the Kalman filtering algorithm.

[0016] Preferably, the distribution difference among the error distribution characteristics of all filtering angles is denoted as tracking error stability, and the specific steps include the following: First, divide the error distribution characteristics of all filter angles by a2 respectively, and then denote the variance of the quotients obtained for all filter angles as the tracking error stability.

[0017] The beneficial effects of the technical solution of the present invention are: This invention further integrates visual and acoustic information in a reasonable manner. In the initial tracking phase, the tracking system prioritizes visual information while actively exploring and evaluating environmental noise using an acoustic module. Only when the error distribution characteristics (A1) of the acoustic module's positioning results are sufficiently close to the error distribution characteristics of visual tracking (i.e., the difference is less than a first preset threshold) after multiple rounds of random denoising, does the system formally incorporate acoustic information into the fusion tracking. This intelligent switching mechanism from "single-modal" to "dual-modal fusion" effectively avoids interference from large errors in the relatively reliable visual tracking during the initial stage when the acoustic information signal-to-noise ratio is too low and reliability is insufficient. This significantly improves the overall robustness and tracking stability of the system in complex substation environments.

[0018] This invention filters effective denoising operations by randomly removing noise from observed sounds and comparing a first difference with a second difference, recording the corresponding second difference. This process is essentially an online, data-driven noise assessment method. The tracking system adaptively selects denoising methods that effectively improve sound source localization accuracy through numerous random trials, and finally averages the noise removed by these effective denoising operations to obtain a highly representative reference noise. This method can dynamically adapt to changing environmental noise in substations, laying a solid foundation for accurate acoustic localization in subsequent stages and overcoming the difficulty of fixed filtering algorithms in handling complex and variable field noise.

[0019] This invention ensures the accuracy and continuity of long-term tracking. Upon entering the fusion tracking phase, the tracking system uses extracted reference noise to purify subsequent sound observations, obtaining a more accurate denoised azimuth angle. This denoised azimuth angle is then fused with the visual filter angle, and together they serve as the Kalman filter observation value to predict the target's next azimuth. This fusion strategy leverages the complementary advantages of smooth visual tracking trajectories and unaffected acoustic positioning. Even when visual tracking errors occur due to brief occlusion or image blurring, accurate acoustic positioning information can correct and supplement them, and vice versa. This complementary mechanism significantly reduces the probability of target loss, ensuring long-term, high-precision continuous tracking of intruding targets, and meeting the stringent reliability requirements of substation safety monitoring.

[0020] In summary, this invention effectively addresses the pain points of poor tracking reliability and susceptibility to interference in complex substation scenarios by using a phased fusion framework and adaptive noise processing technology, thereby achieving a synergistic improvement in tracking accuracy, robustness, and continuity. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a framework diagram of a cross-modal tracking system for substation intrusion targets that integrates vision and acoustics, provided in an embodiment of the present invention. Figure 2 Here is a flowchart of the steps for the visual monitoring module; Figure 3 Here is a flowchart of the steps for the acoustic monitoring module; Figure 4 This is a flowchart of the steps in the filtering and tracking module. Detailed Implementation

[0023] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the cross-modal tracking system for substation intrusion targets integrating vision and acoustics proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] The following description, in conjunction with the accompanying drawings, details the specific scheme of the cross-modal tracking system for substation intrusion targets that integrates vision and acoustics provided by the present invention.

[0026] Example 1: like Figure 1 As shown, this embodiment provides a cross-modal tracking system for substation intrusion targets that integrates vision and acoustics. The system includes a visual monitoring module, an acoustic monitoring module, and a filtered tracking module.

[0027] The visual monitoring module uses computer vision technology to identify and track intrusion targets. For example... Figure 2 As shown, this module contains the following steps: Step S101: Use the images captured by the camera to monitor the azimuth angle of the intrusion target at each moment.

[0028] Step S102: Use the Kalman filter algorithm to filter the azimuth angle to obtain the filtered angle at each time moment, and use the Kalman filter algorithm to track the intrusion target.

[0029] The acoustic monitoring module uses sound source localization technology to identify the location of the sound source (i.e., the intrusion target). For example... Figure 3 As shown, this module contains the following steps: Step S201: Collect the observed sound at each moment based on the microphone array, and randomly remove the noise from the observed sound to obtain the filtered sound.

[0030] Step S202: Using a time-delay estimation positioning method, obtain the observed azimuth and denoised azimuth at each moment based on the observed sound and the filtered sound, respectively. When the first difference between the observed azimuth and the filtered angle is greater than the second difference between the denoised azimuth and the filtered angle, record the second difference.

[0031] The filtering and tracking module is used to remove noise from the observed sound of the sound source (i.e., the intruding target), and then the visual and acoustic modalities are fused together for target tracking. For example... Figure 4 As shown, this module contains the following steps: Step S301: Obtain the distribution feature A1 of all recorded second differences, obtain the error distribution feature of the filter angle obtained when recording the second differences, and determine whether the difference between A1 and the error distribution feature is less than the first preset threshold.

[0032] Step S302: When the difference between A1 and the error distribution characteristics at the current time is less than the first preset threshold, the average noise of the noise removed when recording all the second differences is used as the reference noise. The denoised azimuth angle is obtained again after removing the reference noise from the observed sound collected after the current time.

[0033] Step S303: The denoised azimuth angle is fused with the filtered angle obtained after the current moment to realize the tracking of the intrusion target after the current moment.

[0034] As can be seen from the above, in the initial stage of tracking the intrusion target (that is, the period before it is "less than the first preset threshold"), the visual monitoring module is mainly used to track the intrusion target; the acoustic monitoring module obtains a large amount of noise interference from the observed sound, and the obtained azimuth angle error is large, so it is not used as the basis for tracking the intrusion target to avoid the problem of large tracking error caused by the noise of the observed sound; in the initial stage, the acoustic monitoring module is used to explore the noise distribution of the observed sound and evaluate the acoustic positioning error (that is, the second difference) after removing different noises, so as to prepare data for the subsequent removal of noise from the observed sound.

[0035] In the later stage (i.e., the period after the threshold is "less than the first preset threshold"), the acoustic monitoring module explores or estimates the distribution characteristics of noise in the observed sound (i.e., reference noise). At this time, the visual and acoustic modes are fused together to perform target tracking, ensuring the accuracy and stability of target tracking.

[0036] In summary, this embodiment avoids both the inaccuracy issues associated with long-term visual tracking alone and the excessive noise interference that arises when directly fusing acoustic and visual tracking. This embodiment achieves accurate and continuous tracking of intrusion targets over extended periods by indirectly and appropriately fusing visual and acoustic modalities.

[0037] The following is a detailed description of each step contained in each of the above modules: Step S101: Use the images captured by the camera to monitor the azimuth angle of the intrusion target at each moment.

[0038] Cameras are installed in an open-air substation. At each moment, the collected images are input into the target monitoring model to obtain the rectangular bounding box of the intrusion target.

[0039] The targets described in this embodiment include people, drones, birds, animals, etc.

[0040] In this embodiment, the camera captures RGB color images with a width and height of 1024×512 pixels and a sampling frequency of 20Hz. Furthermore, the camera's viewing angle is controllable. In this embodiment, the camera is mounted on a fixed pan-tilt unit on a tower. The camera's viewing angle is freely controlled by adjusting the pan-tilt unit's horizontal rotation angle (denoted as the rotation angle) and vertical rotation angle (denoted as the pitch angle). In this embodiment, the rotation angle is 0~360°, and the pitch angle is -80~90°. When the camera's viewing angle is parallel to the horizontal plane, the pitch angle is 0°; when looking downwards, the pitch angle is less than 0°.

[0041] In this embodiment, each second is considered a time interval, and the target monitoring model used is the YOLOv5 model.

[0042] Furthermore, to ensure that the intrusion target is always in the center of the image, after obtaining the rectangular bounding box of the intrusion target, the pixel point where the center of the rectangular bounding box is located is recorded as the target position. The camera's viewing angle (including rotation angle and pitch angle) is controlled so that the distance between the target position and the pixel point at the center of the image is 0 (that is, the target position coincides with the image center). The method of controlling the camera's viewing angle is to use a PID algorithm. In this embodiment, the proportional gain, integral gain, and derivative gain of the PID algorithm are 1.4, 0.7, and 0.11, respectively, and the sampling frequency of the PID algorithm is 20Hz.

[0043] When the target location coincides with the image center, the camera's viewpoint (a two-dimensional vector consisting of rotation and pitch angles) is used as the azimuth angle of the intrusion target at each moment.

[0044] Thus, the azimuth angle of the intruding target at each moment was obtained using the target monitoring model.

[0045] Step S102: Use the Kalman filter algorithm to filter the azimuth angle to obtain the filtered angle at each time moment, and use the Kalman filter algorithm to track the intrusion target.

[0046] It should be noted that, due to errors in the process of acquiring the azimuth angle by the target detection model, this embodiment does not directly use the azimuth angle for tracking. These errors arise from the target detection model's identification errors (e.g., larger errors occur when the intruding target is small, obstructed, or subject to significant ambient lighting interference) and camera view control errors. Therefore, this embodiment utilizes the Kalman filter algorithm to track the intruding target.

[0047] Specifically: Starting from the first time the camera detects an intruding target (marked as the first moment), for every moment after the first moment (excluding the first moment), and for all azimuth angles obtained at each moment and before each moment, these azimuth angles constitute a time series. This time series is filtered using the Kalman filter algorithm, and the filtered azimuth angle at each moment is denoted as the filtered angle. Furthermore, the Kalman filter algorithm not only performs filtering but also predicts the azimuth angle for the next moment at each moment, denoted as the predicted angle.

[0048] The camera's viewing angle is set to the predicted angle for the next time step at each moment. Then, an image is acquired at that next time step, and the azimuth angle for that next time step is obtained. For the time series consisting of all azimuth angles obtained at the next time step and before, the Kalman filter algorithm is used again to filter the time series and predict the azimuth angle. This process is repeated to achieve real-time tracking of the intrusion target.

[0049] The filtered angle obtained at each of the above time points is closer to the true azimuth angle of the intrusion target than the azimuth angle at each of the above time points.

[0050] Step S201: Collect the observed sound at each moment based on the microphone array, and randomly remove the noise from the observed sound to obtain the filtered sound.

[0051] A microphone array is installed in the substation. The microphone array consists of several microphones. Taking any one microphone as an example, the sound signal collected by the microphone within a first preset time period before each time moment is recorded as the observed sound at that time moment. In this embodiment, the first preset time period refers to the 3 seconds before each time moment (inclusive).

[0052] The observed sound includes not only the sound of the intruding target but also noise. This noise includes background noise (such as the fixed-frequency sound generated by substation electrical equipment during operation) and other undetermined noise.

[0053] In this embodiment, background noise is removed from the observed sound. The observed sound mentioned below refers to the observed sound after background noise has been removed. Note that the observed sound after background noise removal still contains back noise.

[0054] Furthermore, the spectrum of the observed sound is extracted using the Fast Fourier Transform algorithm. This spectrum contains the response values ​​of the observed sound at different frequencies. The frequencies of the spectrum include both the sound frequencies generated by the sound source (i.e., the intrusion target) and the noise frequencies.

[0055] Several frequencies are randomly selected in the spectrum. For the spectrum of the observed sound collected by all microphones, the response values ​​corresponding to these frequencies are set to 0. Then, the inverse Fourier transform algorithm is used to transform each spectrum into filtered sound.

[0056] The process of randomly selecting several (e.g., 5) frequencies and setting the response value to 0 is called the random noise removal process.

[0057] Furthermore, for the spectrum of the observed sound collected by each microphone, the response value of the selected frequency is kept unchanged, and the response value of the frequencies other than the selected frequency is set to 0, thus obtaining the noise spectrum of each microphone.

[0058] The noise spectrum is obtained by averaging the noise spectra of all microphones (averaging all response values ​​at the same frequency separately). In this embodiment, the noise spectrum is referred to as noise.

[0059] Step S202: Using a time-delay estimation positioning method, obtain the observed azimuth and denoised azimuth at each moment based on the observed sound and the filtered sound, respectively. When the first difference between the observed azimuth and the filtered angle is greater than the second difference between the denoised azimuth and the filtered angle, record the second difference.

[0060] The Time Delay Estimation Location (TDOA) method is used to obtain the sound source azimuth of the sound source (i.e., the intrusion target) based on all the observed sounds collected by the microphone array, and it is denoted as the observation azimuth angle (which is also a two-dimensional vector composed of the rotation angle and the elevation angle).

[0061] In addition, the Time Delay Estimation Location (TDOA) method is used to obtain the sound source location (i.e., the intrusion target) based on all the filtered sounds obtained from the microphone array, and it is recorded as the denoised azimuth angle.

[0062] The observed azimuth angle is obtained based on the observed sound containing noise, and therefore has a certain degree of error. The denoised azimuth angle is obtained after the observed sound has undergone random noise reduction, and it cannot directly represent the true location of the sound source.

[0063] The Euclidean distance between the observed azimuth and the filtered angle is denoted as the first difference; the Euclidean distance between the denoised azimuth and the filtered angle is denoted as the second difference.

[0064] When the first difference is less than or equal to the second difference, it indicates that the noise removal effect of the filtered sound after denoising is unreasonable. There may be a situation where the sound source is weakened (or the noise is relatively more pronounced), causing the denoised azimuth angle to deviate excessively from the true azimuth angle of the intruding target. When the first difference is greater than the second difference, it indicates that the noise removal effect of the filtered sound after denoising is usable. There may be a situation where the noise is weakened (or the sound source is relatively more pronounced), causing the denoised azimuth angle to be closer to the true azimuth angle of the intruding target. In this case, the second difference is recorded.

[0065] Thus far, steps S201 to S202 describe the process of acquiring the observed azimuth and the denoised azimuth at each time point, as well as whether to record the second difference.

[0066] Steps S101~S102 are executed synchronously with steps S201~S202.

[0067] Step S301: Obtain the distribution feature A1 of all recorded second differences, obtain the error distribution feature of the filter angle obtained when recording the second differences, and determine whether the difference between A1 and the error distribution feature is less than the first preset threshold.

[0068] With the synchronous execution of steps S101~S102 and S201~S202, at the current moment, the distribution characteristics A1 of all the second differences recorded are obtained; A1 represents the distribution characteristics of all the second differences recorded.

[0069] As an example, this embodiment uses the variance of all second differences as distribution feature A1. Specifically, when the number of all recorded second differences is less than or equal to 4, all steps of the filtering tracking module are not executed.

[0070] For the times when all second differences are recorded, and for all the filter angles obtained at these times, the following processing is performed: Let A2 be the error distribution characteristic of each filter angle. A2 describes the distribution of the error (error relative to the true azimuth of the intruding target) of each filter angle. Average the error distribution characteristics A2 corresponding to all filter angles to obtain the average distribution characteristic a2.

[0071] Furthermore, it is determined whether the difference between A1 and the average distribution feature a2 is small. In this embodiment, if the difference is less than the first preset threshold th1, the difference is determined to be small; otherwise, it is not small.

[0072] When the difference is small, it means that the denoised azimuth angle obtained after denoising is closer to the true azimuth angle of the intruding target and has a small error. This error is close to the error of the filter angle. At this time, the noise removal is relatively reasonable or accurate. The denoised azimuth angle and the filter angle after noise removal can be used together to track the target. The combination of the two can further reduce the error when relying solely on the filter angle for tracking.

[0073] When the difference is large, it means that the denoised azimuth angle obtained after denoising, which is closer to the true azimuth angle of the intruding target, still has a large error. The error is significantly different from that of the filtered angle. In this case, the noise removal is insufficient or inaccurate. The denoised azimuth angle and the filtered angle after noise removal cannot be used together to track the target. Otherwise, it will bring greater uncertainty and error to the tracking process.

[0074] As an optional example, methods for determining whether the difference between A1 and the average distribution characteristic a2 is small include: Using b=(A1-a2) / a2 as the difference, when b is less than th1, the difference is considered to be small. That is, the denoised azimuth angle obtained after denoising is closer to the true azimuth angle of the intrusion target and is close to or matches the error of the filter angle. At this time, the denoised azimuth angle and the filter angle after removing noise can be used together to track the target. The combination of the two can further reduce the error when relying solely on the filter angle for tracking.

[0075] As a preferred example, methods for determining whether the difference between A1 and the average distribution characteristic a2 is small include: First, divide the error distribution characteristics A2 of all filter angles by a2 respectively. Then, the variance of the quotients obtained for all filter angles is denoted as the tracking error stability, expressed as w; the purpose of a2 is to remove the order of magnitude of A2.

[0076] A larger tracking error stability w indicates instability in the error distribution of the filter angle, and the distribution of the second difference recorded based on the filter angle also has uncertainty. In this case, even if b is less than th1 in the above optional example, it cannot be determined whether the distribution of the second difference has reached the level of approaching the error of the filter angle. In other words, a larger tracking error stability w indicates that the obtained distribution feature A1 is inaccurate and cannot be judged solely by b being less than th1. The denoised azimuth angle obtained after denoising is closer to the true azimuth angle of the intruding target and approaches or matches the error of the filter angle. The denoised azimuth angle and the filter angle after noise removal cannot be used together to track the target, otherwise it may bring greater uncertainty to the tracking process.

[0077] In this preferred example, when the difference b obtained from the optional example above is less than th1×(1-w), the difference b is determined to be small. The purpose of this determination condition is that when w is large, it is more difficult to achieve the determination condition. In this case, the acoustic monitoring module will continue to remove noise and record more second differences. On the one hand, this avoids the situation where the denoising azimuth and filtering angles are combined too early when the uncertainty of the filtering angle error distribution is large, which introduces greater uncertainty into the tracking process. On the other hand, it enhances the noise removal strength of the acoustic monitoring module, ensuring the accuracy of noise removal (i.e., ensuring the accuracy and completeness of subsequent reference noise), thereby reducing the error of the denoising azimuth. Furthermore, it can use the denoising azimuth after the error reduction to compensate for the error of the filtering angle with serious uncertainty, thereby ensuring the accuracy and reliability of the tracking process. Specifically, when w is small, for example, when w is less than 0.2, w=0, indicating that when the filtering angle has a definite error distribution, the optional example above is implemented. When w is large, for example, when w is greater than 0.8, w=0.8, the purpose is to avoid the above conditions of this preferred example being difficult to achieve when w is too large.

[0078] In summary, the differences described in this preferred example When b is less than th1, the difference is small.

[0079] At this point, after obtaining the filter angle, observation azimuth angle, and denoising azimuth angle at each time step, it is determined whether the difference b between all the second difference distribution features and the error distribution features of the filter angle is less than the first preset threshold.

[0080] Step S302: When the difference between A1 and the error distribution characteristics at the current time is less than the first preset threshold, the average noise of the noise removed when recording all the second differences is used as the reference noise. The denoised azimuth angle is obtained again after removing the reference noise from the observed sound collected after the current time.

[0081] After obtaining the filter angle, observation azimuth angle and denoising azimuth angle at each moment, if the difference b is not less than the first preset threshold, then at the next moment, the above steps S101~S102 and steps S201~S202 will continue to be executed synchronously.

[0082] Assuming that after obtaining the filter angle, observation azimuth angle, and denoised azimuth angle at the current moment, the difference b is less than the first preset threshold, then the second difference will no longer be recorded, and all recorded second differences will be deleted. Then, the following processing will be performed: For each time point when recording all second differences, noise was removed once in the filtering and tracking module; this noise is a noise spectrum. The average noise spectrum obtained when recording all second differences is calculated (averaging all response values ​​at the same frequency separately), and the resulting average noise spectrum is denoted as the reference noise.

[0083] For each time after the current time, after each microphone collects the observed sound at each time, the reference noise is removed from the observed sound to obtain the filtered sound. Then, the denoised azimuth angle at each time is obtained based on the filtered sound using the time delay estimation positioning method. Subsequently, the denoised azimuth angle is used to achieve the fusion of visual and acoustic modalities.

[0084] As an example, th1 is set to 0.2 in this example. Other examples can set th1 to a larger value, such as 0.5. This allows for earlier fusion of the visual and acoustic modalities, which is particularly suitable for scenarios with low camera resolution, target detection models, or low camera gimbal accuracy. Still other examples can set th1 to a smaller value, such as 0.1, or even less than 0. This delays the fusion process of the visual and acoustic modalities, further removing noise from the observed sound, and is particularly suitable for scenarios with severe environmental noise interference and low microphone accuracy. In this example, the preferred value range for th1 is [-0.1, 0.5].

[0085] As an example, methods for removing reference noise from observed sound to obtain filtered sound include: The response value of each frequency in the spectrum of the observed sound is subtracted from the response value of each frequency in the reference noise to obtain the difference spectrum (response values ​​less than 0 in the difference spectrum are set to 0). The difference spectrum is then transformed using the inverse Fourier transform algorithm to obtain the filtered sound.

[0086] Step S303: The denoised azimuth angle is fused with the filtered angle obtained after the current moment to realize the tracking of the intrusion target after the current moment.

[0087] (1) After obtaining the denoised azimuth and filter angle at each time point after the current time, the average of the denoised azimuth and filter angle is used as the estimated azimuth of the intruding target at each time point. In particular, if the absolute value of the difference between the rotation angle or pitch angle in the denoised azimuth and filter angle is greater than 15°, the estimated azimuth is set to be equal to the filter angle. The purpose is to avoid the situation where the denoised azimuth is obtained incorrectly when the sound source is weak or non-existent.

[0088] (2) Further, the estimated azimuth angle of the intrusion target at each time moment is used as the observation value of the Kalman filter algorithm, and the azimuth angle of the next time moment at each time moment is predicted by the Kalman filter algorithm, which is denoted as the prediction angle.

[0089] The camera's viewing angle at each time step is set to the predicted angle for the next time step. Then, an image is acquired at the next time step, and the azimuth angle at the next time step is obtained (see step S101 for details). For the time series consisting of all the azimuth angles obtained at the next time step and before, the Kalman filter algorithm is used again to filter the time series to obtain the filtered angle for the next time step.

[0090] In some other embodiments, for the azimuth angle at each moment in the time series, if an estimated azimuth angle is calculated at a certain moment, then the estimated azimuth angle is used to replace the azimuth angle at that moment in the time series.

[0091] (3) Then, for the observed sound collected by all microphones in the microphone array at the next moment, after removing the reference noise from the observed sound, the noise-reduced azimuth angle for the next moment is obtained again by using the time delay estimation positioning method.

[0092] Finally, repeat (1) to (3) to achieve long-term accurate tracking of the intrusion target. In this embodiment, the images collected at each moment during the camera tracking process are sent to the display on the server through the 5G network to realize the real-time tracking and monitoring of the intrusion target.

[0093] This concludes the example.

[0094] Example 2: In this embodiment, when using the Time Delay Estimation Location (TDOA) method to obtain the observation azimuth of the sound source (i.e., the intrusion target) based on all the observed sounds collected by the microphone array, it is necessary to ensure that the observation azimuth and the azimuth obtained in step S101 are aligned in the same coordinate system.

[0095] Specifically, for the camera gimbal, when the gimbal rotates in the horizontal plane, all the microphones in the microphone array in this embodiment are evenly distributed in a circle with the rotation center as the center, with a distribution radius of 5 cm, and there are 8 microphones, each with an sampling frequency of 8 kHz.

[0096] With the rotation center as the origin, the horizontal plane containing it is the plane containing the xy-axis of the coordinate system, and the vertically upward direction passing through the origin is the z-axis direction of the coordinate system. In this embodiment, the azimuth angle obtained in step S101 and the sound source azimuth (i.e., the observation azimuth angle) obtained by the TDOA method are both represented in this coordinate system. Both are represented by a two-dimensional vector consisting of the rotation angle (the rotation angle in the plane containing the xy-axis) and the pitch angle (the rotation angle in the vertical plane).

[0097] The method of representing the azimuth and sound source orientation in a coordinate system is well-known geometric knowledge, and the TDOA method is also existing technology, so it will not be described in detail in this embodiment.

[0098] It should be noted that in this embodiment, the camera (or microphone array) is installed at a height higher than all the power equipment in the substation, allowing the camera to capture the entire area of ​​the substation from a top-down perspective. If the obtained estimated azimuth angle has a pitch angle greater than -20° (greater than -20° indicates upward or towards the distance), then the target tracking stops, the tracking process ends, and the camera's view is reset. In this embodiment, the state where the camera rotation angle is 0° and the pitch angle is -45° is taken as the view reset state. In the view reset state, the camera monitors the substation from a top-down perspective and re-detects the bounding box of the target. Since it is impossible to distinguish whether the sound source is above or below the microphone array when all the microphones in the microphone array are distributed on the same plane, all microphones in this embodiment are arranged in a three-dimensional manner. There is a 2 cm height difference between adjacent microphones in the vertical direction, meaning that 4 of the 8 microphones are in the plane containing the x and y axes, and the other 4 are in a horizontal plane 2 cm below the plane containing the x and y axes.

[0099] In particular, considering that there may be multiple intrusion targets, that is, when the camera first detects a target, it detects multiple bounding boxes at the same time; in some embodiments, only the target with the largest bounding box area is retained and tracked, thereby achieving single target tracking.

[0100] In other embodiments, multiple cameras (and microphone arrays) are installed at different locations in the substation, and then a feasible multi-target tracking method is implemented using the following steps: (1) The ID of each target within each camera is obtained using the target re-identification (Re-ID) technology. The ID of the same target obtained in different cameras is the same (the cosine similarity of the IDs is considered to be the same if it is greater than 0.8). Cameras whose bounding boxes are not detected are not considered next. The target re-identification (Re-ID) technology is a well-known technology and will not be described in detail in this embodiment.

[0101] (2) The target with the largest bounding box area in each camera is designated as the target to be tracked for each camera. The camera with the largest bounding box area among all the targets to be tracked by all cameras is designated as the reference camera. If the target to be tracked by a camera other than the reference camera has a different ID than the target to be tracked by the reference camera, then that camera is also designated as the reference camera; the reference camera is marked as being tracked, indicating that it is tracking the target to be tracked, and the target to be tracked is marked as being tracked.

[0102] (3) For cameras other than the reference camera, they are denoted as candidate cameras (the IDs of the targets to be tracked in the candidate cameras and the reference cameras are the same), and the targets to be tracked in the candidate cameras are denoted as tracked states. For all targets marked as tracked states, if the IDs of these targets are the same as the IDs of some targets in the candidate cameras, then that target is also marked as tracked state.

[0103] (4) If there is no target with the second largest bounding box area in the candidate camera, then the view of the candidate camera is reset and the target marked as being tracked is no longer tracked. The candidate camera is marked as standby.

[0104] (5) Next, cameras marked as tracking or standby are no longer considered, and targets marked as being tracked are no longer considered. Based on this, (2) to (4) are repeated until all cameras are marked as tracking or standby.

[0105] When a camera in standby mode detects no target marked as being tracked, it continues to perform multi-target tracking according to the methods described in (1) to (5) above; cameras in tracking mode and targets marked as being tracked no longer participate in the process in (1) to (5). Specifically, when a camera ends the tracking process, it resets its viewpoint and marks itself as standby mode.

[0106] It should be noted that when there are multiple targets, each camera only tracks one target. In this case, the sound sources generated by other targets are regarded as noise. Therefore, even if there is interference from the sound generated by multiple targets, it will not affect the implementation of Example 1.

[0107] Example 3: Example 1 includes removing background noise from the observed sound. As an example, this process specifically includes: When there are no intrusion targets in the substation, the sound collected by each microphone over a continuous 5-minute period is recorded as background sound. The spectrum of the background sound is obtained using a Fast Fourier Transform algorithm. The response value of each frequency in this spectrum is reduced by 20% to obtain the noise spectrum. In this embodiment, once the noise spectrum is obtained, it remains unchanged in subsequent processes, essentially treating the noise spectrum as a constant. In other embodiments, the noise spectrum may be updated once a week.

[0108] For the obtained spectrum of the observed sound, the response value of each frequency in the noise spectrum is subtracted from the response value of each frequency in the spectrum (if the response value is less than 0, it is directly set to 0) to obtain the first spectrum. The first spectrum is then transformed using the inverse Fourier transform algorithm to obtain the observed sound after removing the background noise.

[0109] The purpose of reducing the response value by 20% is to avoid accidentally removing useful sound signals. This process is equivalent to attenuating background noise, rather than completely removing it. In some embodiments, 20% can be replaced with 0%, but this embodiment does not specify a particular value.

[0110] Example 1 includes: randomly selecting several frequencies from the spectrum. As an example, this process specifically includes: For any observed noise spectrum, all frequencies with a response value greater than 0 are designated as candidate frequencies. The Otsu thresholding algorithm is used to divide the response values ​​of all candidate frequencies into two parts: the frequencies with the largest average response values ​​are designated as the first frequency, and the frequencies with the smallest average response values ​​are designated as the second frequency. One-quarter of the first frequency is randomly selected (rounded down), and one-sixth of the second frequency is randomly selected (rounded down). The purpose is that even if noise exists at the second frequency, the interference is relatively small, requiring less frequency selection for noise removal; while at the first frequency, the noise interference is larger, resulting in a more significant noise removal effect and helping to remove the main noise signals from the observed sound.

[0111] Example 1 also includes: denoting the error distribution characteristics of each filtering angle as A2. As an example, the method for obtaining the error distribution characteristics of each filtering angle is as follows: The filtering angle is obtained by the Kalman filtering algorithm described in step S102. The state quantity and the observation quantity in this algorithm are both azimuth angles (that is, the azimuth angles in the time series described in step S102). The observation quantity of the Kalman filtering algorithm in step S303 is the estimated azimuth angle. Since this embodiment does not involve control quantity, the control quantity in this algorithm is set to 0.

[0112] The mean of the diagonal elements in the state covariance matrix of the Kalman filter algorithm at each time step is used as the average variance of each filtering angle, and is used as the error distribution feature A2.

[0113] In some embodiments, the parameters of the Kalman filter algorithm are initialized using the following method: The process of target intrusion is simulated artificially (e.g., different people pass by the camera, or a remotely controlled drone passes by the camera). During each intrusion process, for two adjacent moments, the ratio of the rotation angle and pitch angle of the azimuth angle at the next moment to the rotation angle and pitch angle at the next moment is calculated to obtain the rate of change of the rotation angle and pitch angle respectively. The two-dimensional column vector formed by the mean of the rate of change of the rotation angle and the mean of the rate of change of the pitch angle at all adjacent moments in all intrusion processes is used as the state transition matrix.

[0114] During all intrusion processes, several images (e.g., 5) captured by the camera are randomly sampled. The number of horizontal pixels (h1) and vertical pixels (v1) from the target center to the center of each image are manually labeled. When the camera rotation angle changes, the absolute value of the change in rotation angle (h2) when the center of each image moves one pixel is measured. When the pitch angle changes, the absolute value of the change in pitch angle (v2) when the center of each image moves one pixel is measured. [h1×h2, v1×v2] is used as the error vector for each image. The covariance matrix of all image error vectors is used as the initial value of the state covariance matrix for the Kalman filter algorithm (elements outside the diagonal are set to 0), and also as the measurement noise covariance of the Kalman filter algorithm. Specifically, if the diagonal elements of the initial state covariance matrix approach 0 (e.g., less than or equal to 0.01), then that element is replaced with 0.01. The purpose is to ensure that the randomly sampled samples are not unreasonable, resulting in a situation where the error is 0. In this embodiment, the diagonal elements of the process noise covariance matrix are set to 0.1 (other elements are set to 0).

[0115] The Kalman filter algorithm is a well-known technology, and its specific principle will not be described in detail in this embodiment. In other embodiments, the Kalman filter algorithm can be replaced with the extended Kalman filter algorithm. Note that the computational load will increase after replacing it with the extended Kalman filter algorithm.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cross-modal tracking system for substation intrusion targets integrating vision and acoustics, characterized in that, The system includes the following modules: The visual monitoring module is used to monitor the azimuth angle of an intrusion target at each moment using images captured by a camera, and to filter the azimuth angle using a Kalman filter algorithm to obtain the filtered angle at each moment. The acoustic monitoring module is used to collect the observed sound at each moment based on the microphone array, and randomly remove noise from the observed sound to obtain the filtered sound. The time delay estimation positioning method is used to obtain the observed azimuth angle and the noise-reduced azimuth angle at each moment based on the observed sound and the filtered sound respectively. When the first difference between the observed azimuth angle and the filtered angle is greater than the second difference between the noise-reduced azimuth angle and the filtered angle, the second difference is recorded. The filtering and tracking module is used to, at the current moment, for all recorded second difference distribution features A1 and the error distribution features of the filter angle obtained when recording the second differences, when the difference between A1 and the error distribution features is less than a first preset threshold, use the average noise of the noise removed when recording all second differences as reference noise; the denoised azimuth angle obtained after removing the reference noise from the observed sound collected after the current moment, and the filter angle obtained after the current moment, are used to realize the tracking of the intrusion target after the current moment.

2. The cross-modal tracking system for substation intrusion targets integrating vision and acoustics according to claim 1, characterized in that, The specific steps for obtaining the difference between A1 and the error distribution characteristics are as follows: For all the filter angles obtained when recording the second difference, the mean of the error distribution characteristics of all filter angles is denoted as a2; the distribution difference between the error distribution characteristics of all filter angles is denoted as tracking error stability; the difference between A1 and the error distribution characteristics is positively correlated with A1-a2 and tracking error stability, respectively.

3. The cross-modal tracking system for substation intrusion targets integrating vision and acoustics according to claim 1, characterized in that, The specific steps involved in randomly removing noise from the observed sound to obtain the filtered sound are as follows: For the observed sound collected by each microphone in the microphone array, the spectrum of the observed sound is obtained using the Fast Fourier Transform algorithm. The spectrum contains the response values ​​of the observed sound at different frequencies. Several frequencies are randomly selected in the spectrum. In the spectrum of the observed sound collected by all microphones, the response values ​​corresponding to all selected frequencies are first set to 0. Then, the spectrum is transformed into filtered sound using the Inverse Fourier Transform algorithm. In this process, for the spectrum of the observed sound collected by each microphone, the response value of the selected frequency remains unchanged, while the response value of the frequencies other than the selected frequency is set to 0, thus obtaining the noise spectrum of each microphone, which is recorded as the noise randomly removed from the observed sound.

4. The cross-modal tracking system for substation intrusion targets integrating vision and acoustics according to claim 3, characterized in that, The specific steps involved in using the average noise removed when recording all second differences as the reference noise are as follows: For the noise that was randomly removed when recording all the second differences, the average noise spectrum corresponding to all the noise was calculated, and the resulting average noise spectrum was denoted as the reference noise.

5. The cross-modal tracking system for substation intrusion targets integrating vision and acoustics according to claim 4, characterized in that, The denoised azimuth angle obtained by removing reference noise from the observed sound collected after the current moment includes the following specific steps: For any time after the current time, and for the spectrum of the observed sound collected by each microphone in the microphone array at that time, the response value of each frequency in the spectrum is subtracted from the response value of each frequency in the reference noise to obtain the difference spectrum. The difference spectrum is transformed using the inverse Fourier transform algorithm to obtain the filtered sound. The denoised azimuth angle is obtained based on all the filtered sounds obtained by the microphone array using the time delay estimation positioning method.

6. The cross-modal tracking system for substation intrusion targets integrating vision and acoustics according to claim 1, characterized in that, The denoised azimuth angle obtained by removing reference noise from the observed sound collected after the current moment, and the filtered angle obtained after the current moment, are used to track the intrusion target after the current moment. The specific steps include the following: The denoised azimuth angle and the filtered angle obtained at each time after the current time are averaged to obtain the estimated azimuth angle of the intrusion target at each time. The estimated azimuth angle of the intrusion target at each time is used as the observation value of the Kalman filter algorithm. The azimuth angle of the next time at each time is predicted by the Kalman filter algorithm and recorded as the prediction angle. The camera's viewing angle is controlled to face the prediction angle to realize the tracking of the intrusion target.

7. The cross-modal tracking system for substation intrusion targets integrating vision and acoustics according to claim 3, characterized in that, The specific steps involved in randomly selecting several frequencies from the spectrum are as follows: For any observed noise spectrum, all frequencies with a response value greater than 0 in the spectrum are denoted as candidate frequencies. The Otsu threshold segmentation algorithm is used to divide the response values ​​of all candidate frequencies into two parts. The frequency of the part with the largest average response value is denoted as the first frequency, and the frequency of the part with the smallest average response value is denoted as the second frequency. Frequencies are selected from the first frequency and the second frequency respectively, wherein the percentage of the number of frequencies selected from the first frequency is greater than the percentage of the number of frequencies selected from the second frequency.

8. The cross-modal tracking system for substation intrusion targets integrating vision and acoustics according to claim 2, characterized in that, The distribution feature A1 is the variance of all recorded second differences.

9. The cross-modal tracking system for substation intrusion targets integrating vision and acoustics according to claim 2, characterized in that, The error distribution characteristics of each filtering angle are determined by the state covariance matrix of the Kalman filtering algorithm.

10. The cross-modal tracking system for substation intrusion targets integrating vision and acoustics according to claim 2, characterized in that, The distribution differences among the error distribution characteristics of all filtering angles are denoted as tracking error stability, and the specific steps involved are as follows: First, divide the error distribution characteristics of all filter angles by a2 respectively, and then denote the variance of the quotients obtained for all filter angles as the tracking error stability.

Images