Portable Multimodal Law Enforcement Evidence Collection and Equipment Status Monitoring Integrated Method and System

By combining a ring-shaped acoustic sensor array and optical imaging module with adaptive beamforming and convolutional neural networks, a portable multimodal law enforcement evidence collection and equipment status monitoring system has been integrated, solving the problems of inaccurate sound source localization and equipment redundancy in existing technologies, and improving the accuracy and flexibility of evidence collection and equipment monitoring.

CN122260233BActive Publication Date: 2026-11-17SHEGU (TAIZHOU) TECHNOLOGY CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202610238070.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-02-28
Publication Date
2026-11-17
Estimated Expiration
2046-02-28

AI Technical Summary

Technical Problem

Existing law enforcement evidence collection and equipment status monitoring systems lack precise positioning capabilities and cannot effectively distinguish between multiple sound sources, resulting in unclear sound sources and insufficient persuasiveness of evidence during the evidence collection process. Furthermore, the equipment is redundant, inconvenient to carry, and makes it difficult to achieve comprehensive data analysis and processing.

Method used

An acoustic sensor array and optical imaging module with a ring layout are used to calculate the three-dimensional spatial coordinates of the sound source through adaptive beamforming processing. The coordinates are then combined with visual image data for time alignment, and a convolutional neural network is used for fault feature extraction and classification, thus realizing the integration of sound source imaging and equipment status monitoring.

Benefits of technology

It achieves high-precision sound source localization and equipment fault diagnosis, improves the accuracy of evidence collection and the flexibility of equipment status monitoring, and ensures the credibility of evidence and the timely detection and handling of equipment status.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122260233B_ABST
    Figure CN122260233B_ABST
Patent Text Reader

Abstract

This invention provides a portable, multimodal integrated method and system for law enforcement evidence collection and equipment status monitoring, relating to the field of law enforcement evidence collection and equipment monitoring technology. It includes acquiring data through a ring-shaped acoustic sensor array and optical modules, performing beamforming processing to locate the sound source and generate a sound source image; marking the sound source location in law enforcement mode and storing it with visual data; and analyzing frequency domain features to identify fault types in detection mode. This invention achieves integrated law enforcement evidence collection and equipment fault detection functions, improving the spatial positioning accuracy of law enforcement evidence collection and the accuracy of equipment status monitoring.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of law enforcement evidence collection and equipment monitoring technology, and in particular to a portable multimodal integrated method and system for law enforcement evidence collection and equipment status monitoring. Background Technology

[0002] Portable multimodal law enforcement evidence collection and equipment condition monitoring technologies have wide applications in public safety, law enforcement supervision, and industrial equipment maintenance. With the advancement of technology, acoustic signal processing and visual image analysis technologies have become increasingly mature, and combining these two technologies for law enforcement evidence collection and equipment condition monitoring has become an industry trend. Currently, law enforcement agencies typically use audio and video recording equipment to record evidence during on-site evidence collection, while equipment condition monitoring mainly relies on specialized testing instruments for single-function testing.

[0003] Traditional law enforcement evidence collection and equipment monitoring systems have significant technical limitations. They lack the ability to accurately locate sound sources and cannot effectively distinguish between multiple sound sources in complex environments, leading to unclear sound sources and insufficient persuasiveness of evidence during the evidence collection process. Existing technical solutions typically design law enforcement evidence collection and equipment status monitoring as two independent systems, resulting in redundant and inconvenient equipment, and difficulty in achieving comprehensive data analysis and processing, failing to meet the actual needs of rapid on-site response. With the advancement of artificial intelligence and signal processing technologies, it has become possible to apply multimodal information fusion to law enforcement evidence collection and equipment status monitoring. However, how to construct an integrated solution that combines high sound source localization accuracy, strong fault diagnosis capabilities, and portability and ease of use has become a pressing technical challenge in this field. Summary of the Invention

[0004] The present invention provides a portable multimodal law enforcement evidence collection and equipment status monitoring integrated method and system, which can solve the problems in the prior art.

[0005] A first aspect of the present invention provides a portable, multimodal law enforcement evidence collection and device status monitoring integrated method, comprising: Acquire omnidirectional acoustic signals collected by an array of acoustic sensors arranged in a ring, and visual image data acquired by an optical imaging module; Adaptive beamforming is applied to the omnidirectional acoustic signal to calculate the arrival time difference between each acoustic sensor. The three-dimensional spatial coordinates of the sound source are obtained by solving the time delay equations. A sound source imaging map is generated based on the sound field intensity data corresponding to the three-dimensional spatial coordinates. When in law enforcement evidence collection mode, the three-dimensional spatial coordinates of the sound source are marked in the sound source imaging image, and the visual image data and the omnidirectional acoustic signal are time-aligned and encoded for storage; when in device detection mode, the ultrasonic frequency band sensor and vibration sensor are activated to collect the device status signal of the device under test. The frequency domain feature data is obtained by performing spectral transformation on the device status signal, a multi-type fault feature library is constructed, and deep feature extraction and classification verification are performed on the frequency domain feature data through a convolutional neural network to output fault type identification and confidence level. The fault type identifier of the device under test is marked in the sound source imaging image and displayed synchronously with the visual image data. The confidence level is used to determine whether an alarm is triggered, and a device fault detection record is generated and stored.

[0006] Adaptive beamforming is applied to the omnidirectional acoustic signal to calculate the arrival time difference between each acoustic sensor. The three-dimensional spatial coordinates of the sound source are obtained by solving a set of time delay equations. A sound source imaging map is generated based on the sound field intensity data corresponding to the three-dimensional spatial coordinates, including: The omnidirectional acoustic signal is transformed by time and frequency to obtain the energy distribution in the time and frequency domain and to identify sparse time and frequency points. The sparse time and frequency points are determined by calculating the phase difference variance between each acoustic sensor. The time and frequency points whose phase difference variance is less than the sparse threshold are identified as the sparse time and frequency points. For each of the sparse time-frequency points, the arrival time difference is calculated based on the phase difference between each acoustic sensor, a set of time delay geometric equations is constructed and solved to obtain the candidate sound source positions corresponding to the sparse time-frequency points. Calculate the Euclidean distance between the candidate sound source positions corresponding to the sparse time-frequency points, group the candidate sound source positions whose Euclidean distance is less than the cluster radius into the same cluster, calculate the centroid coordinates of each cluster as the three-dimensional spatial coordinates of each sound source, and construct a guiding vector pointing to the three-dimensional spatial coordinates. The adaptive weighting coefficients of each acoustic sensor are calculated based on the guidance vector, and the omnidirectional acoustic signals are weighted and superimposed to obtain the beamforming output signals of each three-dimensional spatial coordinate. The cumulative energy values ​​of the sparse time-frequency points associated with each three-dimensional spatial coordinate in the time-frequency domain are statistically analyzed and used as the sound field intensity data corresponding to the three-dimensional spatial coordinates. The three-dimensional spatial coordinates of each sound source are then mapped to the spatial display coordinate system to generate the sound source imaging map containing the location and intensity distribution of multiple sound sources.

[0007] For each of the sparse time-frequency points, the arrival time difference is calculated based on the phase difference between each acoustic sensor, a set of time-delay geometric equations is constructed and solved to obtain the candidate sound source locations corresponding to the sparse time-frequency points, including: For each of the sparse time-frequency points, the complex representation of each acoustic sensor in the time-frequency domain is extracted and phase subtraction is performed to obtain the phase difference between each pair of acoustic sensors; The phase difference is unwound, the periodic ambiguity is eliminated by constraining the phase difference continuity of adjacent acoustic sensor pairs, the phase difference that jumps beyond the circumferential phase is compensated for with integer periods, and the arrival time difference is calculated based on the unwound phase difference and angular frequency. Based on the spatial coordinates of each acoustic sensor and the arrival time difference, a set of time delay geometric equations is constructed. Each time delay geometric equation expresses the relationship between the propagation distance difference from the sound source to different acoustic sensor pairs and the product of the arrival time difference and the speed of sound. Hyperbolic surface intersection analysis is performed on the set of time delay geometric equations to determine the intersection region as the sound source search space, with the geometric center of the search space as the initial value. The nonlinear optimization method minimizes the sum of squared residuals, calculates the residual vector between the theoretical arrival time difference and the actual arrival time difference corresponding to the current estimated value, and determines whether the norm of the residual vector is less than the convergence threshold. If it is not satisfied, the estimated value is corrected according to the gradient direction and iterated. If it is satisfied, the current estimated value is output as the candidate location of the sound source corresponding to the sparse time-frequency point.

[0008] When in law enforcement evidence collection mode, the three-dimensional spatial coordinates of the sound source are marked in the sound source imaging image. The visual image data and the omnidirectional acoustic signal are time-aligned and encoded for storage, including: The three-dimensional spatial coordinates of the sound source are marked in the sound source imaging image. The spatial orientation of the sound source relative to the visual image acquisition device is calculated. It is determined whether the sound source is within the field of view of the visual image data. When the sound source is within the field of view, a visual indicator mark synchronized with the three-dimensional spatial coordinates is superimposed in the visual image data. Extract the frame-level timestamp of the visual image data and the sampling-level timestamp of the omnidirectional acoustic signal, calculate the time alignment window between the frame-level timestamp and the sampling-level timestamp, and adaptively determine the window width according to the frame rate of the visual image data and the sampling rate of the omnidirectional acoustic signal; Within the time alignment window, time synchronization processing is performed on the visual image data and the omnidirectional acoustic signal, and a time mapping relationship between the frame level and the sampling level is established by timestamp interpolation method. A multimodal synchronization data structure is constructed, which includes the aligned visual image data, the omnidirectional acoustic signal, the sound source imaging map, and the time mapping relationship; hierarchical encoding is performed on the multimodal synchronization data structure, and the encoding result is encapsulated into an evidence file with a timestamp index and stored.

[0009] The device status signal is subjected to spectral transformation to obtain frequency domain feature data. A multi-type fault feature library is constructed, and deep feature extraction and classification verification are performed on the frequency domain feature data through a convolutional neural network. The fault type identifier and confidence level are output, including: The device status signal is subjected to adaptive window length spectrum transformation to extract frequency domain peak frequency features, sideband modulation features and frequency offset features. The correlation strength between each feature and the fault type is calculated through mutual information analysis. Features with correlation strength exceeding the correlation threshold are selected to form frequency domain feature data. Establish a hierarchical fault feature library architecture that supports dynamic expansion. Store the frequency domain feature data of known fault types in a hierarchical index according to equipment type and fault severity. Reserve the feature space interface for adding new fault types to form a multi-type fault feature library that can be incrementally updated. The frequency domain feature data is input into a convolutional neural network. The receptive field range is adaptively adjusted by a deformable convolutional kernel to extract local and global features. After passing through a feature fusion layer and a classification decision layer, a preliminary classification result is output. The preliminary classification result is matched and verified with the historical diagnostic cases in the multi-type fault feature library. When the matching similarity is higher than the verification threshold, the preliminary classification result is confirmed as the fault type identifier. The final confidence level is calculated by weighted fusion based on the matching similarity and the probability value output by the convolutional neural network.

[0010] The frequency domain feature data is input into a convolutional neural network. The receptive field is adaptively adjusted using deformable convolutional kernels to extract local and global features. After passing through a feature fusion layer and a classification decision layer, a preliminary classification result is output, including: The frequency domain feature data is input into the convolutional neural network, and initial receptive field parameters of different scales are set in each branch. The receptive field range of the deformable convolutional kernel of each branch is adjusted by offset learning. High-frequency detail features in the frequency domain feature data are extracted as local features by deformable convolution kernels with receptive fields of different scales, and low-frequency envelope features in the frequency domain feature data are extracted as global features. A cross-scale feature alignment mechanism is established in the feature fusion layer. The local and global features extracted from different branches are aligned and transformed in terms of spatial and channel dimensions. The fusion coefficients are dynamically calculated through a learnable fusion weight network to generate multi-scale fusion features. The multi-scale fused features are input into the classification decision layer, and feature dimensionality reduction and nonlinear mapping are performed through cascaded fully connected layers to output the classification scores of each fault type. The classification scores are then activated by a function that adjusts the temperature parameter to obtain the calibrated fault type probability distribution as the preliminary classification result.

[0011] A second aspect of the present invention provides a portable multimodal law enforcement evidence collection and equipment status monitoring integrated system, comprising: The first unit is used to acquire omnidirectional acoustic signals collected by an array of acoustic sensors arranged in a ring, as well as visual image data collected by an optical imaging module. The second unit is used to perform adaptive beamforming processing on the omnidirectional acoustic signal, calculate the arrival time difference between each acoustic sensor, obtain the three-dimensional spatial coordinates of the sound source by solving the time delay equations, and generate a sound source imaging map based on the sound field intensity data corresponding to the three-dimensional spatial coordinates. The third unit is used to mark the three-dimensional spatial coordinates of the sound source in the sound source imaging map when in law enforcement evidence collection mode, and to perform time alignment and encoding storage by combining the visual image data with the omnidirectional acoustic signal; when in equipment detection mode, it activates the ultrasonic frequency band sensor and vibration sensor to collect the equipment status signal of the equipment to be detected. The fourth unit is used to perform spectral transformation on the device status signal to obtain frequency domain feature data, construct a multi-type fault feature library, and perform deep feature extraction and classification verification on the frequency domain feature data through a convolutional neural network, and output fault type identifier and confidence level. The fifth unit is used to mark the fault type identifier of the device under test in the sound source imaging image, display it synchronously with the visual image data, determine whether to trigger an alarm based on the confidence level, generate a device fault detection record and store it.

[0012] A third aspect of the present invention, An electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0013] Fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0014] The beneficial effects of this application are as follows: This invention acquires omnidirectional acoustic signals by using a ring-shaped acoustic sensor array and combines them with visual image data from an optical imaging module, thereby achieving the fusion processing of acoustic and visual data and significantly improving the comprehensiveness and accuracy of information acquisition.

[0015] Adaptive beamforming technology is used to process omnidirectional acoustic signals. By calculating the arrival time difference between acoustic sensors and solving the time delay equations, the three-dimensional spatial coordinates of the sound source are accurately obtained, achieving high-precision sound source localization and improving the accuracy and reliability of evidence collection.

[0016] It innovatively integrates law enforcement evidence collection and equipment status monitoring into one device, automatically switching sensor combinations and processing flows according to different modes, greatly improving the practicality and flexibility of the equipment.

[0017] In equipment detection mode, by acquiring ultrasonic frequency bands and vibration signals, combined with spectrum analysis and deep learning technology, a multi-type fault feature library is constructed, enabling intelligent identification and classification of equipment faults and significantly improving diagnostic accuracy. The sound source localization results and fault diagnosis information are intuitively marked on the sound source imaging map and displayed synchronously with the visual image. Simultaneously, an alarm mechanism based on confidence level ensures timely detection and handling of abnormal conditions. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the portable multimodal law enforcement evidence collection and equipment status monitoring integrated method according to an embodiment of the present invention; Figure 2 This is a schematic flowchart of the method for generating a sound source imaging map according to an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0021] Figure 1 This is a flowchart illustrating the portable multimodal law enforcement evidence collection and equipment status monitoring integrated method according to an embodiment of the present invention. Figure 1 As shown, the method includes: Acquire omnidirectional acoustic signals collected by an array of acoustic sensors arranged in a ring, and visual image data acquired by an optical imaging module; Adaptive beamforming is applied to the omnidirectional acoustic signal to calculate the arrival time difference between each acoustic sensor. The three-dimensional spatial coordinates of the sound source are obtained by solving the time delay equations. A sound source imaging map is generated based on the sound field intensity data corresponding to the three-dimensional spatial coordinates. When in law enforcement evidence collection mode, the three-dimensional spatial coordinates of the sound source are marked in the sound source imaging image, and the visual image data and the omnidirectional acoustic signal are time-aligned and encoded for storage; when in device detection mode, the ultrasonic frequency band sensor and vibration sensor are activated to collect the device status signal of the device under test. The frequency domain feature data is obtained by performing spectral transformation on the device status signal, a multi-type fault feature library is constructed, and deep feature extraction and classification verification are performed on the frequency domain feature data through a convolutional neural network to output fault type identification and confidence level. The fault type identifier of the device under test is marked in the sound source imaging image and displayed synchronously with the visual image data. The confidence level is used to determine whether an alarm is triggered, and a device fault detection record is generated and stored.

[0022] Figure 2 This is a schematic flowchart of a method for generating a sound source imaging map according to an embodiment of the present invention. In one optional embodiment, the omnidirectional acoustic signal is subjected to adaptive beamforming processing, the arrival time difference between each acoustic sensor is calculated, the three-dimensional spatial coordinates of the sound source are obtained by solving a set of time delay equations, and a sound source imaging map is generated based on the sound field intensity data corresponding to the three-dimensional spatial coordinates, including: The omnidirectional acoustic signal is transformed by time and frequency to obtain the energy distribution in the time and frequency domain and to identify sparse time and frequency points. The sparse time and frequency points are determined by calculating the phase difference variance between each acoustic sensor. The time and frequency points whose phase difference variance is less than the sparse threshold are identified as the sparse time and frequency points. For each of the sparse time-frequency points, the arrival time difference is calculated based on the phase difference between each acoustic sensor, a set of time delay geometric equations is constructed and solved to obtain the candidate sound source positions corresponding to the sparse time-frequency points. Calculate the Euclidean distance between the candidate sound source positions corresponding to the sparse time-frequency points, group the candidate sound source positions whose Euclidean distance is less than the cluster radius into the same cluster, calculate the centroid coordinates of each cluster as the three-dimensional spatial coordinates of each sound source, and construct a guiding vector pointing to the three-dimensional spatial coordinates. The adaptive weighting coefficients of each acoustic sensor are calculated based on the guidance vector, and the omnidirectional acoustic signals are weighted and superimposed to obtain the beamforming output signals of each three-dimensional spatial coordinate. The cumulative energy values ​​of the sparse time-frequency points associated with each three-dimensional spatial coordinate in the time-frequency domain are statistically analyzed and used as the sound field intensity data corresponding to the three-dimensional spatial coordinates. The three-dimensional spatial coordinates of each sound source are then mapped to the spatial display coordinate system to generate the sound source imaging map containing the location and intensity distribution of multiple sound sources.

[0023] In this specific embodiment, omnidirectional acoustic signals collected by a ring-shaped acoustic sensor array and visual image data collected by an optical imaging module are acquired. The ring-shaped acoustic sensor array includes eight omnidirectional sensors that are evenly distributed on a circumference. The optical imaging module includes one or more cameras to acquire visual images of the spatial scene and provide a visual reference for the scene.

[0024] Time-frequency analysis was performed on the acquired omnidirectional acoustic signals. The time-domain signals were converted to the time-frequency domain using short-time Fourier transform (STFT) to obtain the time spectrum of the signals acquired by each acoustic sensor. The acoustic sensor array consists of 8 omnidirectional sensors with a signal sampling rate of 48kHz, a frame length of 1024 points, and a frame shift of 512 points. After the transformation, the complex coefficients of each sensor at each time-frequency point can be obtained.

[0025] Identifying sparse time-frequency points in the time-frequency domain is a crucial step. The phase difference variance between each sensor is calculated to determine if a time-frequency point is sparse. Specifically, for any time frame t and frequency point f, the phase difference between the i-th and j-th sensors is calculated, and then the variance of the phase differences between all sensor pairs is calculated. If this variance value is less than a preset sparsity threshold (e.g., 0.15), the time-frequency point is identified as a sparse point, indicating that it is mainly contributed by a single sound source. In practice, the sparsity of speech signals in the time-frequency domain means that most time-frequency points originate primarily from a single sound source, providing a foundation for subsequent localization.

[0026] For each sparse time-frequency point, the time difference of arrival (TDOA) between the sensors is calculated based on the phase difference. Specifically, the TDOA value is obtained by dividing the phase difference by the angular frequency. For example, for a signal with frequency f, the TDOA between sensors i and j can be expressed as the phase difference between the two sensors divided by 2πf. Considering that the microphone array contains N sensors, N(N-1) / 2 TDOA measurements can be obtained.

[0027] Based on the obtained TDOA values, a system of time-delay geometric equations is constructed. Assuming the sound source location is an unknown variable S(x, y, z) and the sensor location is a known variable Mi(xi, yi, zi), the TDOA between sensors i and j can be expressed as the difference in distance from the sound source to the two sensors divided by the sound speed c. An equation system is constructed for all sensor pairs, and the least squares method is used to solve this overdetermined system of equations to obtain the candidate sound source locations corresponding to sparse time-frequency points.

[0028] To handle multi-source scenarios, it is necessary to cluster the candidate locations of all sparse time-frequency points. The Euclidean distance between candidate locations is calculated, and a density-based spatial clustering method is used to group candidate locations with Euclidean distances smaller than the cluster radius (e.g., 0.5 meters) into the same cluster. For each cluster, the average coordinates of all points within it are calculated as the three-dimensional spatial coordinates of the sound source. This method effectively addresses noise and reflection interference, improving positioning accuracy.

[0029] After obtaining the spatial coordinates of the sound sources, steering vectors pointing to the locations of each sound source are constructed. For a given frequency *f* and sound source location, the propagation delay of the sound wave from the source to each sensor is calculated based on the sound wave propagation model. The propagation delay is equal to the Euclidean distance from the source to the sensor divided by the speed of sound. Based on the propagation delay, the phase difference of the signals received by each sensor is calculated, forming a steering vector. This vector, after normalization, is used to indicate the phase relationship of the signals received by each sensor when the sound wave propagates from a specific direction.

[0030] Based on the steering vector, the Minimum Variance Distortionless Response (MVDR) beamforming algorithm is used to calculate the adaptive weighting coefficients for each sensor. Specifically, the noise covariance matrix is ​​estimated, and weighting coefficients are calculated based on the steering vector and the covariance matrix to minimize the output noise power of the beamformer while maintaining the signal in the direction of the sound source without distortion. These weighting coefficients are then applied to the omnidirectional acoustic signal and superimposed to obtain the beamformed output signal at each three-dimensional spatial coordinate point.

[0031] The cumulative energy in the time-frequency domain of sparse time-frequency points associated with each three-dimensional spatial coordinate is statistically analyzed and used as the sound field intensity data for that coordinate point. The three-dimensional spatial coordinates of the sound source and its corresponding intensity data are mapped onto a spatial display coordinate system to generate a sound source image. Specifically, the space can be divided into a grid, the sound field intensity can be calculated for each grid point, and a pseudo-color image can be used to represent the intensity distribution, where the color depth corresponds to the magnitude of the sound field intensity.

[0032] Through the aforementioned technologies, this method can accurately locate multiple sound sources in complex acoustic environments and generate high-resolution sound source images, providing a powerful tool for acoustic monitoring and analysis.

[0033] In one optional implementation, for each of the sparse time-frequency points, the arrival time difference is calculated based on the phase difference between each acoustic sensor, a set of time-delay geometric equations is constructed and solved to obtain the candidate sound source locations corresponding to the sparse time-frequency points, including: For each of the sparse time-frequency points, the complex representation of each acoustic sensor in the time-frequency domain is extracted and phase subtraction is performed to obtain the phase difference between each pair of acoustic sensors; The phase difference is unwound, the periodic ambiguity is eliminated by constraining the phase difference continuity of adjacent acoustic sensor pairs, the phase difference that jumps beyond the circumferential phase is compensated for with integer periods, and the arrival time difference is calculated based on the unwound phase difference and angular frequency. Based on the spatial coordinates of each acoustic sensor and the arrival time difference, a set of time delay geometric equations is constructed. Each time delay geometric equation expresses the relationship between the propagation distance difference from the sound source to different acoustic sensor pairs and the product of the arrival time difference and the speed of sound. Hyperbolic surface intersection analysis is performed on the set of time delay geometric equations to determine the intersection region as the sound source search space, with the geometric center of the search space as the initial value. The nonlinear optimization method minimizes the sum of squared residuals, calculates the residual vector between the theoretical arrival time difference and the actual arrival time difference corresponding to the current estimated value, and determines whether the norm of the residual vector is less than the convergence threshold. If it is not satisfied, the estimated value is corrected according to the gradient direction and iterated. If it is satisfied, the current estimated value is output as the candidate location of the sound source corresponding to the sparse time-frequency point.

[0034] In this specific embodiment, for the sparse time-frequency points obtained through time-frequency analysis, the complex representation of each acoustic sensor in the time-frequency domain is extracted. The acoustic signals collected by the eight omnidirectional acoustic sensors in a ring layout are represented in the time-frequency domain as complex form X_i(t, f) after short-time Fourier transform, where i is the sensor number, t is the time frame index, and f is the frequency point index. For the identified sparse time-frequency points (t, f), the complex representation of each sensor is extracted, and the complex phase angle φ_i(t, f) is calculated. For any two sensors i and j, the phase difference φ_ij(t, f) = φ_i(t, f) - φ_j(t, f) is calculated, resulting in a total of twenty-eight pairs of sensor phase difference values.

[0035] To eliminate periodic ambiguity, the phase difference is unwrapped. Since the phase value is limited to the range of [-π, π], periodic folding occurs when the actual phase difference exceeds this range, leading to errors in the calculated arrival time difference. The unwrapping process must start from the low-frequency band, assuming that there is no folding in the phase difference at low frequencies, and gradually unfold towards the high-frequency band. If the phase difference change between adjacent frequency points exceeds π, it is determined as a phase jump, and it needs to be corrected by adding or subtracting an integer multiple of 2π. In the specific implementation, the threshold for the phase difference change between frequency points f and f+1 is set to π / 2. If |φ_ij(t, f+1) - φ_ij(t, f)| > π / 2, then φ_ij(t, f+1) is added or subtracted by 2π × n to minimize |φ_ij(t, f+1) - φ_ij(t, f)|, where n is an integer. The optimal value is determined through iteration. The unwrapped phase difference is divided by the angular frequency 2π × f to obtain the arrival time difference τ_ij(t, f) of the sensor for ij.

[0036] Based on the spatial coordinates and arrival time differences of each acoustic sensor, a set of time-delay geometric equations is constructed. In the portable device, the acoustic sensors are arranged on a ring array, and the spatial coordinates of each sensor are precisely calibrated. If the location of the sound source is an unknown variable s=(x, y, z), and the location of sensor i is m_i=(x_i, y_i, z_i), then the distance from the sound source to sensor i is d_i=||s-m_i||. According to the principle of sound wave propagation, the arrival time difference τ_ij between sensors i and j is related to the difference between the two distances: c×τ_ij=d_i-d_j, where c is the speed of sound. For an array of 8 sensors, 28 equations are constructed, forming an overdetermined set of equations. Each equation represents a hyperboloid, and the sound source location must be near the intersection region of all hyperboloids. In actual monitoring, this step transforms the time difference information into spatial geometric constraints, laying the foundation for accurately locating abnormal sound sources in the equipment.

[0037] Hyperbolic intersection analysis was performed on the time delay geometric equations to determine the search space. The monitoring space was divided into grids with a grid spacing of 10 cm. For each grid point, the sum of squared errors between the theoretical arrival time difference and the actual measured arrival time difference was calculated. The set of grid points with smaller errors constituted the sound source search space. The geometric center point s_0=(x_0, y_0, z_0) in the search space was used as the initial value for nonlinear optimization. Selecting the geometric center rather than the point with the minimum error as the initial value can improve the stability of the optimization and avoid getting trapped in local minima.

[0038] The sound source location is accurately calculated using a nonlinear optimization method. The Levenberg-Marquardt algorithm is used to iteratively solve for the optimal sound source location. The specific process is as follows: Calculate the theoretical time difference of arrival τ based on the current estimated value s_k. ij k Secondly, calculate the time difference τ between the actual arrival time and the actual arrival time. ij The residual vector r k Sum of squared residuals E k =Σ(τ ij -τ ij k ) 2 ; Calculate the Jacobian matrix J of the residual relative to the sound source coordinates. k Construct the normal equation (J k T ·J k +λ·I)·Δs k =-J k T ·r k Where λ is the damping factor, which can be dynamically adjusted according to the iterative performance; the update quantity Δs is obtained by solving the normal equation. k Update the sound source location estimate s k+1 =s k+Δs k Determine whether the sum of squared residuals is less than the preset convergence threshold (e.g., 0.001) or whether the update norm is less than the accuracy requirement (e.g., 0.001m). If the conditions are met, terminate the iteration; otherwise, continue. To improve stability, the maximum number of iterations is set to 20. After the iteration ends, output the optimized sound source position as the candidate sound source position corresponding to the sparse time-frequency point.

[0039] To further enhance the robustness of the optimization algorithm, a weighting mechanism is introduced to handle outliers. The measurement error of each sensor pair is evaluated, with higher-reliability sensor pairs assigned greater weights and lower-reliability pairs assigned smaller weights. In some cases, obviously anomalous measurements are even removed. The weights can be determined based on the signal cross-correlation between sensor pairs or the sensor signal-to-noise ratio. The optimization objective is modified to the weighted sum of squared residuals E. k =Σw ij ·(τ ij -τ ij k ) 2 , where w ij is the weighting coefficient of the sensor for ij. In noisy law enforcement scenes or when the equipment is partially obstructed, this weighting strategy can effectively reduce the impact of abnormal measurements on the positioning results.

[0040] In actual law enforcement and evidence collection scenarios, each candidate sound source location can be evaluated using confidence level assessment. Confidence level can be comprehensively assessed through multiple indicators, including the magnitude of the residual sum of squares, the optimization convergence speed, and the correlation with the energy of time and frequency points. Candidate locations with high confidence level are given higher weight in subsequent spatial clustering, thereby improving the accuracy of the final sound source localization. For example, when monitoring abnormal sounds emitted by suspicious luggage in airport security check areas, confidence level assessment can quickly filter out environmental noise interference and accurately locate the suspicious luggage.

[0041] For sparse time-frequency points in different frequency bands, a frequency band adaptive processing strategy can be adopted. In the low-frequency band (less than 500 Hz), the sound wave wavelength is longer and the diffraction effect is significant. The unwinding threshold can be appropriately relaxed and the search space can be increased. In the high-frequency band (greater than 2000 Hz), the directionality is stronger and the positioning accuracy is higher, but it is easily affected by obstruction. The convergence accuracy requirement can be appropriately increased. In equipment condition monitoring, this frequency band adaptive strategy can simultaneously capture low-frequency vibration anomalies and high-frequency friction noise of the equipment, and comprehensively assess the health status of the equipment.

[0042] A temporal consistency check is performed on candidate sound source locations at multiple sparse time-frequency points. The location of the same sound source should remain relatively stable in adjacent time frames. Temporal filtering techniques such as Kalman filtering can be used to smooth the sound source trajectory and suppress transient localization errors. This method is particularly suitable for tracking moving sound sources during law enforcement operations, such as tracing the movement path of objects in complex environments.

[0043] In one optional implementation, when in law enforcement evidence collection mode, the three-dimensional spatial coordinates of the sound source are marked in the sound source imaging image, and the visual image data and the omnidirectional acoustic signal are time-aligned and encoded for storage, including: The three-dimensional spatial coordinates of the sound source are marked in the sound source imaging image. The spatial orientation of the sound source relative to the visual image acquisition device is calculated. It is determined whether the sound source is within the field of view of the visual image data. When the sound source is within the field of view, a visual indicator mark synchronized with the three-dimensional spatial coordinates is superimposed in the visual image data. Extract the frame-level timestamp of the visual image data and the sampling-level timestamp of the omnidirectional acoustic signal, calculate the time alignment window between the frame-level timestamp and the sampling-level timestamp, and adaptively determine the window width according to the frame rate of the visual image data and the sampling rate of the omnidirectional acoustic signal; Within the time alignment window, time synchronization processing is performed on the visual image data and the omnidirectional acoustic signal, and a time mapping relationship between the frame level and the sampling level is established by timestamp interpolation method. A multimodal synchronization data structure is constructed, which includes the aligned visual image data, the omnidirectional acoustic signal, the sound source imaging map, and the time mapping relationship; hierarchical encoding is performed on the multimodal synchronization data structure, and the encoding result is encapsulated into an evidence file with a timestamp index and stored.

[0044] In this specific embodiment, under the law enforcement evidence collection mode, the three-dimensional spatial coordinates of the sound source are marked in the sound source imaging image. The position of the sound source in space is calculated by using the collected omnidirectional acoustic signals and sound source localization algorithms (such as MUSIC, SRP-PHAT, etc.). Specifically, a three-dimensional coordinate system is set, in which the center of the acoustic sensor array is taken as the origin, the front is the positive X-axis direction, the top is the positive Y-axis direction, and the right is the positive Z-axis direction. The three-dimensional spatial coordinates of the sound source are represented as (x, y, z), and the unit is meters.

[0045] When calculating the spatial orientation of the sound source relative to the visual image acquisition device, the relative positional relationship between the visual device and the acoustic sensor array is considered. Assuming the visual device is located 0.1 meters in front of the acoustic sensor array, a coordinate system transformation is required to obtain the azimuth angle θ and elevation angle φ of the sound source relative to the visual device. The azimuth angle represents the angle of the sound source relative to the visual device in the horizontal plane, and the elevation angle represents the elevation or depression angle of the sound source relative to the horizontal plane.

[0046] When determining whether a sound source is within the field of view of the visual image data, the calculated azimuth angle θ is compared with the horizontal field of view α of the visual device, and the pitch angle φ is compared with the vertical field of view β. When |θ| When the sound source is within the field of view, a visual indicator mark is superimposed on the visual image to convert the three-dimensional spatial coordinates into two-dimensional image coordinates (u, v). The conversion formula involves the camera's intrinsic and extrinsic parameters, and a conspicuous mark is drawn at that location, such as a red circle, triangle, or a label with distance information. The size of the mark can be dynamically adjusted according to the distance of the sound source; the mark is smaller when the distance is far and larger when the distance is close, enhancing intuitiveness.

[0047] Time alignment is crucial for ensuring synchronization between visual images and acoustic signals. Frame-level timestamps of visual image data and sample-level timestamps of omnidirectional acoustic signals are extracted. Visual images are timed in frames, such as 30 frames per second, with timestamp accuracy in the millisecond range. Omnidirectional acoustic signals are timed in sampling points, such as a sampling rate of 48 kHz, with timestamp accuracy in the microsecond range.

[0048] When calculating the time alignment window, the window width is adaptively determined based on the frame rate of the visual image and the sampling rate of the omnidirectional acoustic signal. Let the visual image frame rate be fr frames / second and the omnidirectional acoustic signal sampling rate be fs samples / second. Then, the number of acoustic signal samples corresponding to one frame is fs / fr. For example, if the image frame rate is 30fps and the acoustic signal sampling rate is 48kHz, then each frame corresponds to 1600 acoustic signal samples. The time alignment window width is set to the duration of two frames to ensure sufficient overlap for accurate synchronization.

[0049] Time synchronization is performed within the time alignment window. A mapping relationship between the frame level and the sampling level is established using timestamp interpolation. For each frame image, the acoustic signal sample closest to its timestamp is found, and the precise corresponding acoustic signal position is calculated through linear interpolation. This mapping relationship allows for precise positioning of the image and sound at any moment during playback.

[0050] A multimodal synchronization data structure was constructed, comprising aligned visual image data, omnidirectional acoustic signals, sound source imaging maps, and time mapping relationships. This data structure adopts a hierarchical design: the top layer is the time index, the middle layer contains the data of each modality, and the bottom layer contains the specific data content. Each time node contains the corresponding image frame, acoustic signal segment, and sound source location information.

[0051] Layered coding is performed on the multimodal synchronous data structure. Visual images adopt video coding standards such as H.264 / H.265, omnidirectional acoustic signals adopt lossless or high-quality lossy audio coding (such as FLAC, AAC), sound source imaging maps adopt efficient image coding (such as JPEG2000), time mapping relationships are stored in a compact binary format, the coding results of each layer are merged into a single data stream, and checksums and digital signatures are added to ensure data integrity.

[0052] The encoded results are encapsulated into a timestamped indexed forensic file and stored. The file format includes a header (recording device information, acquisition parameters, time reference, etc.) and a data area (storing encoded multimodal data). The timestamped index table is located after the header, supporting quick location of data at any time point. The forensic file is encrypted to prevent unauthorized access and supports breakpoint resumption and segmented storage, adapting to long-term forensic scenarios. This technical solution can be applied to scenarios such as criminal investigation, public security monitoring, and court evidence collection, providing both acoustic and visual evidence to enhance the accuracy and credibility of the evidence.

[0053] In one optional implementation, the device status signal is subjected to spectral transformation to obtain frequency domain feature data, a multi-type fault feature library is constructed, and deep feature extraction and classification verification are performed on the frequency domain feature data through a convolutional neural network to output fault type identifiers and confidence levels, including: The device status signal is subjected to adaptive window length spectrum transformation to extract frequency domain peak frequency features, sideband modulation features and frequency offset features. The correlation strength between each feature and the fault type is calculated through mutual information analysis. Features with correlation strength exceeding the correlation threshold are selected to form frequency domain feature data. Establish a hierarchical fault feature library architecture that supports dynamic expansion. Store the frequency domain feature data of known fault types in a hierarchical index according to equipment type and fault severity. Reserve the feature space interface for adding new fault types to form a multi-type fault feature library that can be incrementally updated. The frequency domain feature data is input into a convolutional neural network. The receptive field range is adaptively adjusted by a deformable convolutional kernel to extract local and global features. After passing through a feature fusion layer and a classification decision layer, a preliminary classification result is output. The preliminary classification result is matched and verified with the historical diagnostic cases in the multi-type fault feature library. When the matching similarity is higher than the verification threshold, the preliminary classification result is confirmed as the fault type identifier. The final confidence level is calculated by weighted fusion based on the matching similarity and the probability value output by the convolutional neural network.

[0054] In this specific embodiment, the acquired device status signal needs to undergo adaptive window-length spectral transformation. For high-speed rotating equipment such as motors, a shorter time window is selected, typically 50 to 100 milliseconds; for low-speed equipment such as large pumps, a window of up to 2 seconds is selected. Fast Fourier Transform is used for spectral analysis, and the Hanning window function is used to reduce spectral leakage. Three key features are extracted from the spectrum: peak frequency features, including the frequency position, amplitude, and bandwidth of each spectral peak; sideband modulation features, recording the distribution pattern and intensity of sidebands around the fundamental frequency to reflect the modulation effect; and frequency offset features, calculating the deviation ratio between the actual operating frequency and the nominal frequency.

[0055] Mutual information analysis is used to screen effective features and improve monitoring accuracy. Mutual information values ​​are calculated between the extracted features and known fault types to quantify the correlation strength between them. Mutual information calculation is achieved through sample frequency statistics to avoid making too many assumptions about feature distribution. Taking rolling bearings as an example, when there are microcracks in the inner ring of the bearing, the mutual information value of features related to the characteristic frequency of the inner ring and its harmonics is significantly higher than that of other features. A correlation threshold of 0.6 is set, and features with mutual information values ​​exceeding this threshold are retained to form frequency domain feature data. The correlation threshold can be dynamically adjusted according to specific application scenarios. It can be appropriately reduced to 0.5 in high-noise environments and increased to 0.7 in precise monitoring scenarios. The frequency domain feature data is stored in a standardized format, including fields such as feature type, frequency position, intensity value, and correlation score, which facilitates subsequent processing and analysis.

[0056] Establishing a dynamically expandable hierarchical fault feature library architecture is crucial for long-term monitoring. This feature library is organized in a tree structure, with the root node representing the equipment type, the second layer representing the fault location, and the third layer representing the fault severity. Each leaf node stores a set of frequency domain feature data for the corresponding fault type, containing multiple instance samples. The feature library is designed to support dynamic expansion; new fault types or equipment types can be added at any time through predefined feature interface specifications. The interface specifications define the format of the feature data, necessary metadata, and indexing methods to ensure compatibility. The feature library employs a distributed storage architecture, with core features stored locally on portable devices and the complete feature library stored on a backend server. Portable devices can incrementally synchronize as needed. In practical applications, such as when law enforcement officers inspect equipment in suspicious factories, the local feature library is sufficient to support common fault identification. In complex situations, remote synchronization can be triggered to obtain the complete feature library.

[0057] The feature library is updated and maintained in three ways: expert review and inclusion, where typical fault cases verified by equipment diagnostic experts are directly added to the feature library; semi-supervised learning incremental addition, where samples that cannot be clearly classified but have typical features are recommended by the algorithm and then confirmed by a lightweight human review process before being added to the library; and online self-learning, where the system automatically identifies high-confidence diagnostic results and periodically integrates them into feature library update packages. Each entry in the feature library contains fields such as equipment information, operating parameters, fault characteristics, diagnostic results, and reliability scores, supporting multi-dimensional retrieval. In edge computing scenarios, portable devices can use a lightweight version of the feature library, containing only key features of high-frequency faults, occupying no more than 50 megabytes of storage space.

[0058] Frequency domain feature data is input into a convolutional neural network (CNN). The CNN employs a multi-branch structure and deformable convolutional kernels, with the receptive field adaptively adjustable between 3×3 and 7×7. The deformable convolutional kernels are implemented by learning an offset field, dynamically adjusting the convolution operation region based on the distribution characteristics of different frequency domain features. In the network structure, the local feature extraction branch uses a smaller convolutional kernel (3×3) to capture fine-grained fault features; the global feature extraction branch uses a larger convolutional kernel (5×5) and dilated convolutions to capture macroscopic feature patterns. These features are merged through a feature fusion layer, and an attention mechanism is used to assign different weights to different feature channels. After passing through a fully connected layer, a preliminary classification result is output, including the probability distribution of fault types.

[0059] To improve diagnostic reliability, the preliminary classification results are compared with historical diagnostic cases in a multi-type fault feature library for similarity matching verification. For the preliminary classification results output by the convolutional neural network, historical cases of the same fault type are retrieved from the feature library. Similarity matching uses weighted Euclidean distance calculation, focusing on feature dimensions highly correlated with the fault to determine the search scope. A set of cases with the same fault type as the preliminary classification is selected from the feature library. The similarity between the current feature and each sample in the case set is calculated, and the top five cases with the highest similarity are selected. The average similarity is calculated as the matching score. When the matching similarity is higher than the verification threshold (usually set to 0.75), the preliminary classification result is confirmed as the fault type identifier. If the similarity is lower than the threshold, a secondary analysis process is triggered, using an alternative classifier or requesting manual intervention.

[0060] The matching and verification process also considers the influence of operating conditions. To address the differences in the performance of the same fault under different operating conditions, an adaptive factor for operating conditions is introduced. When there are differences between the detected operating conditions and the operating conditions of historical cases, the similarity calculation is corrected. For example, when monitoring bearing faults under variable speed conditions, the characteristic frequency position is adjusted according to the speed ratio before matching. When inspecting industrial equipment at the law enforcement site, this technology can effectively eliminate false alarms caused by fluctuations in operating conditions and improve the targeting of law enforcement.

[0061] The final confidence level is calculated by weighting and fusing the matching similarity and the probability value output by the convolutional neural network. The fusion uses a weighted average method, with the neural network probability value having a weight of 0.6 and the matching similarity value having a weight of 0.4. The calculated final confidence level is used to assess the reliability of the diagnostic results and determine subsequent processing strategies. The confidence level is divided into three levels: high confidence (greater than 0.85), directly confirming the diagnostic result and recording it; medium confidence (0.7 to 0.85), triggering auxiliary verification mechanisms, such as temporal analysis or envelope analysis; and low confidence (less than 0.7), marked as "suspected fault" and recommending further testing.

[0062] In portable multimodal law enforcement evidence collection applications, equipment status monitoring results are integrated with on-site evidence collection data. When an abnormal equipment status is detected, a high-definition camera module is automatically triggered to photograph the equipment and collect evidence, while simultaneously recording acoustic and vibration signals, generating an evidence package containing multimodal data. The evidence package is timestamped and includes fault diagnosis results, confidence scores, key feature values, and corresponding audio and video evidence, supporting subsequent in-depth analysis and law enforcement corroboration.

[0063] In one optional implementation, the frequency domain feature data is input into a convolutional neural network. The receptive field is adaptively adjusted using deformable convolutional kernels to extract local and global features. After passing through a feature fusion layer and a classification decision layer, a preliminary classification result is output, including: The frequency domain feature data is input into the convolutional neural network, and initial receptive field parameters of different scales are set in each branch. The receptive field range of the deformable convolutional kernel of each branch is adjusted by offset learning. High-frequency detail features in the frequency domain feature data are extracted as local features by deformable convolution kernels with receptive fields of different scales, and low-frequency envelope features in the frequency domain feature data are extracted as global features. A cross-scale feature alignment mechanism is established in the feature fusion layer. The local and global features extracted from different branches are aligned and transformed in terms of spatial and channel dimensions. The fusion coefficients are dynamically calculated through a learnable fusion weight network to generate multi-scale fusion features. The multi-scale fused features are input into the classification decision layer, and feature dimensionality reduction and nonlinear mapping are performed through cascaded fully connected layers to output the classification scores of each fault type. The classification scores are then activated by a function that adjusts the temperature parameter to obtain the calibrated fault type probability distribution as the preliminary classification result.

[0064] In this specific embodiment, during the input stage of the network, the frequency domain feature data is preprocessed, including normalization and data augmentation. The normalization process adjusts the feature value range to the [-1, 1] interval to improve the training stability of the model. The data augmentation techniques include random frequency masking and spectral perturbation to enhance the robustness of the model to noise.

[0065] The network employs a three-branch parallel structure, with convolutional kernels having initial receptive field parameters of 3×3, 5×5, and 7×7, respectively. For each branch, a deformable convolution mechanism is introduced to adaptively adjust the receptive field. The deformable convolution adds a learnable two-dimensional offset to the sampling positions of the regular convolution through an additional offset learning module. Specifically, each branch extracts an initial feature map through a standard convolutional layer and feeds it into the offset learning module. This module consists of two convolutional layers and outputs an offset field matching the number of sampling points of the convolutional kernel. These offsets are applied to subsequent convolutional operations, enabling the convolutional kernel to adaptively adjust its sampling position based on the input features.

[0066] In the feature extraction stage, the small-scale receptive field branch (3×3) mainly focuses on high-frequency details in the frequency domain features to capture local changes in fault features; the medium-scale receptive field branch (5×5) balances the extraction of local and global features; and the large-scale receptive field branch (7×7) focuses more on the low-frequency envelope features of the spectrum to extract global features that reflect the overall state of the equipment. Through deformable convolution, the receptive field of the small-scale branch can adaptively expand in the high-frequency region to improve the ability to capture local abrupt changes; while the receptive field of the large-scale branch can focus more accurately in the low-frequency region to enhance the ability to represent global trends.

[0067] After the features extracted from each branch are processed by multiple convolutional layers, they enter the feature fusion layer for integration. The feature fusion layer solves the problem of inconsistent size of output features from different branches and achieves spatial dimension unification through a cross-scale feature alignment mechanism. Specifically, a bilinear interpolation algorithm is used to resample feature maps of different resolutions to a uniform size to ensure the consistency of spatial correspondence. In terms of channel dimension, a 1×1 convolutional layer is used to adjust the number of feature channels of each branch to the same value to achieve channel dimension alignment.

[0068] To achieve dynamic feature fusion, a learnable fusion weight network was designed. This network receives aligned features from each branch and calculates the fusion weight coefficients for each position and channel through an attention mechanism. Specifically, it employs a dual attention mechanism combining channel attention and spatial attention. Channel attention is implemented through global average pooling followed by a fully connected layer, while spatial attention is implemented through depthwise separable convolutions. The outputs of both attention mechanisms are then subjected to element-wise multiplication and addition operations to generate the final fusion weight matrix. The fusion weights are applied to the features of each branch, and multi-scale fusion features are generated through weighted summation.

[0069] Multi-scale fused features are input into the classification decision layer for fault type identification. The classification decision layer consists of three fully connected layers with a batch normalization layer and a ReLU activation function inserted in the middle to achieve non-linear mapping and dimensionality reduction of features. The first fully connected layer reduces the dimension of the fused features to 512, the second layer further reduces it to 128, and the last layer outputs a classification score vector equal to the number of fault types.

[0070] To improve classification reliability, a softmax activation process with temperature parameter adjustment is applied to the classification scores. Specifically, an adjustable temperature parameter T is introduced, and the original classification scores are scaled before applying the softmax function. When T > 1, the output probability distribution is smoother, mitigating model overconfidence; when T < 1, the confidence of high-probability classes is enhanced. The temperature parameter is determined through validation set optimization, typically set between 0.8 and 1.2 to improve the calibration of the model's predicted probabilities. Through this process, a calibrated fault type probability distribution is finally output as the preliminary result of fault classification.

[0071] This invention provides a portable, multimodal integrated system for law enforcement evidence collection and equipment status monitoring, comprising: The first unit is used to acquire omnidirectional acoustic signals collected by an array of acoustic sensors arranged in a ring, as well as visual image data collected by an optical imaging module. The second unit is used to perform adaptive beamforming processing on the omnidirectional acoustic signal, calculate the arrival time difference between each acoustic sensor, obtain the three-dimensional spatial coordinates of the sound source by solving the time delay equations, and generate a sound source imaging map based on the sound field intensity data corresponding to the three-dimensional spatial coordinates. The third unit is used to mark the three-dimensional spatial coordinates of the sound source in the sound source imaging map when in law enforcement evidence collection mode, and to perform time alignment and encoding storage by combining the visual image data with the omnidirectional acoustic signal; when in equipment detection mode, it activates the ultrasonic frequency band sensor and vibration sensor to collect the equipment status signal of the equipment to be detected. The fourth unit is used to perform spectral transformation on the device status signal to obtain frequency domain feature data, construct a multi-type fault feature library, and perform deep feature extraction and classification verification on the frequency domain feature data through a convolutional neural network, and output fault type identifier and confidence level. The fifth unit is used to mark the fault type identifier of the device under test in the sound source imaging image, display it synchronously with the visual image data, determine whether to trigger an alarm based on the confidence level, generate a device fault detection record and store it.

[0072] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0073] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0074] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A portable, multimodal law enforcement evidence collection and equipment status monitoring integrated method, characterized in that, include: Acquire omnidirectional acoustic signals collected by an array of acoustic sensors arranged in a ring, and visual image data acquired by an optical imaging module; Adaptive beamforming is applied to the omnidirectional acoustic signal to calculate the arrival time difference between each acoustic sensor. The three-dimensional spatial coordinates of the sound source are obtained by solving the time delay equations. A sound source imaging map is generated based on the sound field intensity data corresponding to the three-dimensional spatial coordinates. When in law enforcement evidence collection mode, the three-dimensional spatial coordinates of the sound source are marked in the sound source imaging image. The visual image data and the omnidirectional acoustic signal are time-aligned and encoded for storage, including: The three-dimensional spatial coordinates of the sound source are marked in the sound source imaging image. The spatial orientation of the sound source relative to the visual image acquisition device is calculated. It is determined whether the sound source is within the field of view of the visual image data. When the sound source is within the field of view, a visual indicator mark synchronized with the three-dimensional spatial coordinates is superimposed in the visual image data. Extract the frame-level timestamp of the visual image data and the sampling-level timestamp of the omnidirectional acoustic signal, calculate the time alignment window between the frame-level timestamp and the sampling-level timestamp, and adaptively determine the window width according to the frame rate of the visual image data and the sampling rate of the omnidirectional acoustic signal; Within the time alignment window, time synchronization processing is performed on the visual image data and the omnidirectional acoustic signal, and a time mapping relationship between the frame level and the sampling level is established by timestamp interpolation method. A multimodal synchronization data structure is constructed, which includes the aligned visual image data, the omnidirectional acoustic signal, the sound source imaging map, and the time mapping relationship; hierarchical encoding is performed on the multimodal synchronization data structure, and the encoding result is encapsulated into an evidence file with a timestamp index and stored. When in device detection mode, the ultrasonic frequency sensor and vibration sensor are activated to collect the device status signal of the device under test. The device status signal is subjected to spectral transformation to obtain frequency domain feature data. A multi-type fault feature library is constructed, and deep feature extraction and classification verification are performed on the frequency domain feature data through a convolutional neural network. The fault type identifier and confidence level are output, including: The device status signal is subjected to adaptive window length spectrum transformation to extract frequency domain peak frequency features, sideband modulation features and frequency offset features. The correlation strength between each feature and the fault type is calculated through mutual information analysis. Features with correlation strength exceeding the correlation threshold are selected to form frequency domain feature data. Establish a hierarchical fault feature library architecture that supports dynamic expansion. Store the frequency domain feature data of known fault types in a hierarchical index according to equipment type and fault severity. Reserve the feature space interface for adding new fault types to form a multi-type fault feature library that can be incrementally updated. The frequency domain feature data is input into a convolutional neural network. The receptive field range is adaptively adjusted by a deformable convolutional kernel to extract local and global features. After passing through a feature fusion layer and a classification decision layer, a preliminary classification result is output. The preliminary classification result is matched and verified with the historical diagnostic cases in the multi-type fault feature library. When the matching similarity is higher than the verification threshold, the preliminary classification result is confirmed as the fault type identifier. The final confidence level is calculated by weighted fusion based on the matching similarity and the probability value output by the convolutional neural network. The fault type identifier of the device under test is marked in the sound source imaging image and displayed synchronously with the visual image data. The confidence level is used to determine whether an alarm is triggered, and a device fault detection record is generated and stored.

2. The method according to claim 1, characterized in that, Adaptive beamforming is applied to the omnidirectional acoustic signal to calculate the arrival time difference between each acoustic sensor. The three-dimensional spatial coordinates of the sound source are obtained by solving a set of time delay equations. A sound source imaging map is generated based on the sound field intensity data corresponding to the three-dimensional spatial coordinates, including: The omnidirectional acoustic signal is transformed by time and frequency to obtain the energy distribution in the time and frequency domain and to identify sparse time and frequency points. The sparse time and frequency points are determined by calculating the phase difference variance between each acoustic sensor. The time and frequency points whose phase difference variance is less than the sparse threshold are identified as the sparse time and frequency points. For each of the sparse time-frequency points, the arrival time difference is calculated based on the phase difference between each acoustic sensor, a set of time delay geometric equations is constructed and solved to obtain the candidate sound source positions corresponding to the sparse time-frequency points. Calculate the Euclidean distance between the candidate sound source positions corresponding to the sparse time-frequency points, group the candidate sound source positions whose Euclidean distance is less than the cluster radius into the same cluster, calculate the centroid coordinates of each cluster as the three-dimensional spatial coordinates of each sound source, and construct a guiding vector pointing to the three-dimensional spatial coordinates. The adaptive weighting coefficients of each acoustic sensor are calculated based on the guidance vector, and the omnidirectional acoustic signals are weighted and superimposed to obtain the beamforming output signals of each three-dimensional spatial coordinate. The cumulative energy values ​​of the sparse time-frequency points associated with each three-dimensional spatial coordinate in the time-frequency domain are statistically analyzed and used as the sound field intensity data corresponding to the three-dimensional spatial coordinates. The three-dimensional spatial coordinates of each sound source are then mapped to the spatial display coordinate system to generate the sound source imaging map containing the location and intensity distribution of multiple sound sources.

3. The method according to claim 2, characterized in that, For each of the sparse time-frequency points, the arrival time difference is calculated based on the phase difference between each acoustic sensor, a set of time-delay geometric equations is constructed and solved to obtain the candidate sound source locations corresponding to the sparse time-frequency points, including: For each of the sparse time-frequency points, the complex representation of each acoustic sensor in the time-frequency domain is extracted and phase subtraction is performed to obtain the phase difference between each pair of acoustic sensors; The phase difference is unwound, the periodic ambiguity is eliminated by constraining the phase difference continuity of adjacent acoustic sensor pairs, the phase difference that jumps beyond the circumferential phase is compensated for with integer periods, and the arrival time difference is calculated based on the unwound phase difference and angular frequency. Based on the spatial coordinates of each acoustic sensor and the arrival time difference, a set of time delay geometric equations is constructed. Each time delay geometric equation expresses the relationship between the propagation distance difference from the sound source to different acoustic sensor pairs and the product of the arrival time difference and the speed of sound. Hyperbolic surface intersection analysis is performed on the set of time delay geometric equations to determine the intersection region as the sound source search space, with the geometric center of the search space as the initial value. The nonlinear optimization method minimizes the sum of squared residuals, calculates the residual vector between the theoretical arrival time difference and the actual arrival time difference corresponding to the current estimated value, and determines whether the norm of the residual vector is less than the convergence threshold. If it is not satisfied, the estimated value is corrected according to the gradient direction and iterated. If it is satisfied, the current estimated value is output as the candidate location of the sound source corresponding to the sparse time-frequency point.

4. The method according to claim 1, characterized in that, The frequency domain feature data is input into a convolutional neural network. The receptive field is adaptively adjusted using deformable convolutional kernels to extract local and global features. After passing through a feature fusion layer and a classification decision layer, a preliminary classification result is output, including: The frequency domain feature data is input into the convolutional neural network, and initial receptive field parameters of different scales are set in each branch. The receptive field range of the deformable convolutional kernel of each branch is adjusted by offset learning. High-frequency detail features in the frequency domain feature data are extracted as local features by deformable convolution kernels with receptive fields of different scales, and low-frequency envelope features in the frequency domain feature data are extracted as global features. A cross-scale feature alignment mechanism is established in the feature fusion layer. The local and global features extracted from different branches are aligned and transformed in terms of spatial and channel dimensions. The fusion coefficients are dynamically calculated through a learnable fusion weight network to generate multi-scale fusion features. The multi-scale fused features are input into the classification decision layer, and feature dimensionality reduction and nonlinear mapping are performed through cascaded fully connected layers to output the classification scores of each fault type. The classification scores are then activated by a function that adjusts the temperature parameter to obtain the calibrated fault type probability distribution as the preliminary classification result.

5. A portable multimodal law enforcement evidence collection and equipment status monitoring integrated system, used to implement the method as described in any one of claims 1-4, characterized in that, include: The first unit is used to acquire omnidirectional acoustic signals collected by an array of acoustic sensors arranged in a ring, as well as visual image data collected by an optical imaging module. The second unit is used to perform adaptive beamforming processing on the omnidirectional acoustic signal, calculate the arrival time difference between each acoustic sensor, obtain the three-dimensional spatial coordinates of the sound source by solving the time delay equations, and generate a sound source imaging map based on the sound field intensity data corresponding to the three-dimensional spatial coordinates. The third unit is used to mark the three-dimensional spatial coordinates of the sound source in the sound source imaging map when in law enforcement evidence collection mode, and to perform time alignment and encoding storage by combining the visual image data with the omnidirectional acoustic signal; when in equipment detection mode, it activates the ultrasonic frequency band sensor and vibration sensor to collect the equipment status signal of the equipment to be detected. The fourth unit is used to perform spectral transformation on the device status signal to obtain frequency domain feature data, construct a multi-type fault feature library, and perform deep feature extraction and classification verification on the frequency domain feature data through a convolutional neural network, and output fault type identifier and confidence level. The fifth unit is used to mark the fault type identifier of the device under test in the sound source imaging image, display it synchronously with the visual image data, determine whether to trigger an alarm based on the confidence level, generate a device fault detection record and store it.

6. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 4.

Citation Information

Patent Citations

  • A method and system for timestamp marking of live streams

    CN112188286B

  • Acoustic phase instrument, implementation method thereof and electronic equipment

    CN118091542A

  • System and method for monitoring machine health

    CN118550269A