Ultra-precision laser radar and acoustic image integrated monitoring system and method for industrial environment

Abstract

The invention relates to an ultra-precision laser radar and acoustic image integrated monitoring system and method for an industrial environment, and the system integrates a laser radar and an acoustic imager, and carries out the data collection and processing at the same time through the steps: parameter setting, data collection, information preliminary processing, data fusion and comparison, and visual output and storage. The industrial environment can be monitored from multiple dimensions, the limitation of a single device is made up, the overall monitoring precision is improved, data collected by two devices are sequentially subjected to data layer fusion and preliminary comparison, feature layer fusion and secondary comparison and decision layer fusion and final comparison, and therefore the overall monitoring precision is improved. Noise and abnormal values are effectively removed, more valuable features are extracted, the accuracy of monitoring results is further improved, abnormal conditions in the industrial environment can be found in time, guarantee is provided for production safety and quality control, and losses caused by the fact that the abnormal conditions are not found in time are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial environment monitoring, and in particular to an ultra-precision laser radar and audio-visual integrated monitoring system and method for industrial environments. Background Art

[0002] Since the industrial production environment is complex and changeable, any slight change may have a significant impact on the production process and product quality. Therefore, in the industrial production process, real-time and accurate monitoring of the environment is crucial; on the one hand, real-time and accurate monitoring can ensure the safe and stable operation of the production process; on the other hand, accurate monitoring helps to improve product quality. In traditional monitoring methods, the accuracy of many sensors is limited by many factors. For example, some contact sensors may interfere with the measured object during the measurement process, thereby affecting the accuracy of the measurement results. Moreover, traditional methods may only provide limited measurement range and resolution, which is difficult to meet the needs of modern industry for high-precision monitoring. The emergence of LiDAR and audio-visual cameras has brought new opportunities for industrial monitoring. With the continuous advancement of technology, the performance of LiDAR and audio-visual cameras has been continuously improved, the cost has been gradually reduced, and their application scope in industrial monitoring has become more and more extensive. However, there are still problems of one kind or another. For example, patent number CN116699553A discloses a 3D scanning laser radar method, which uses laser radar to collect object data information. However, laser radar cannot obtain visual features such as color, texture, surface defects and sound features of objects, and in some specific industrial environments full of dust, smoke or strong light, or for some transparent or translucent objects, the laser radar signal may be weakened or interfered, making it difficult to accurately measure the position and shape of the object, resulting in reduced measurement accuracy or failure to work properly. For example, the patent number sound imager may not be able to provide sufficiently accurate spatial information for industrial scenes that require accurate measurement of the spatial position and size of objects, such as in mechanical processing, where the position and size deviation of parts need to be accurately determined. In addition, the image quality and sound recognition effect of the sound imager will be disturbed when the light is insufficient or the object is blocked, or in a noisy industrial workshop, affecting the identification and analysis of the object. Therefore, a technology integrating industrial laser radar and sound imager is developed to give full play to the advantages of both and realize comprehensive perception and accurate analysis of industrial scenes. Summary of the invention

[0003] 1. Technical issues to be resolved

[0004] In view of the deficiencies in the prior art, the present invention aims to provide an ultra-precision laser radar and audio-visual integrated monitoring system and method for industrial environments, which integrates laser radar and audio-visual instrument and simultaneously collects and processes data through parameter setting, data collection, preliminary information processing, data fusion and comparison, and visual output and storage. The industrial environment can be monitored from multiple dimensions to make up for the limitations of a single device, thereby improving the overall monitoring accuracy. By fusing and comparing the data collected by the two devices in turn, fusing and comparing the data layer, fusing and comparing the feature layer, and fusing and comparing the decision layer, the noise and outliers can be effectively removed, more valuable features can be extracted, the accuracy of the monitoring results can be further improved, and abnormal conditions in the industrial environment can be discovered in a timely manner, providing protection for production safety and quality control, and reducing losses caused by abnormal conditions not being discovered in a timely manner.

[0005] (II) Technical solution

[0006] To achieve the above object, the present invention provides the following technical solution: an ultra-precision laser radar and audio-visual integrated monitoring method for industrial environments, comprising the following steps:

[0007] Parameter setting: After receiving the start-up command, the central control module checks the status of the laser radar and the audio-visual camera. At the same time, the user sets the detection parameters of the laser radar and the audio-visual camera on the control interface. First, check the status of the equipment. The inspection of the laser radar includes but is not limited to checking whether its power supply is normal, whether the sensor is in good working condition, and whether the communication connection is stable. The status inspection of the audio-visual camera includes whether the image sensor is working normally, whether there is noise or failure in the audio acquisition part, and whether the storage capacity of the device is sufficient.

[0008] Data collection: The central control module triggers the laser radar and the imager to start data collection at the same time. The laser radar measures the distance between the target object and the laser radar by emitting a laser beam and receiving the reflected light signal. At the same time, the position of the target object in space can be determined according to the scanning angle of the laser beam. The laser radar can perform continuous scanning according to the preset parameters and continuously update the position information of the target object. During the data collection process, the laser radar records the coordinates, reflection intensity and other information of each measurement point. The imager collects image and audio data at the same time. The image part captures the scene in the industrial environment through the image sensor and records the appearance, color, shape and other information of the object. The audio collection part receives the sound signals in the environment through the microphone, including the sound of the equipment running, possible abnormal noise, etc.

[0009] Initial information processing: The signals collected by the laser radar and the audio-visual camera will be amplified by their respective internal signal processors. At the same time, the image information collected by the audio-visual camera will be initially processed by the image processor. Subsequently, the amplified and initially processed signals will enter the analog-to-digital converter ADC to be converted into digital signals, and finally transmitted to the central control module through the high-speed data bus. Since the light signal reflected back to the laser radar by the object is usually very weak, it needs to be amplified for better subsequent processing.

[0010] Data fusion and comparison: After receiving data from the lidar and audio-visual camera, the central control module will first store the received data, then perform data-layer fusion on the data, and make a preliminary comparison between the fused data and the preset data to determine whether there are any abnormalities. Then, perform feature-layer fusion on the data, and make a secondary comparison between the fused data and the preset data to determine whether there are any abnormalities. Finally, perform decision-layer fusion on the data, and make a final comparison between the fused data and the preset data to determine whether there are any abnormalities.

[0011] It should be noted that after the lidar and audio-visual camera collect data and transmit it to the central control module, the central control module will first store the received data, that is, store the data in a temporary database first.

[0012] Visual output and storage: The processed data is stored in a suitable database system through the communication module and visualized through the output module.

[0013] To further explain, the data storage here is different from the data storage before data fusion, which mainly involves the reasonable storage of the fused data. The visual output is visualized through the output module, presenting the fused data of the lidar and the audio-visual camera to the user in an intuitive way.

[0014] Preferably, in the step of data fusion and comparison, data layer fusion includes:

[0015] Time synchronization: Add a timestamp to each data point when collecting data, and then match and synchronize the data of the two devices based on the timestamp.

[0016] Spatial alignment: Project the lidar point cloud data onto the image plane of the imager through coordinate transformation and calibration.

[0017] Weighted average method to fuse data: Assign weights to the lidar and imager data based on the performance analysis results and fuse all corresponding data points to obtain fused point cloud data and imager image data.

[0018] Furthermore, time synchronization is performed by aligning the data through data difference synchronization. That is, if the lidar collects a point cloud data at a certain moment, and the audio-visual camera collects an image and an audio data at a similar time, by comparing the timestamps, it can be determined that the two sets of data were collected in the same time window, and further fusion processing can be performed.

[0019] Preferably, in the step data fusion and comparison, feature layer fusion includes:

[0020] Data normalization: After further inspection and cleaning of the data layer fusion, the data is normalized so that the data with different features have the same scale and range.

[0021] Feature extraction: For lidar data and camera data, further extract higher-level and discriminative features.

[0022] Feature fusion: A fusion algorithm combining principal component analysis (PCA) and independent component analysis (ICA) is used to select important components as fused features.

[0023] Furthermore, data normalization is performed because after data layer fusion, the data from the lidar and the imager may have different scales and ranges.

[0024] Preferably, in the step data fusion and comparison, the decision layer fusion includes:

[0025] Generate independent decisions: Use clustering algorithms to identify and classify objects to obtain the decision results of the lidar; use image recognition algorithms and sound recognition algorithms to identify and classify objects respectively to obtain the decision results of the audio and video camera.

[0026] Decision fusion: The DS evidence theory fusion algorithm is used to fuse the decision results of the lidar and the audio-visual camera.

[0027] Preferably, in the step of data fusion and comparison, the preliminary comparison includes:

[0028] Range comparison: For the point cloud data of the LiDAR, determine the preset value of the spatial coordinate range; for the image data of the imager, set the range of pixel values.

[0029] Statistical feature comparison: Calculate the statistical features of the fused data and compare them with the preset normal statistical feature range.

[0030] Consistency comparison: Compare the consistency of the fused data of LiDAR and audio-visual camera in corresponding positions and times.

[0031] Furthermore, the statistical features may include mean, variance, standard deviation, median, mode, etc. For the fused LiDAR point cloud data and the image data of the acoustic imager, these statistical features may be calculated respectively.

[0032] Preferably, in the step of data fusion and comparison, the secondary comparison includes:

[0033] Feature distance comparison: Calculate the distance between the fused feature and the preset normal feature.

[0034] Feature pattern comparison: Establish a feature pattern library under normal conditions and compare the fused features with the feature patterns in the pattern library.

[0035] Preferably, in the step data fusion and comparison, the final comparison includes:

[0036] Comparison of decision-making result consistency: Compare the results of decision-making layer fusion with the preset normal decision results.

[0037] Abnormal probability comparison: According to the algorithm fused by the decision layer, the probability of abnormality in the current data is calculated and compared with the preset abnormal probability threshold.

[0038] Comprehensive evaluation and comparison: Combine the comparison results of data layer fusion and feature layer fusion to comprehensively evaluate the results of decision layer fusion. When conducting comprehensive evaluation and comparison, it is necessary to combine the comparison results of the three levels of data layer, feature layer and decision layer to comprehensively evaluate the results of decision layer fusion.

[0039] Furthermore, principal component analysis (PCA) includes calculating the covariance matrix of the data, performing eigenvalue decomposition on the covariance matrix, obtaining eigenvalues ​​and eigenvectors, selecting eigenvectors corresponding to the largest eigenvalues, forming a new feature space, and projecting the original data into the new feature space to obtain the reduced-dimensionality lidar and audio-visual data features.

[0040] Independent component analysis (ICA) involves merging the features of the lidar and audio-visual camera data after dimensionality reduction into one data set, performing ICA processing on the merged data set and obtaining independent components, and then selecting the important components as the fused features.

[0041] Preferably, an integrated system for performing an ultra-precision laser radar and audio-visual integrated monitoring method for an industrial environment comprises:

[0042] LiDAR equipment: It is used to emit laser beams and receive reflected signals to obtain three-dimensional spatial information of objects.

[0043] Audio-visual equipment: It consists of an image acquisition part and a sound acquisition part; the image acquisition part can obtain visual image information of an object, including color, texture, shape and other features; the sound acquisition part can capture sound signals in the environment to analyze the sound characteristics of an object or detect abnormal sounds.

[0044] Central control module: It is composed of a microprocessor and a memory unit. The microprocessor is responsible for receiving startup instructions, checking sensor status, triggering data acquisition, receiving and storing data from the lidar and audio-visual camera, and performing operations such as data fusion and comparison. The memory unit is used to store program code, intermediate data and final processing results.

[0045] Signal processing module: It is composed of a lidar signal processor, an audio-visual instrument signal processor and an image processor; the lidar signal processor is used to amplify, filter, digitize and perform other processing on the signals collected by the lidar, and the audio-visual instrument signal processor and image processor are used to perform preliminary processing on the image and sound signals collected by the audio-visual instrument.

[0046] Data fusion and comparison module: consists of a time synchronization unit, a space alignment unit, a data layer fusion unit, a feature layer fusion unit and a comparison unit; the time synchronization unit is used to add timestamps to the data points of the laser radar and the imager during data collection, and match and synchronize the data of the two devices according to the timestamps; the space alignment unit is used to project the point cloud data of the laser radar onto the image plane of the imager to achieve space alignment; the data layer fusion unit is used to preliminarily fuse the data of the laser radar and the imager; the feature layer fusion unit is used to extract and fuse features of the data after data layer fusion; the comparison unit is used to perform preliminary comparison after data layer fusion, secondary comparison after feature layer fusion, and final comparison after decision layer fusion.

[0047] Decision layer fusion module: It consists of an independent decision generation unit and a decision fusion unit; the independent decision generation unit is used to identify and classify the lidar data and the audio-visual device data to obtain their respective decision results, and the decision fusion unit is used to fuse the decision results of the lidar and the audio-visual device to obtain the final decision result.

[0048] Visual output and storage module: consists of a communication unit and an output unit; the communication unit is responsible for storing the processed data in a suitable database system, and the output unit is used for visual display so that users can intuitively understand the monitoring results.

[0049] (III) Beneficial effects

[0050] (1) The present invention integrates a laser radar and an audio-visual device, and simultaneously collects and processes data through steps such as parameter setting, data collection, preliminary information processing, data fusion and comparison, and visual output and storage. It can monitor the industrial environment from multiple dimensions, make up for the limitations of a single device, and improve the monitoring accuracy of the position, shape, appearance, and sound characteristics of objects in the industrial environment. At the same time, the data collected by the two devices are sequentially fused and compared at the data layer, fused and compared at the feature layer, and fused and compared at the decision layer. Such multi-layer data fusion and comparison can effectively remove noise and outliers, extract more valuable features, and further improve the accuracy of the monitoring results.

[0051] (2) The present invention simultaneously utilizes laser radar and audio-visual equipment to obtain three-dimensional spatial information, visual image information, and sound information of an object, thereby providing a more comprehensive source of information for industrial environmental monitoring and facilitating more accurate analysis of various situations in the industrial production process, such as equipment operating status, product quality issues, potential safety hazards, etc.

[0052] (3) The present invention sets up a multi-level comparison process, including range comparison, statistical feature comparison, consistency comparison, feature distance comparison, feature pattern comparison, decision result consistency comparison, abnormal probability comparison and comprehensive evaluation comparison, etc., which can timely discover abnormal situations in industrial environments, provide protection for production safety and quality control, and reduce losses caused by abnormal situations not being discovered in time. (IV) Description of the drawings

[0053] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0054] Figure 1 It is a schematic diagram of the equipment and module composition of an ultra-precision laser radar and integrated audio-visual monitoring system and method for industrial environments in an embodiment of the present application.

[0055] Figure 2 This is a flow chart of an ultra-precision laser radar and integrated audio-visual monitoring system and method for industrial environments in an embodiment of the present application.

[0056] Figure 3 This is a flow chart of data layer fusion in data fusion and comparison of an ultra-precision laser radar and integrated audio-visual monitoring system and method steps for industrial environments in an embodiment of the present application.

[0057] Figure 4This is a flowchart of feature layer fusion in data fusion and comparison of an ultra-precision laser radar and integrated audio and video monitoring system and method steps for industrial environments in an embodiment of the present application.

[0058] Figure 5 This is a flowchart of the decision-making layer fusion in data fusion and comparison of an ultra-precision laser radar and integrated audio-visual monitoring system and method steps for industrial environments in an embodiment of the present application. (V) Specific implementation methods

[0059] The following will be combined with the attached examples of the present invention Figure 1 -Attached Figure 5 A clear and complete description is given of the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0060] Embodiment 1: An ultra-precision laser radar and audio-visual integrated monitoring method for industrial environments, comprising the following steps:

[0061] Parameter setting: After receiving the start-up command, the central control module checks the status of the laser radar and the audio-visual camera. At the same time, the user sets the detection parameters of the laser radar and the audio-visual camera on the control interface. First, check the status of the equipment. The inspection of the laser radar includes but is not limited to checking whether its power supply is normal, whether the sensor is in good working condition, and whether the communication connection is stable. The status inspection of the audio-visual camera includes whether the image sensor is working normally, whether there is noise or failure in the audio acquisition part, and whether the storage capacity of the device is sufficient.

[0062] Users can set the detection parameters of the LiDAR on the control interface, such as scanning range, resolution, scanning frequency, etc. For example, if you are monitoring a large industrial area, you can set the scanning range of the LiDAR to be larger, and adjust the resolution and scanning frequency according to the accuracy requirements. For the audio and video camera, users can set parameters such as image resolution, frame rate, and audio acquisition sensitivity. For example, in scenarios where subtle changes need to be captured clearly, the image resolution and frame rate can be increased; and in environments that are sensitive to specific sound frequencies, the sensitivity of audio acquisition can be adjusted to better capture relevant sound signals.

[0063] Data collection: The central control module triggers the laser radar and the imager to start data collection at the same time. The laser radar measures the distance between the target object and the laser radar by emitting a laser beam and receiving the reflected light signal. At the same time, the position of the target object in space can be determined according to the scanning angle of the laser beam. The laser radar can perform continuous scanning according to the preset parameters and continuously update the position information of the target object. During the data collection process, the laser radar records the coordinates, reflection intensity and other information of each measurement point. The imager collects image and audio data at the same time. The image part captures the scene in the industrial environment through the image sensor and records the appearance, color, shape and other information of the object. The audio collection part receives the sound signals in the environment through the microphone, including the sound of the equipment running, possible abnormal noise, etc.

[0064] Preliminary information processing: The signals collected by the laser radar and the audio-visual instrument will be amplified by their respective internal signal processors. At the same time, the image information collected by the audio-visual instrument will be preliminarily processed by the image processor. Subsequently, the amplified and preliminarily processed signals will enter the analog-to-digital converter ADC to be converted into digital signals, and finally transmitted to the central control module through the high-speed data bus. Since the light signal reflected back to the laser radar by the object is usually very weak, it needs to be amplified for better subsequent processing. For example, the photocurrent signal is amplified to a suitable amplitude by using an amplifier circuit for analog-to-digital conversion. The audio signal may also become weak due to interference from environmental noise during the acquisition process, and the signal strength and signal-to-noise ratio can be improved by amplification, that is, the sound signal collected by the microphone is amplified by an audio amplifier for better subsequent audio processing. In addition, the image information collected by the audio-visual instrument will be preliminarily processed by the image processor, which includes operations such as removing image noise, enhancing image contrast and brightness, and removing random noise in the image by using a filter to improve image clarity; by adjusting contrast and brightness, objects in the image are made clearer and more discernible.

[0065] Subsequently, the amplified and preliminarily processed signal enters the analog-to-digital converter (ADC) to be converted into a digital signal. For the lidar, this step is to convert the analog light signal into a digital signal for digital signal processing and transmission, that is, to convert the photocurrent signal into a digital signal through the analog-to-digital converter (ADC), and each digital value represents a specific light intensity or distance measurement value. For the audio-visual instrument, the analog-to-digital conversion of the audio signal is to convert the analog sound waveform into digital audio data, and the analog-to-digital conversion of the image signal is to convert the analog image brightness and color information into digital image data.

[0066] After completing the digital signal conversion, the data of the lidar and the audio-visual camera are transmitted to the central control module through a high-speed data bus, ensuring fast data transmission and reducing data transmission delays.

[0067] Data fusion and comparison: After receiving data from the laser radar and the sound imager, the central control module will first store the received data, then perform data layer fusion on the data, and make a preliminary comparison between the fused data and the preset data to determine whether there is an abnormality, then perform feature layer fusion on the data, and make a secondary comparison between the fused data and the preset data to determine whether there is an abnormality, and finally perform decision layer fusion on the data, and make a final comparison between the fused data and the preset data to determine whether there is an abnormality. After the laser radar and the sound imager collect the data and transmit it to the central control module, the central control module will first store the received data, that is, store the data in a temporary database first. Specifically, a lightweight database SQLite based on the file system suitable for use in an industrial environment is established, that is, a small database can be created on the local file system to facilitate data storage and management. In the database SQLite, data can be classified and stored according to information such as data type and timestamp: that is, the laser radar data and the sound imager data are stored in different files in chronological order, and the metadata of the data (such as timestamp, device ID, data type, etc.) is also stored in the database SQLite for subsequent retrieval and analysis. It should be noted that after the temporary storage is completed, the data in the memory buffer is also periodically written to the long-term storage device solid-state hard disk. The solid-state hard disk may adopt a timed writing strategy to write the data in the memory buffer to the solid-state hard disk at regular intervals;

[0068] When writing data, you can use RAID technology to combine multiple hard disks into a disk array to achieve parallel writing and redundant storage of data. In addition, in order to facilitate subsequent data retrieval and analysis, you need to create a data index, that is, create an index field in the data table, store data collected at different times in different directories, and name each file with timestamps and device information, so that you can quickly locate and retrieve data.

[0069] Visual output and storage: The processed data is stored in a suitable database system through the communication module, and visualized through the output module. The data storage here is different from the data storage before data fusion. It mainly stores the fused data reasonably, establishes relational databases, non-relational databases and distributed file systems, and selects according to the characteristics of the data and storage requirements. For a large amount of historical data, a distributed file system can be used for storage for long-term data preservation and analysis; for real-time monitoring data, an in-memory database or cache can be used to increase the data reading and writing speed.

[0070] The visual output here is visualized through the output module, presenting the fused data of the lidar and the imager to the user in an intuitive way. That is, the point cloud data of the lidar and the image data of the imager are fused together using 3D visualization software to display the 3D scene of the industrial environment. In the image, different colors and marks can be used to represent different types of objects and abnormal situations, allowing users to quickly understand the status of the industrial environment.

[0071] Visual display also includes charts, reports and other forms, which present the statistical characteristics and comparative results of data in an intuitive way.

[0072] In the step data fusion and comparison, data layer fusion includes:

[0073] Time synchronization: Add a timestamp to each data point when collecting data, and then match and synchronize the data of the two devices based on the timestamp.

[0074] Spatial alignment: Project the lidar point cloud data onto the image plane of the imager through coordinate transformation and calibration.

[0075] Weighted average method to fuse data: Assign weights to the lidar and imager data based on the performance analysis results and fuse all corresponding data points to obtain fused point cloud data and imager image data.

[0076] Time synchronization is to align by data difference synchronization. The first step is to add a timestamp. When the lidar and the imager start to collect data, the system will add an accurate timestamp to each collected data point. For the lidar, while emitting the laser beam and receiving the reflected signal, the timestamp of each measurement point is recorded, so as to accurately know the collection time of each point cloud data. For the imager, at the moment of taking the image and recording the audio, the corresponding timestamp is also added to the image and audio data. With the timestamp, the system can match and synchronize the data of the two devices according to the timestamp. First, sort the data of the lidar and the imager and arrange them in the order of the timestamps. Then, by comparing the difference of the timestamps, find the lidar data points and the imager data points that are closest in time for matching.

[0077] That is to say, if the laser radar collects a point cloud data at a certain moment, and the audio-visual camera collects an image and an audio data at a similar time, by comparing the timestamps, it can be determined that the two sets of data were collected in the same time window, and further fusion processing can be performed. In the matching process, the timestamps may not be completely consistent. In this case, the data interpolation algorithm is used for time synchronization:

[0078] Assume that the data acquisition rate of the lidar is f1, the data acquisition rate of the imager is f2, and f1>f2. Interpolation can be performed between the data points of the imager so that the data points of the imager correspond one-to-one with the data points of the lidar in time. Specifically, first, determine the data acquisition time interval of the lidar and the imager. Assume that the time interval of the lidar is Δt1, and the time interval of the imager is Δt2; then, for each data point of the imager, calculate the corresponding time point on the lidar time axis according to its timestamp and acquisition time interval; finally, use the interpolation algorithm to interpolate between the data points of the imager to obtain the imager data value at the corresponding time point on the lidar time axis. The interpolation algorithm can be linear interpolation, that is, between two known data points, the value of the intermediate unknown data point is estimated by the equation of a straight line. The formula for linear interpolation is:

[0079]

[0080] In time synchronization applications, х usually represents time and у represents the data value of the audio and video instrument.

[0081] Spatial alignment first requires determining the coordinate system. LiDAR usually uses a three-dimensional Cartesian coordinate system to represent point cloud data, where each point is represented by three coordinate values ​​(x, y, z) to represent its position in space. Image data from an imager is usually in a two-dimensional plane coordinate system, where each pixel is represented by two coordinate values ​​(u, v) to represent its position on the image plane.

[0082] First, we need to clarify the definitions and relationships of these two coordinate systems in order to perform coordinate transformation.

[0083] In order to transform the laser radar point cloud data from the three-dimensional coordinate system to the two-dimensional image plane coordinate system of the imager, a suitable transformation matrix needs to be established. This transformation matrix can be determined by the relative position and posture information of the laser radar and the imager. Then, a transformation matrix is ​​constructed based on these parameters, which can transform the three-dimensional coordinates of the laser radar into the two-dimensional coordinates on the image plane of the imager.

[0084] For each point cloud data point (x, y, z) of the laser radar, by multiplying it with the transformation matrix, the corresponding two-dimensional coordinates (u, v) on the image plane of the imager can be obtained. This process can be expressed by the mathematical formula:

[0085]

[0086] Among them, M is the transformation matrix, which contains transformation parameters such as rotation, translation and scaling.

[0087] After coordinate transformation, the accuracy of the transformation matrix may be affected by the accuracy of the measuring equipment and environmental factors. Therefore, calibration is required to improve the accuracy of the data.

[0088] Initial calibration can be performed by placing some markers of specific shapes and colors with known positions and sizes in the industrial environment. Then, the data of these markers are collected using the lidar and the imager, respectively, and compared with the known position information. By adjusting the parameters of the transformation matrix, the point cloud data of the lidar can be better aligned with the image data of the imager.

[0089] Error correction technology is then used to further improve the accuracy of the data. This involves filtering the lidar point cloud data to remove noise points, and performing geometric correction and color correction on the image data of the imager to eliminate image distortion and color deviation.

[0090] In step data fusion and comparison, feature layer fusion includes:

[0091] Data normalization: After further inspection and cleaning of the data layer fusion, the data is normalized so that the data with different features have the same scale and range.

[0092] Feature extraction: For lidar data and camera data, further extract higher-level and discriminative features.

[0093] Feature fusion: A fusion algorithm combining principal component analysis (PCA) and independent component analysis (ICA) is used to select important components as fused features.

[0094] Data normalization is because after the data layer is fused, the data of the lidar and the audio-visual camera may have different scales and ranges. If normalization is not performed, these data of different scales and ranges may cause some features to be over-emphasized and some features to be ignored during feature extraction and fusion. Normalization can convert data of different features to the same scale and range, so that they have the same importance in subsequent analysis. This can improve the accuracy and reliability of data analysis and avoid erroneous results caused by differences in data scale and range. Specifically, the minimum-maximum normalization method is used, that is, the data is mapped to a specific interval, usually [0,1] or [-1,1]. The calculation formula is:

[0095]

[0096] In the step data fusion and comparison, the decision layer fusion includes:

[0097] Generate independent decisions: Use clustering algorithms to identify and classify objects to obtain the decision results of the lidar; use image recognition algorithms and sound recognition algorithms to identify and classify objects respectively to obtain the decision results of the audio and video camera.

[0098] Decision fusion: The DS evidence theory fusion algorithm is used to fuse the decision results of the lidar and the audio-visual camera.

[0099] The clustering algorithm used by LiDAR is an unsupervised learning method that can be used to identify and classify objects in the point cloud data collected by LiDAR. The density-based clustering algorithm (DBSCAN) can be used: the data points are divided into different clusters according to the density distribution of the point cloud. Each cluster represents a possible object. The specific DBSCAN algorithm is mainly based on two core parameters: radius (Eps) and minimum number of points (MinPts): the radius (Eps) specifically defines a neighborhood range, that is, for a data point in the point cloud, if the number of data points in the spherical area with a radius of Eps centered on the point is not less than MinPts, then the point is called a core point; the minimum number of points (MinPts) is the threshold for determining whether a point is a core point, that is, if the number of data points in the neighborhood of a point is less than MinPts, then the point is marked as a noise point or a boundary point. The specific steps are: randomly select an unvisited point from the point cloud and determine whether it is a core point; if it is a core point, take this point as the center, classify all points in its neighborhood (direct density reachable points) into a cluster, and continue to expand the cluster, and include points in the neighborhood of the points in the neighborhood into the cluster (as long as these points are also core points, that is, indirectly density reachable points); if it is not a core point, it may be marked as a noise point or a boundary point. If it is subsequently found that the point is density reachable from other core points, it will be reclassified into the corresponding cluster; repeat the above process until all points have been visited, and the different clusters obtained at this time represent different objects.

[0100] DS evidence theory is a method for processing uncertain information. It can fuse evidence from different sources to obtain a comprehensive decision result. In DS evidence theory, each piece of evidence is represented as a trust function, which defines the degree of trust in different hypotheses. By combining multiple trust functions, a comprehensive trust function can be obtained, thereby obtaining the final decision result. The specific fusion process is as follows: First, the decision results of the lidar and the audio-visual camera are respectively represented as evidence in the DS evidence theory. That is, if the lidar determines that the probability of an object being category A is 0.7, the probability of being category B is 0.2, and the probability of being category C is 0.1, then it can be represented as a trust allocation function m1, where m1(A) = 0.7, m1(B) = 0.2, m1(C) = 0.1, and m1(Θ) = 0 (Θ represents the entire recognition framework, and the remaining probability is allocated to the entire framework here). Similarly, the decision result of the audio-visual instrument can also be expressed as a trust assignment function m2; then, using the Dempster combination rule, by assuming that m1 and m2 are two trust assignment functions, their combination result m = m1 ⊕ m2 is calculated by the following formula:

[0101]

[0102] The fusion result m obtained through the evidence combination rule is also a trust distribution function, which represents the degree of trust in different object categories after comprehensively considering the decision results of the lidar and the audio-visual camera. That is to say, if the fused m(A) value is high, it means that based on the decision results of the two devices, it is more likely that the object is of category A.

[0103] In the step data fusion and comparison, the preliminary comparison includes:

[0104] Range comparison: For the point cloud data of the LiDAR, determine the preset value of the spatial coordinate range; for the image data of the imager, set the range of pixel values.

[0105] Statistical feature comparison: Calculate the statistical features of the fused data and compare them with the preset normal statistical feature range.

[0106] Consistency comparison: Compare the consistency of the fused data of LiDAR and audio-visual camera in corresponding positions and times.

[0107] LiDAR determines the distance and position of the target object by emitting a laser beam and measuring the time of reflected light. The generated point cloud data has clear spatial coordinate information. When performing range comparison, it is necessary to first determine the preset value of the spatial coordinate range. This preset value can be set according to the specific application scenario and requirements. That is to say, in the geographic information system, the preset value may be determined based on the boundary range of the map or the coordinate range of a specific area. By comparing the actual spatial coordinate range of the point cloud data with the preset value, it can be determined whether the data is within a reasonable range, whether there is an abnormal offset or beyond expectations. The image data collected by the imager is usually expressed in pixels. When performing range comparison, it is necessary to set the range of pixel values. This range can include the value range of pixel brightness, color, contrast, etc.

[0108] Statistical features may include mean, variance, standard deviation, median, mode, etc. For the fused LiDAR point cloud data and image data of the image sensor, these statistical features may be calculated separately. For example, for point cloud data, the average distance of the point cloud, the variance of the distance, etc. may be calculated; for image data, the average brightness of the pixel, the variance of the brightness, etc. may be calculated.

[0109] In step data fusion and comparison, the secondary comparison includes:

[0110] Feature distance comparison: Calculate the distance between the fused feature and the preset normal feature.

[0111] Feature pattern comparison: Establish a feature pattern library under normal conditions, and compare the fused features with the feature patterns in the pattern library. First, it is necessary to determine the preset normal features, which can be obtained by analyzing and counting a large amount of data under normal conditions. That is, for the data after the fusion of the lidar and the audio-visual camera, multiple sets of data under normal working conditions can be collected, and the average and standard deviation of their features can be calculated as the reference range of the preset normal features.

[0112] Then, the distance between the fused feature and the preset normal feature is calculated. The distance calculation method can be selected according to the type of feature. If the calculated distance is large, it means that there is a large difference between the fused feature and the preset normal feature, which may indicate that there is anomaly in the data.

[0113] The purpose of establishing a feature pattern library under normal circumstances is to provide a reference standard for comparing the fused data features. The feature pattern library can be obtained by analyzing and summarizing historical data, or it can be constructed based on domain knowledge and experience. When establishing a feature pattern library, different application scenarios and data types need to be considered. When comparing feature patterns, the fused features are matched one by one with the feature patterns in the pattern library.

[0114] In the step data fusion and comparison, the final comparison includes:

[0115] Comparison of decision-making result consistency: Compare the results of decision-making layer fusion with the preset normal decision results.

[0116] Abnormal probability comparison: According to the algorithm fused by the decision layer, the probability of abnormality in the current data is calculated and compared with the preset abnormal probability threshold.

[0117] Comprehensive evaluation and comparison: Combine the comparison results of data layer fusion and feature layer fusion to comprehensively evaluate the results of decision layer fusion. When conducting comprehensive evaluation and comparison, it is necessary to combine the comparison results of the three levels of data layer, feature layer and decision layer to comprehensively evaluate the results of decision layer fusion.

[0118] First, we review the comparison results of data layer fusion, including the consistency of data sources, the rationality of data range, etc. These results can reflect the basic quality and reliability of the data. Next, we consider the comparison results of feature layer fusion, such as feature similarity, distance measurement, etc. These results can reveal the intrinsic characteristics and patterns of the data. Finally, we combine the comparison results of decision layer fusion, such as the consistency of decision results and the probability of abnormality, to comprehensively judge the accuracy and reliability of the fusion results.

[0119] Principal component analysis (PCA) includes calculating the covariance matrix of the data, performing eigenvalue decomposition on the covariance matrix, obtaining eigenvalues ​​and eigenvectors, selecting the eigenvectors corresponding to the largest eigenvalues, forming a new feature space, and projecting the original data into the new feature space to obtain the reduced dimension laser radar and audio-visual camera data features. The specific steps are as follows:

[0120] First, the data of the laser radar and the imager are integrated to form a unified data matrix. Assuming that the original data has n samples and each sample has m features, then the data matrix is ​​an n×m matrix, where each row represents a sample and each column represents a feature. The element C of the covariance matrix is ij Represents the covariance between the i-th feature and the j-th feature. The formula for calculating covariance is:

[0121]

[0122] Where X ki represents the i-th eigenvalue of the k-th sample, represents the mean of the i-th feature.

[0123] Then, the covariance matrix is ​​decomposed by eigenvalue, that is, a diagonal matrix Λ and a matrix V are found such that

[0124] C=VΛV -1 .

[0125] Then, numerical calculation methods, such as Jacobi method and QR decomposition method, are used to perform eigenvalue decomposition on the covariance matrix.

[0126] Since the eigenvalue represents the variance of the data in the direction of the corresponding eigenvector, the larger the eigenvalue, the more obvious the data change in that direction, and the more important the eigenvector. The eigenvectors can be sorted according to the size of the eigenvalue, and the eigenvectors corresponding to the largest eigenvalues ​​can be selected.

[0127] The selected k feature vectors form a new feature space. The dimension of this feature space is m×k, where each column represents a feature vector.

[0128] After that, the original data is projected into the new feature space to obtain the reduced dimension lidar and audio-visual data features.

[0129] That is, the original data matrix X (after centralization) is combined with the new feature space matrix V k (composed of the selected k eigenvectors) are multiplied to obtain the reduced-dimensional data matrix Y = XV k .

[0130] The data after dimensionality reduction retains the main information of the original data while removing noise and redundant information.

[0131] Independent component analysis (ICA) involves merging the reduced-dimensional lidar and audio-visual camera data features into one data set, performing ICA processing on the merged data set and obtaining independent components, and then selecting the important components as the fused features. After principal component analysis (PCA) dimensionality reduction, the lidar and audio-visual camera data features are converted into low-dimensional representations. These reduced-dimensional features usually have fewer dimensions while retaining the main information of the original data. By assuming that the lidar data is reduced by PCA, an n×k1 matrix X1 is obtained, where n is the number of samples and k1 is the feature dimension after dimensionality reduction. Similarly, the audio-visual camera data is reduced to an n×k2 matrix X2.

[0132] Then, the reduced-dimensional lidar and audio-visual camera data features are merged into one dataset. The two matrices can be merged into an n×(k1+k2) matrix X through a simple concatenation operation.

[0133] After that, the merged data set X is first centered to make its mean zero. Then whitening is performed to convert the covariance matrix of the data into a unit matrix. Whitening can simplify subsequent calculations and make each dimension of the data have the same variance. Then the ICA algorithm is used to estimate the independent component matrix and determine the order and scale of the independent components; after obtaining the independent components, the importance of each independent component needs to be evaluated, including but not limited to calculating the variance, information entropy, correlation, etc. of the components. According to the evaluation results of the importance of the components, the important components are selected as the fused features.

[0134] An integrated system for performing ultra-precision laser radar and audio-visual integrated monitoring methods for industrial environments, comprising:

[0135] LiDAR equipment: It is used to emit laser beams and receive reflected signals to obtain three-dimensional spatial information of objects.

[0136] Audio-visual equipment: It consists of an image acquisition part and a sound acquisition part; the image acquisition part can obtain visual image information of an object, including color, texture, shape and other features; the sound acquisition part can capture sound signals in the environment to analyze the sound characteristics of an object or detect abnormal sounds.

[0137] Central control module: It is composed of a microprocessor and a memory unit. The microprocessor is responsible for receiving startup instructions, checking sensor status, triggering data acquisition, receiving and storing data from the lidar and audio-visual camera, and performing operations such as data fusion and comparison. The memory unit is used to store program code, intermediate data and final processing results.

[0138] Signal processing module: It is composed of a lidar signal processor, an audio-visual instrument signal processor and an image processor; the lidar signal processor is used to amplify, filter, digitize and perform other processing on the signals collected by the lidar, and the audio-visual instrument signal processor and image processor are used to perform preliminary processing on the image and sound signals collected by the audio-visual instrument.

[0139] Data fusion and comparison module: consists of a time synchronization unit, a space alignment unit, a data layer fusion unit, a feature layer fusion unit and a comparison unit; the time synchronization unit is used to add timestamps to the data points of the laser radar and the imager during data collection, and match and synchronize the data of the two devices according to the timestamps; the space alignment unit is used to project the point cloud data of the laser radar onto the image plane of the imager to achieve space alignment; the data layer fusion unit is used to preliminarily fuse the data of the laser radar and the imager; the feature layer fusion unit is used to extract and fuse features of the data after data layer fusion; the comparison unit is used to perform preliminary comparison after data layer fusion, secondary comparison after feature layer fusion, and final comparison after decision layer fusion.

[0140] Decision layer fusion module: It consists of an independent decision generation unit and a decision fusion unit; the independent decision generation unit is used to identify and classify the lidar data and the audio-visual device data to obtain their respective decision results, and the decision fusion unit is used to fuse the decision results of the lidar and the audio-visual device to obtain the final decision result.

[0141] Visual output and storage module: consists of a communication unit and an output unit; the communication unit is responsible for storing the processed data in a suitable database system, and the output unit is used for visual display so that users can intuitively understand the monitoring results.

[0142] Working principle: When working, the user line sets the laser radar scanning range to the entire factory space through the central control module, adjusts the resolution to be able to detect smaller object changes, and the scanning frequency is 10 times per second. At the same time, the image resolution of the audio and video camera is set to high-definition level, the frame rate is 30 frames / second, and the audio acquisition sensitivity is set to medium to capture the sound of equipment operation and possible abnormal noise.

[0143] After receiving the start command, the central control module checks the status of the laser radar and the audio-visual camera. After confirming that the equipment is normal, the laser radar and the audio-visual camera are triggered at the same time to start data collection. The laser radar continuously scans the objects in the factory and records the location and distance information of the objects. The audio-visual camera collects the image and sound data in the factory at the same time.

[0144] The signal processing module amplifies and preliminarily processes the collected signals. The laser radar signal processor amplifies the weak light signal and filters it to remove noise. The audio-visual signal processor and image processor remove noise, enhance contrast and brightness, and amplify the audio signal to improve the signal-to-noise ratio.

[0145] The data fusion and comparison module starts working. The time synchronization unit adds timestamps to the data points of the lidar and the imager, and performs data matching and synchronization. The spatial alignment unit projects the point cloud data of the lidar onto the image plane of the imager and calibrates it by placing markers of known position and size. The data layer fusion unit fuses the data using the weighted average method. The feature layer fusion unit normalizes, extracts features, and fuses the fused data. The comparison unit performs comparisons at different stages to determine whether there are any anomalies.

[0146] The decision layer fusion module identifies and classifies objects. The laser radar uses a clustering algorithm to divide the point cloud data into different clusters, representing different objects. The imager uses an image recognition algorithm and a sound recognition algorithm to identify objects respectively. The decision fusion unit uses the DS evidence theory fusion algorithm to fuse the decision results of the two devices.

[0147] The visualization output and storage module stores the processed data in the database system, and integrates the point cloud data of the laser radar and the image data of the audio-visual camera through the 3D visualization software to display the 3D scene of the factory. At the same time, the statistical characteristics and comparison results of the data are presented in the form of charts and reports, which makes it convenient for managers to intuitively understand the safety status of the factory.

Claims

1. An ultra-precision laser radar and audio-visual integrated monitoring method for industrial environments, characterized in that: The following steps are involved: Parameter setting: After receiving the start-up command, the central control module checks the status of the laser radar and the audio-visual camera, and the user sets the detection parameters of the laser radar and the audio-visual camera on the control interface; Data collection: The central control module triggers the lidar and the audio-visual camera simultaneously to start data collection; Initial information processing: The signals collected by the laser radar and the audio-visual camera will be amplified by their respective internal signal processors. At the same time, the image information collected by the audio-visual camera will be initially processed by the image processor. Subsequently, the amplified and initially processed signals will enter the analog-to-digital converter ADC to be converted into digital signals, and finally transmitted to the central control module through the high-speed data bus. Data fusion and comparison: After receiving data from the lidar and the audio-visual camera, the central control module will first store the received data, then perform data-layer fusion on the data, and make a preliminary comparison between the fused data and the preset data to determine whether there is an abnormality. Then, perform feature-layer fusion on the data, and make a secondary comparison between the fused data and the preset data to determine whether there is an abnormality. Finally, perform decision-layer fusion on the data, and make a final comparison between the fused data and the preset data to determine whether there is an abnormality. Visual output and storage: The processed data is stored in a suitable database system through the communication module and visualized through the output module.

2. The ultra-precision laser radar and audio-visual integrated monitoring method for industrial environments according to claim 1 is characterized in that: In the data fusion and comparison steps, data layer fusion includes: Time synchronization: Add a timestamp to each data point when collecting data, and then match and synchronize the data of the two devices based on the timestamp; Spatial alignment: Project the lidar point cloud data onto the image plane of the imager through coordinate transformation and calibration; Weighted average method to fuse data: Assign weights to the lidar and imager data based on the performance analysis results and fuse all corresponding data points to obtain fused point cloud data and imager image data.

3. The ultra-precision laser radar and audio-visual integrated monitoring method for industrial environments according to claim 1 is characterized in that: In the data fusion and comparison steps, feature layer fusion includes: Data normalization: After further inspection and cleaning of the data layer fusion, the data is normalized so that the data with different features have the same scale and range; Feature extraction: For lidar data and audio-visual data, further extract more advanced and discriminative features; Feature fusion: A fusion algorithm combining principal component analysis and independent component analysis is used to select important components as fused features.

4. The ultra-precision laser radar and audio-visual integrated monitoring method for industrial environments according to claim 1 is characterized in that: In the data fusion and comparison steps, the decision layer fusion includes: Generate independent decisions: Use clustering algorithms to identify and classify objects to obtain the decision results of the lidar; use image recognition algorithms and sound recognition algorithms to identify and classify objects respectively to obtain the decision results of the audio-visual camera; Decision fusion: The DS evidence theory fusion algorithm is used to fuse the decision results of the lidar and the audio-visual camera.

5. The ultra-precision laser radar and audio-visual integrated monitoring method for industrial environments according to claim 1, characterized in that: In the data fusion and comparison steps, the preliminary comparison includes: Range comparison: For the point cloud data of the laser radar, determine the preset value of the spatial coordinate range; for the image data of the imager, set the range of pixel values; Statistical feature comparison: Calculate the statistical features of the fused data and compare them with the preset normal statistical feature range; Consistency comparison: Compare the consistency of the fused data of LiDAR and audio-visual camera in corresponding positions and times.

6. The ultra-precision laser radar and audio-visual integrated monitoring method for industrial environments according to claim 1, characterized in that: In the data fusion and comparison steps, the secondary comparison includes: Feature distance comparison: calculate the distance between the fused feature and the preset normal feature; Feature pattern comparison: Establish a feature pattern library under normal conditions and compare the fused features with the feature patterns in the pattern library.

7. The ultra-precision laser radar and audio-visual integrated monitoring method for industrial environments according to claim 1, characterized in that: In the data fusion and comparison steps, the final comparison includes: Comparison of decision-making result consistency: Compare the results of decision-making layer fusion with the preset normal decision results; Abnormal probability comparison: According to the algorithm integrated by the decision layer, the probability of abnormality in the current data is calculated and compared with the preset abnormal probability threshold; Comprehensive evaluation and comparison: Combined with the comparison results of data layer fusion and feature layer fusion, a comprehensive evaluation is conducted on the results after decision layer fusion.

8. The ultra-precision laser radar and audio-visual integrated monitoring method for industrial environments according to claim 3 is characterized in that: The principal component analysis includes calculating the covariance matrix of the data, performing eigenvalue decomposition on the covariance matrix, obtaining eigenvalues ​​and eigenvectors, selecting eigenvectors corresponding to the largest eigenvalues, forming a new feature space, and projecting the original data into the new feature space to obtain the reduced-dimensional laser radar and audio-visual device data features.

9. The ultra-precision laser radar and audio-visual integrated monitoring method for industrial environments according to claim 3, characterized in that: The independent component analysis includes merging the laser radar and audio-visual camera data features after dimensionality reduction into one data set, processing the merged data set and obtaining independent components, and then selecting important components as fused features.

10. An integrated system for executing the ultra-precision laser radar and audio-visual integrated monitoring method for industrial environments as described in any one of claims 1 to 9, characterized in that: include: LiDAR equipment: It is used to emit laser beams and receive reflected signals to obtain three-dimensional spatial information of objects; Audio-visual equipment: It consists of an image acquisition part and a sound acquisition part; the image acquisition part can obtain visual image information of an object, including characteristics such as color, texture, and shape; The sound collection part can capture sound signals in the environment for analyzing the sound characteristics of objects or detecting abnormal sounds; Central control module: It consists of a microprocessor and a memory unit. The microprocessor is responsible for receiving startup instructions, checking sensor status, triggering data acquisition, receiving and storing data from the lidar and the audio-visual camera, and performing operations such as data fusion and comparison. The memory unit is used to store program code, intermediate data and final processing results. Signal processing module: composed of a laser radar signal processor, an audio-visual instrument signal processor and an image processor; the laser radar signal processor is used to amplify, filter, digitize and process the signals collected by the laser radar, and the audio-visual instrument signal processor and image processor are used to perform preliminary processing on the image and sound signals collected by the audio-visual instrument; Data fusion and comparison module: composed of a time synchronization unit, a space alignment unit, a data layer fusion unit, a feature layer fusion unit and a comparison unit; the time synchronization unit is used to add timestamps to the data points of the laser radar and the imager during data collection, and match and synchronize the data of the two devices according to the timestamps; the space alignment unit is used to project the point cloud data of the laser radar onto the image plane of the imager to achieve space alignment; the data layer fusion unit is used to preliminarily fuse the data of the laser radar and the imager; the feature layer fusion unit is used to extract and fuse the features of the data after data layer fusion; the comparison unit is used to perform preliminary comparison after data layer fusion, secondary comparison after feature layer fusion, and final comparison after decision layer fusion; Decision layer fusion module: It consists of an independent decision generation unit and a decision fusion unit; the independent decision generation unit is used to identify and classify the laser radar data and the sound imager data to obtain their respective decision results, and the decision fusion unit is used to fuse the decision results of the laser radar and the sound imager to obtain the final decision result; Visual output and storage module: consists of a communication unit and an output unit; the communication unit is responsible for storing the processed data in a suitable database system, and the output unit is used for visual display so that users can intuitively understand the monitoring results.

Images