Multi-sensor fused all-weather operation target sensing and measuring system and method

Abstract

The invention discloses a multi-sensor fused all-weather operation target sensing and measuring system and method, and belongs to the field of intelligent sensing and measuring, the system comprises a data acquisition module, an all-weather image enhancement module, a three-dimensional modeling and semantic segmentation module and a motion detection module, and the method comprises the following steps: performing data acquisition on an operation target; performing all-weather image enhancement processing on the acquired image data; performing image semantic segmentation on the enhanced image; performing point cloud semantic modeling by using a result after image semantic segmentation in combination with laser point cloud and IMU (Inertial Measurement Unit) data; and using a result after image semantic segmentation to assist optical flow method calculation to realize motion detection. According to the invention, the limitation of a single sensor can be overcome, the sensing and measuring capabilities of the system in a complex environment are improved, and all-weather and all-time high-precision sensing and measuring tasks are realized. More reliable technical support is provided for application in the fields of automatic driving, unmanned aerial vehicles, intelligent robots and the like, and more intelligent sensing and measurement are achieved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent perception and measurement, and in particular to a multi-sensor fusion all-weather operation target perception and measurement system and method. Background Art

[0002] With the development of technologies such as autonomous driving, intelligent robots, and drones, the requirements for the accuracy and stability of target perception and measurement are increasing. In practical applications, the complex and changeable environmental conditions, especially inclement weather such as night, rain, fog, and snow, pose severe challenges to traditional single sensor systems. For example, visual sensors have limited perception capabilities under low light or strong light, lidar is affected by scattering effects in rainy and foggy weather, and inertial measurement units (IMUs) are easily affected by sensor drift, which reduces the accuracy of perception.

[0003] In the existing technology, common operation target perception and measurement methods mostly rely on a single sensor, such as relying solely on vision or lidar to perceive the target. However, these single sensor solutions are difficult to provide stable and accurate perception data under complex weather conditions and cannot meet the needs of all-weather operations. Therefore, how to effectively integrate multi-sensor information from vision, lidar and IMU to form a stable and robust all-weather perception and measurement system has become a research hotspot in the current technology field. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a multi-sensor fusion all-weather operation target perception and measurement system and method. It aims to solve the problem of insufficient perception and measurement capabilities of a single sensor in complex outdoor environments, especially in severe weather conditions such as night, rain, fog and snow. The method achieves high-precision perception and measurement of operation targets by combining multi-sensor data of vision, lidar and IMU.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a multi-sensor fusion all-weather operation target perception and measurement system, including a data acquisition module composed of a visual sensor, a laser radar and an IMU sensor, and the data acquisition module is provided with an interconnected all-weather image enhancement module, a three-dimensional modeling and semantic segmentation module and a motion detection module;

[0006] The all-weather image enhancement module includes a weather recognition module, a gating network, a processing module and a weighted fusion module which are connected in sequence, wherein the processing module includes a defogging module, a low-light enhancement module, a rain removal module and a snow removal module; the visual sensor is connected to the weather recognition module in communication;

[0007] The 3D modeling and semantic segmentation module includes an interconnected image semantic segmentation network, a mileage calculation module, a semantic mapping module, and a mapping module; the laser radar, IMU sensor, and mileage module are connected in communication; the weighted fusion module and the image semantic segmentation network are connected in communication;

[0008] The motion detection module includes a region classification module, an optical flow calculation module, a region motion extraction module and a motion detection result module which are connected in sequence. The weighted fusion module and the image semantic segmentation network are both connected to the region classification module in communication.

[0009] The further improvement of the technical solution of the present invention is that the image semantic segmentation network adopts an encoding-decoding structure, the encoder part is an STDC network responsible for gradually extracting multi-scale feature information, and the last layer of the encoder introduces a pyramid strip pooling module PSPM. The PSPM module first extracts local features in the horizontal and vertical directions through strip pooling with kernel sizes of 1×1, 2×2, and 4×4. The pooled multi-scale feature map is restored to the size of the original feature map through convolution and upsampling. The convolution kernel size is 1×1, and then the features with the same pooling kernel size are spliced ​​together. The spliced ​​feature maps are further fused through the convolution layer, and finally the fused multi-scale features are added and the features are finely adjusted through the convolution layer to integrate information from different scales to generate the final output features; the decoder part adopts the FLD structure for step-by-step The multi-level features extracted by the encoder are fused in steps to generate the final output image. FLD consists of two dual attention fusion modules DAFM and a segmentation head. The input features of DAFM are composed of shallow features F-low in the encoding stage and deep features F-high in the decoding stage. F-high is upsampled to F-up through bilinear interpolation, which has the same size as F-low. Then the two features are concatenated and the concatenated features are input into the effective channel attention ECA and cross attention CCA modules in turn. Finally, the output fusion feature F-output is obtained through convolution operation. In the segmentation head, convolution, bn, and relu operations are performed to reduce the number of feature channels to the number of classes. Then an upsampling operation is performed to expand the feature size to the input image size, and an argmax operation is performed to predict the label of each pixel.

[0010] A multi-sensor fusion all-weather operation target perception measurement method: The specific steps are as follows:

[0011] Step 1: Collect data on the operation target;

[0012] Step 2: Perform all-weather image enhancement processing on the collected image data;

[0013] Step 3: Perform image semantic segmentation on the enhanced image;

[0014] Step 4: Use the results of image semantic segmentation combined with laser point cloud and IMU data to perform point cloud semantic modeling;

[0015] Step 5: Use the results of image semantic segmentation to assist in optical flow calculation and achieve motion detection.

[0016] The further improvement of the technical solution of the present invention is that the specific steps of step 2 are as follows:

[0017] Step 2.1: Use the image data collected by the visual sensor under different weather and lighting conditions, that is, image data under five weather conditions: sunny, rainy, snowy, foggy, and low-light, to construct an image classification dataset, and use the dataset to train the weather recognition module using a deep learning network to obtain a trained weather recognition module;

[0018] Step 2.2: Input the image data collected by the visual sensor into the weather recognition module for weather recognition, and output the one-hot encoding vector representing the weather category to the gating network and each processing module;

[0019] Step 2.3: The gated network generates the weights of each processing module in real time based on the input image feature vector based on a dynamic adjustment mechanism, and each processing module processes the image accordingly according to the weights;

[0020] Step 2.4: The outputs of each processing module are effectively combined through weighted averaging to generate a higher quality final image.

[0021] A further improvement of the technical solution of the present invention is that the specific steps of step 3 are: inputting the enhanced image into the image semantic segmentation network, extracting semantic information, identifying and segmenting different scenes and targets, and obtaining a semantic label map, which contains category information of various objects in the scene.

[0022] The further improvement of the technical solution of the present invention is that the specific steps of step 4 are as follows:

[0023] Step 4.1: Accumulate the discrete laser points collected by the laser radar into a point cloud frame format, and extract the spatial geometric information of the work scene from the point cloud data collected by the laser radar;

[0024] Step 4.2: Calculate the optimal estimate of the sensor motion between point cloud frames based on the error iterative Kalman filter algorithm, obtain the corrected laser point cloud world coordinates, and transfer the point cloud frames to a unified global map;

[0025] Step 4.3: 3D semantic point cloud reconstruction, combining the extracted visual semantic information with the point cloud data generated by the lidar, and reconstructing the 3D semantic point cloud model of the work target by mapping the semantic labels to the point cloud data;

[0026] Step 4.4: voxel down-sample the laser point cloud frame, use the pre-calibrated laser radar and camera extrinsic RT matrix to perform coordinate transformation, convert the three-dimensional coordinate value of the laser point into the spatial coordinate value under the camera normalized coordinate system, and then calculate the corresponding two-dimensional pixel coordinate value;

[0027] Step 4.5: After obtaining the pixel coordinates corresponding to the laser point, interpolation calculation is performed on the coordinates to obtain the semantic information of the laser point in the image frame;

[0028] Step 4.6: According to the overlapping area of ​​the laser radar and camera FOV, filter out the laser points outside the field of view, assign the calculated semantic information to each laser point in the camera field of view, and use the fused semantic information to perform point cloud filtering, retaining only the point cloud model of the work target. As the laser radar moves, continuously obtain the three-dimensional color semantic point cloud frames of the working environment, and according to the motion estimation results, splice the collected discrete color semantic point cloud frames into a complete color semantic three-dimensional point cloud model.

[0029] The further improvement of the technical solution of the present invention is that the specific steps of step 5 are as follows:

[0030] Step 5.1: The region classification module receives the result of image semantic segmentation, analyzes the semantic labels of each segmented region, and classifies these regions into "reference" regions and "test" regions;

[0031] Step 5.2: Use the optical flow algorithm to calculate the optical flow between two adjacent frames of images, obtain the motion vector of each pixel, process multiple frames of image data, and accumulate optical flow information;

[0032] Step 5.3: Initialize the key points of optical flow tracking in the selected area and track the key points in the selected area;

[0033] Step 5.4: Perform statistical analysis on the optical flow vectors of the tracked reference area and the area to be measured, and calculate their average motion vector to determine the relative motion between the two.

[0034] The further improvement of the technical solution of the present invention is that: after step 5.2, an adaptive weight adjustment mechanism is designed to dynamically adjust the weight according to the optical flow stability and regional importance of each region. The optical flow stability is evaluated by calculating the variance of the regional optical flow vector, and the regional importance is set according to the semantic label. The results of semantic segmentation are used to assign different importance weights to the regions. Combining the optical flow stability and regional importance, the comprehensive weight of each region is calculated. The formula is as follows:

[0035]

[0036] Among them, W i is the final weight of region i, Si is the variance of the optical flow vector in region i, I i is the semantic label weight of region i, α and β are balance coefficients, a threshold of the comprehensive weight is set, and only regions whose comprehensive weight exceeds the threshold are selected for optical flow tracking. This ensures that the selected regions are both stable in motion and highly important in semantics.

[0037] The further improvement of the technical solution of the present invention is that the specific steps of step 5.4 are as follows:

[0038] Step 5.41: Calculate the average motion vector of the reference area. Perform statistics on the optical flow vectors in the reference area and calculate its average motion vector:

[0039]

[0040] in:

[0041] V static,i Represents the optical flow vector of the i-th pixel in the reference area; N is the number of all pixels in the reference area; Avg_Static_Motion obtained by this formula is used to represent the overall motion trend of the reference area;

[0042] Step 5.4.2: Calculate the average motion vector of the area to be tested. Perform statistics on the optical flow vectors of the area to be tested and calculate the average motion vector of the area:

[0043]

[0044] in:

[0045] V motion,j Represents the optical flow vector of the jth pixel in the area to be tested; M is the number of pixels in the area to be tested; this average motion vector Avg_Motion_Motion can reflect the motion characteristics of the object in the area to be tested;

[0046] Step 5.4.3: Calculation of relative motion vector. In order to determine the motion of the tested area relative to the reference area, the average motion vectors of the two are compared. The calculation formula of the relative motion vector is:

[0047] Relative_Motion=Avg_Motion_Motion-Avg_Static_Motion

[0048] in:

[0049] Avg_Motion_Motion is the average motion vector of the area to be tested; Avg_Static_Motion is the average motion vector of the reference area;

[0050] Step 5.4.4: Based on the calculated relative motion vector Relative_Motion, determine the motion state of the area to be tested relative to the reference area.

[0051] Due to the adoption of the above technical solution, the technical progress achieved by the present invention is: the accuracy and stability of perception are improved through multi-sensor information fusion technology. The visual sensor can provide high-resolution image information for identifying and classifying targets in the scene; the laser radar provides accurate three-dimensional spatial distance information to build a three-dimensional model of the operating target; the IMU is used to provide the system's motion state estimation and realize accurate registration and data fusion between multiple sensors. The visual image is processed by a deep learning algorithm to extract the semantic information of the target, and combined with the point cloud data generated by the laser radar, a high-precision three-dimensional model reconstruction is achieved, and then the sensor registration and filtering are performed in combination with the IMU's motion data.

[0052] This multi-sensor fusion method can overcome the limitations of a single sensor and improve the system's perception and measurement capabilities in complex environments, especially in severe weather conditions, to achieve high-precision perception and measurement tasks all day and all night. The invention provides more reliable technical support for applications in the fields of autonomous driving, drones, intelligent robots, etc., and can cope with complex and changeable outdoor environments, achieving more intelligent perception and measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 It is a schematic diagram of the structure of the measurement system;

[0055] Figure 2 It is a schematic diagram of the structure of the image enhancement module;

[0056] Figure 3 It is a schematic diagram of the structure of the image semantic segmentation network;

[0057] Figure 4 It is a schematic diagram of the structure of PSPM in the image semantic segmentation network;

[0058] Figure 5 It is a schematic diagram of the structure of DAFM in the image semantic segmentation network;

[0059] Figure 6 It is the image semantic segmentation result map;

[0060] Figure 7 It is a three-dimensional semantic point cloud modeling structure diagram;

[0061] Figure 8 This is the result of semantic segmentation of 3D point cloud of bulk carrier;

[0062] Fig. 9 It is the structural diagram of the motion detection module;

[0063] Fig.10 It is the result of bulk carrier movement detection in the port; DETAILED DESCRIPTION

[0064] The present invention is further described in detail below in conjunction with embodiments:

[0065] Embodiment 1:

[0066] like Figure 1 As shown, it is a structural schematic diagram of a multi-sensor fusion all-weather operation target perception and measurement system. This embodiment is an implementation method of the present invention applied to bulk cargo ports. It includes a data acquisition module composed of a visual sensor, a solid-state laser radar and an IMU sensor, and an all-weather image enhancement module, a three-dimensional modeling and semantic segmentation module and a motion detection module are arranged behind the data acquisition module; wherein the visual sensor is a monocular color camera. The integrated monocular color camera and solid-state laser radar are deployed on a gantry crane, and an industrial computer is used as a processor. The all-weather operation target perception and measurement method proposed by the present invention is applied in a port environment, respectively targeting image enhancement under different port weather and lighting conditions, semantic modeling of ships, and motion detection problems of gantry cranes.

[0067] All-weather image enhancement module such as Figure 2 As shown, it includes a weather recognition module, a gating network, a processing module and a weighted fusion module connected in sequence, wherein the processing module includes a defogging module, a low-light enhancement module, a rain removal module and a snow removal module. The defogging module applies an advanced defogging algorithm; the low-light enhancement module combines a multi-scale low-light enhancement algorithm to adaptively adjust brightness and contrast; the rain removal module uses a rain removal method based on a generative adversarial network (GAN); and the snow removal module is used to remove snow and snowflake interference in the image.

[0068] The visual sensor and the weather recognition module are connected in communication; the weather recognition module consists of a convolutional layer and a fully connected layer. An attention mechanism is introduced between the convolutional layer and the fully connected layer to improve the accuracy of feature extraction. The gated network generates the weights of each processing module in real time. These weights are adaptively adjusted based on the features of the input image.

[0069] The 3D modeling and semantic segmentation module includes an interconnected image semantic segmentation network, a mileage calculation module, a semantic mapping module, and a mapping module; the laser radar, IMU sensor, and mileage module are connected in communication; the weighted fusion module and the image semantic segmentation network are connected in communication;

[0070] Image semantic segmentation network Figure 3 As shown in the figure, an encoding-decoding structure is adopted, and the encoder part is the STDC network responsible for gradually extracting multi-scale feature information, such as Figure 4 The last layer of the encoder shown in the figure introduces a pyramid strip pooling module PSPM. The PSPM module first extracts local features in the horizontal and vertical directions through strip pooling pairs with kernel sizes of 1×1, 2×2, and 4×4. The pooled multi-scale feature map is restored to the size of the original feature map through convolution and upsampling. The convolution kernel size is 1×1, and then the features with the same pooling kernel size are spliced ​​together. The spliced ​​feature maps are further fused through the convolution layer. Finally, the fused multi-scale features are added and the features are fine-tuned through the convolution layer to integrate information from different scales and generate the final output features. The decoder part adopts the FLD structure to gradually fuse the multi-level features extracted by the encoder and generate the final output image, as shown in Figure 5 The FLD shown is composed of two dual attention fusion modules DAFM and a segmentation head. The input features of DAFM are composed of shallow features F-low in the encoding stage and deep features F-high in the decoding stage. F-high is upsampled to F-up through bilinear interpolation, which has the same size as F-low. Then the two features are concatenated and the concatenated features are input into the effective channel attention ECA and cross attention CCA modules in turn. Finally, the output fusion feature F-output is obtained through convolution operation. In the segmentation head, convolution, bn, and relu operations are performed to reduce the channels of the features to the number of classes. Then an upsampling operation is performed to expand the feature size to the input image size, and an argmax operation is performed to predict the label of each pixel.

[0071] The motion detection module includes a region classification module, an optical flow calculation module, a region motion extraction module and a motion detection result module which are connected in sequence. The weighted fusion module and the image semantic segmentation network are both connected to the region classification module in communication.

[0072] Based on the above measurement system, a multi-sensor fusion all-weather operation target perception measurement method is implemented to detect bulk cargo ports. The specific steps are as follows:

[0073] Step 1: Collect data on the operation target; first, use a monocular color camera to collect port image data under different weather conditions, including sunny, rainy, snowy, foggy and low-light environments.

[0074] Step 2: Perform all-weather image enhancement processing on the collected image data;

[0075] Step 2.1: Use the image data collected by the visual sensor under different weather and lighting conditions, i.e., image data under five weather conditions: sunny, rainy, snowy, foggy, and low-light, to construct an image classification dataset, and use the dataset to train a weather recognition module using deep learning to obtain a trained weather recognition module;

[0076] Step 2.2: Input the image data collected by the visual sensor into the weather recognition module for weather recognition, and output the one-hot encoding vector representing the weather category to the gating network and each processing module;

[0077] Step 2.3: Based on the dynamic adjustment mechanism, the gated network generates the weights of each processing module in real time according to the input image feature vector, and each processing module processes the image accordingly according to the weight; the outputs of each processing module are fused through a weighted fusion strategy to generate an enhanced image. The formula is:

[0078] Enhanced_I image=∑i(Gate[i]×Modulei(Image))

[0079] Among them, the dynamic adjustment mechanism dynamically adjusts the parameters of each processing module according to the real-time analysis of image features (such as haze intensity, light level, raindrop density, etc.). For example: when the haze intensity is high, increase the processing intensity of the defogging module; in low-light conditions, enhance the brightness and contrast of the low-light enhancement module; when the raindrop density is high, adjust the processing intensity of the rain removal module.

[0080] Step 2.4: The outputs of each processing module are effectively combined through weighted averaging to generate a higher quality final image.

[0081] Step 3: If Figure 6 As shown, the enhanced image is subjected to image semantic segmentation; the enhanced image is input into the image semantic segmentation network to extract semantic information, identify and segment different scenes and targets, and obtain a semantic label map, which contains the category information of various objects in the scene. This information is saved in a specific data format for use in subsequent steps.

[0082] Step 4: Use the results of image semantic segmentation to perform point cloud semantic modeling; Figure 7 As shown:

[0083] Step 4.1: Accumulate the discrete laser points collected by the laser radar into a point cloud frame format, and extract the spatial geometric information of the work scene from the point cloud data collected by the laser radar;

[0084] Step 4.2: Calculate the optimal estimate of the sensor motion between point cloud frames based on the error iterative Kalman filter algorithm, obtain the corrected laser point cloud world coordinates, and transfer the point cloud frames to a unified global map;

[0085] Step 4.3: 3D semantic point cloud reconstruction, combining the extracted visual semantic information with the point cloud data generated by the lidar, and reconstructing the 3D semantic point cloud model of the work target by mapping the semantic labels to the point cloud data;

[0086] Step 4.4: voxel down-sample the laser point cloud frame, use the pre-calibrated laser radar and camera extrinsic RT matrix to perform coordinate transformation, convert the three-dimensional coordinate value of the laser point into the spatial coordinate value under the camera normalized coordinate system, and then calculate the corresponding two-dimensional pixel coordinate value;

[0087] Step 4.5: After obtaining the pixel coordinates corresponding to the laser point, interpolation calculation is performed on the coordinates to obtain the semantic information of the laser point in the image frame;

[0088] Step 4.6: According to the overlapping area of ​​the laser radar and camera FOV, filter out the laser points outside the field of view, assign the calculated semantic information to each laser point in the camera field of view, and use the fused semantic information to perform point cloud filtering, retaining only the point cloud model of the work target. As the laser radar moves, continuously obtain the three-dimensional color semantic point cloud frames of the working environment, and according to the motion estimation results, splice the collected discrete color semantic point cloud frames into a complete color semantic three-dimensional point cloud model.

[0089] In this embodiment, Figure 8 As shown in the figure, during the operation of the gantry crane, the laser radar collects point cloud information of the ship and the port environment. First, the point cloud distortion caused by the movement of the gantry crane is corrected by the IMU sensor data, and then the state of the point cloud frame is calculated by the error iteration Kalman filter algorithm. After iterative optimization, the optimal estimate of the current point cloud frame is obtained. In the semantic mapping module, the laser point cloud frame and the semantic image are first synchronized in time, the corresponding semantic and image data are matched, and the motion error between the semantic and image data is compensated by spherical interpolation. Then, the semantic label is mapped to the point cloud data through coordinate transformation and interpolation calculation. Finally, in the modeling module, the discrete semantic point cloud frame is constructed into a complete bulk carrier point cloud semantic model.

[0090] Step 5: Use the results of image semantic segmentation to assist the optical flow method in motion detection. By introducing the precise image semantic segmentation results provided by the semantic segmentation module into the optical flow calculation, the optical flow method can be optimized according to the segmented area information when analyzing video data, effectively improving the accuracy of motion detection.

[0091] Step 5.1: The region classification module receives the result of image semantic segmentation, analyzes the semantic labels of each segmented region, and classifies these regions into "reference" regions and "test" regions; it uses the temporal information or background knowledge of the image to determine which regions may have motion. The reference region is usually the background or a stationary object, while the "test" region is the region where motion may occur, such as pedestrians, vehicles, etc. Fig. 9 As shown in the figure, the motion detection module receives the enhanced bulk port image and semantic image. Since the gantry crane with sensors is moving along the shore and the hatch of the bulk carrier is opened and closed during operation, the hull part and the shore in the image are defined as the "area to be detected" by combining the semantic information, and the other parts of the bulk carrier and the water surface are defined as the "reference area".

[0092] Step 5.2: Use the optical flow algorithm to calculate the optical flow between two adjacent frames of images, obtain the motion vector of each pixel, process multiple frames of image data, and accumulate optical flow information; introduce an adaptive weight adjustment mechanism, that is, dynamically adjust the weight according to the optical flow stability and regional importance of each region. Specifically, the stability of the optical flow is evaluated by calculating the variance of the regional optical flow vector, and the variance of the optical flow vector in each region is calculated. The smaller the variance, the smoother the movement in the region, the smaller the change, and it is suitable for optical flow tracking. A higher variance usually means that there are drastic changes or noise in the movement in the region, which is not suitable for tracking. The importance of the region sets the weight according to the semantic label, and uses the results of semantic segmentation to assign different importance weights to the region. Key areas are usually more important and need to be tracked first; the weight of the background or secondary objects is lower and can be ignored. Finally, the comprehensive weight of each region is calculated by combining the optical flow stability and regional importance. The formula is as follows:

[0093]

[0094] Among them, W i is the final weight of region i, S i is the variance of the optical flow vector in region i, I i is the semantic label weight of region i, and α and β are balance coefficients. Set a threshold for the comprehensive weight and only select regions whose comprehensive weight exceeds the threshold for optical flow tracking. This ensures that the selected region is both stable in motion and highly important in semantics.

[0095] Step 5.3: Initialize the key points of optical flow tracking in the selected stable and important area. The selection of key points can use feature point detection methods (such as Shi-Tomasi or Harris corner detection) and use classic optical flow tracking algorithms (such as Lucas-Kanade optical flow method) to track the key points in the selected stable area. Fig.10As shown, the "area to be tested" is processed preferentially through an adaptive adjustment mechanism. The optical flow calculation module selects the feature points of the "area to be tested" and calculates the motion vector by accumulating multiple frames. Finally, the module realizes accurate tracking and updating of the position and speed of the moving area of ​​the bulk carrier. In other embodiments, if you want to compare the relative motion of two areas, you only need to define them as a "reference area" and a "area to be tested" respectively, and then select feature points respectively to calculate the relative motion vector of the two.

[0096] Step 5.4: Perform statistical analysis on the optical flow vectors of the tracked reference area and the area to be measured, and calculate their average motion vector to determine the relative motion between the two. The specific steps are as follows:

[0097] Step 5.41: Calculation of the average motion vector of the reference area (Avg Static Motion):

[0098] The reference area is usually defined as a static area, that is, the objects in this area have relatively little or no movement in the scene. For example, the ground, background, or fixed objects can be used as reference areas. In order to evaluate the motion of the reference area, it is necessary to count the optical flow vectors in this area and calculate its average motion vector:

[0099]

[0100] in:

[0101] V static,i Represents the optical flow vector of the i-th pixel in the reference area;

[0102] N is the number of all pixels in the reference area.

[0103] The Avg_Static_Motion obtained by this formula is used to represent the overall motion trend of the reference area. Ideally, this value is close to zero (meaning there is no motion or very little motion in the area).

[0104] Step 5.42: Calculation of the average motion vector of the area to be measured (Avg MotionMotion)

[0105] The area to be measured is generally the area that you want to analyze and track, and the objects in this area may be in motion. Similarly, the optical flow vectors of the area to be measured are counted, and the average motion vector of the area is calculated:

[0106]

[0107] in:

[0108] V motion,j Represents the optical flow vector of the jth pixel in the area to be tested;

[0109] M is the number of pixels in the area to be tested.

[0110] This average motion vector Avg_Motion_Motion can reflect the motion characteristics of the object in the area to be measured.

[0111] Step 5.43: Calculation of relative motion vector (Relative Motion)

[0112] In order to determine the motion of the area to be tested relative to the reference area, the average motion vectors of the two are compared. Specifically, the calculation formula for the relative motion vector is:

[0113] Relative_Motion=Avg_Motion_Motion-Avg_Static_Motion

[0114] in:

[0115] Avg_Motion_Motion is the average motion vector of the area to be measured;

[0116] Avg_Static_Motion is the average motion vector of the reference area.

[0117] Step 5.44: Motion status determination

[0118] The relative motion vector Relative_Motion can be calculated to determine the motion state of the area to be tested relative to the reference area. Based on the results of relative motion, there are the following situations:

[0119] (1) No relative motion (stationary state): When Relative_Motion≈0, it means that there is almost no motion difference between the test area and the reference area, that is, the object in the test area is stationary or in a slight motion state. This means that both the test area and the reference area remain relatively stationary, and it can be considered that there is no significant motion in the test area.

[0120] (2) Consistent Motion (Synchronous Motion) When Avg_Motion_Motion ≈ Avg_Static_Motion and both are not zero, it means that the motion direction and amplitude of the test area and the reference area are similar, indicating that the object in the test area moves with the reference area, which may be the overall movement of the camera or the global motion in the environment.

[0121] (3) Relative motion (motion difference): When Relative_Motion is significantly greater than zero, it means that the object in the test area is moving relative to the reference area, and the direction or speed of movement is significantly different. In this case, it can be inferred that the objects in the test area are moving independently (such as a moving vehicle or a moving object).

[0122] (4) Severe motion or noise: If Relative_Motion is significantly greater than the expected threshold and the variance of the optical flow vector in the area to be measured is high, it may indicate that there is severe motion or noise interference in the area to be measured. At this time, you should consider re-evaluating the area and may need to discard the tracking of the area.

[0123] Re-evaluation: During the optical flow tracking process, the optical flow stability and importance of the area are re-evaluated regularly to ensure that the tracked area still meets the stability and importance requirements, such as readjusting every fixed number of frames. If the optical flow of some areas changes dramatically or loses its importance, consider abandoning the area and reselecting a new area.

[0124] The embodiments described above are merely descriptions of preferred implementation modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multi-sensor fusion all-weather operation target perception and measurement system, including a data acquisition module composed of a visual sensor, a laser radar and an IMU sensor, characterized in that: The data acquisition module is followed by an all-weather image enhancement module, a three-dimensional modeling and semantic segmentation module, and a motion detection module that are interconnected; The all-weather image enhancement module includes a weather recognition module, a gating network, a processing module and a weighted fusion module which are connected in sequence, wherein the processing module includes a defogging module, a low-light enhancement module, a rain removal module and a snow removal module; the visual sensor is connected to the weather recognition module in communication; The 3D modeling and semantic segmentation module includes an interconnected image semantic segmentation network, a mileage calculation module, a semantic mapping module, and a mapping module; the laser radar, IMU sensor, and mileage module are connected in communication; the weighted fusion module and the image semantic segmentation network are connected in communication; The motion detection module includes a region classification module, an optical flow calculation module, a region motion extraction module and a motion detection result module which are connected in sequence. The weighted fusion module and the image semantic segmentation network are both connected to the region classification module in communication.

2. The multi-sensor fusion all-weather target perception and measurement system according to claim 1, characterized in that: The image semantic segmentation network adopts an encoding-decoding structure. The encoder part is an STDC network responsible for gradually extracting multi-scale feature information. The last layer of the encoder introduces a pyramid strip pooling module PSPM. The PSPM module first extracts local features in the horizontal and vertical directions through strip pooling pairs with kernel sizes of 1×1, 2×2, and 4×4. The pooled multi-scale feature map is restored to the size of the original feature map through convolution and upsampling. The convolution kernel size is 1×1, and then the features with the same pooling kernel size are spliced ​​together. The spliced ​​feature maps are further fused through the convolution layer. Finally, the fused multi-scale features are added and the features are finely adjusted through the convolution layer to integrate information from different scales and generate the final output features. The decoder part adopts the FLD structure, which is used to gradually fuse the multi-level features extracted by the encoder and generate the final output image. The FLD consists of two dual attention fusion modules DAFM and a segmentation head. The input features of DAFM are composed of the shallow features F-low in the encoding stage and the deep features F-high in the decoding stage. F-high is upsampled to F-up through bilinear interpolation, which has the same size as F-low. Then the two features are concatenated and the concatenated features are input into the effective channel attention ECA and cross attention CCA modules in turn. Finally, the output fusion feature F-output is obtained through convolution operation. In the segmentation head, convolution, bn, and relu operations are performed to reduce the channels of the features to the number of classes. Then an upsampling operation is performed to expand the feature size to the input image size, and an argmax operation is performed to predict the label of each pixel.

3. A multi-sensor fusion all-weather operation target perception measurement method, based on the measurement system of claim 1 or 2, characterized in that: The specific steps are as follows: Step 1: Collect data on the operation target; Step 2: Perform all-weather image enhancement processing on the collected image data; Step 3: Perform image semantic segmentation on the enhanced image; Step 4: Use the results of image semantic segmentation combined with laser point cloud and IMU data to perform point cloud semantic modeling; Step 5: Use the results of image semantic segmentation to assist in optical flow calculation and achieve motion detection.

4. The multi-sensor fusion all-weather operation target perception measurement method according to claim 3, characterized in that: Step 2 The specific steps are as follows: Step 2.1: Use the image data collected by the visual sensor under different weather and lighting conditions, that is, image data under five weather conditions: sunny, rainy, snowy, foggy, and low-light, to construct an image classification dataset, and use the dataset to train the weather recognition module using a deep learning network to obtain a trained weather recognition module; Step 2.2: Input the image data collected by the visual sensor into the weather recognition module for weather recognition, and output the one-hot encoding vector representing the weather category to the gating network and each processing module; Step 2.3: The gated network generates the weights of each processing module in real time based on the input image feature vector based on a dynamic adjustment mechanism, and each processing module processes the image accordingly according to the weights; Step 2.4: The outputs of each processing module are effectively combined through weighted averaging to generate a higher quality final image.

5. The multi-sensor fusion all-weather operation target perception measurement method according to claim 3, characterized in that: The specific steps of step 3 are: input the enhanced image into the image semantic segmentation network, extract semantic information, identify and segment different scenes and targets, and obtain a semantic label map, which contains the category information of various objects in the scene.

6. The multi-sensor fusion all-weather operation target perception measurement method according to claim 3, characterized in that: Step 4 The specific steps are as follows: Step 4.1: Accumulate the discrete laser points collected by the laser radar into a point cloud frame format, and extract the spatial geometric information of the work scene from the point cloud data collected by the laser radar; Step 4.2: Calculate the optimal estimate of the sensor motion between point cloud frames based on the error iterative Kalman filter algorithm, obtain the corrected laser point cloud world coordinates, and transfer the point cloud frames to a unified global map; Step 4.3: 3D semantic point cloud reconstruction, combining the extracted visual semantic information with the point cloud data generated by the lidar, and reconstructing the 3D semantic point cloud model of the work target by mapping the semantic labels to the point cloud data; Step 4.4: voxel down-sample the laser point cloud frame, use the pre-calibrated laser radar and camera extrinsic RT matrix to perform coordinate transformation, convert the three-dimensional coordinate value of the laser point into the spatial coordinate value under the camera normalized coordinate system, and then calculate the corresponding two-dimensional pixel coordinate value; Step 4.5: After obtaining the pixel coordinates corresponding to the laser point, interpolation calculation is performed on the coordinates to obtain the semantic information of the laser point in the image frame; Step 4.6: According to the overlapping area of ​​the laser radar and camera FOV, filter out the laser points outside the field of view, assign the calculated semantic information to each laser point in the camera field of view, and use the fused semantic information to perform point cloud filtering, retaining only the point cloud model of the work target. As the laser radar moves, continuously obtain the three-dimensional color semantic point cloud frames of the working environment, and according to the motion estimation results, splice the collected discrete color semantic point cloud frames into a complete color semantic three-dimensional point cloud model.

7. The multi-sensor fusion all-weather operation target perception measurement method according to claim 3, characterized in that: Step 5 The specific steps are as follows: Step 5.1: The region classification module receives the result of image semantic segmentation, analyzes the semantic labels of each segmented region, and classifies these regions into "reference" regions and "test" regions; Step 5.2: Use the optical flow algorithm to calculate the optical flow between two adjacent frames of images, obtain the motion vector of each pixel, process multiple frames of image data, and accumulate optical flow information; Step 5.3: Initialize the key points of optical flow tracking in the selected area and track the key points in the selected area; Step 5.4: Perform statistical analysis on the optical flow vectors of the tracked reference area and the area to be measured, and calculate their average motion vector to determine the relative motion between the two.

8. The multi-sensor fusion all-weather operation target perception measurement method according to claim 7, characterized in that: After step 5.2, an adaptive weight adjustment mechanism is designed to dynamically adjust the weight according to the optical flow stability and regional importance of each region. The optical flow stability is evaluated by calculating the variance of the regional optical flow vector, and the regional importance is set according to the semantic label. The results of semantic segmentation are used to assign different importance weights to the regions. Combining the optical flow stability and regional importance, the comprehensive weight of each region is calculated. The formula is as follows: Among them, W i is the final weight of region i, S i is the variance of the optical flow vector in region i, I i is the semantic label weight of region i, α and β are balance coefficients, a threshold of the comprehensive weight is set, and only regions whose comprehensive weight exceeds the threshold are selected for optical flow tracking. This ensures that the selected regions are both stable in motion and highly important in semantics.

9. The multi-sensor fusion all-weather operation target perception and measurement method according to claim 7, characterized in that: Step 5.4 The specific steps are as follows: Step 5.41: Calculate the average motion vector of the reference area. Perform statistics on the optical flow vectors in the reference area and calculate its average motion vector: in: V static,i Represents the optical flow vector of the i-th pixel in the reference area; N is the number of all pixels in the reference area; Avg_Static_Motion obtained by this formula is used to represent the overall motion trend of the reference area; Step 5.4.2: Calculate the average motion vector of the area to be tested. Perform statistics on the optical flow vectors of the area to be tested and calculate the average motion vector of the area: in: V motion,j Represents the optical flow vector of the jth pixel in the area to be tested; M is the number of pixels in the area to be tested; this average motion vector Avg_Motion_Motion can reflect the motion characteristics of the object in the area to be tested; Step 5.4.3: Calculation of relative motion vector. In order to determine the motion of the tested area relative to the reference area, the average motion vectors of the two are compared. The calculation formula of the relative motion vector is: Relative_Motion=Avg_Motion_Motion-Avg_Static_Motion in: Avg_Motion_Motion is the average motion vector of the area to be tested; Avg_Static_Motion is the average motion vector of the reference area; Step 5.4.4: Based on the calculated relative motion vector Relative_Motion, determine the motion state of the area to be tested relative to the reference area.

Images