Unmanned vibrating roller sensing method, device, system and storage medium

Abstract

The present invention relates to the field of unmanned vibratory roller perception technology, and specifically discloses an unmanned vibratory roller perception method, device, system and storage medium, including: obtaining multi-source sensor collected data information, the multi-source sensor collected data information including at least image data and point cloud data, the image data including at least RGB images and thermal imaging images; pre-processing the multi-source sensor collected data information to obtain multi-source sensor pre-processed data; performing feature extraction on the multi-source sensor pre-processed data to obtain multi-source sensor feature data; performing feature fusion on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data; performing feature detection on the multi-source sensor feature fusion data according to a target detection and classification model to obtain a target perception result. The unmanned vibratory roller perception method provided by the present invention can reduce the complexity of the detection network required for perception while improving perception accuracy.

Description

Technical Field

[0001] The present invention relates to the field of unmanned vibrating roller sensing technology, and in particular to an unmanned vibrating roller sensing method, an unmanned vibrating roller sensing device, an unmanned vibrating roller sensing system and a storage medium. Background Art

[0002] With the rapid development of infrastructure construction, the demand for construction automation continues to increase. As a key component of intelligent construction equipment, unmanned vibratory rollers are increasingly maturing their sensing systems. Accurate environmental perception is fundamental to the efficient operation of unmanned rollers. This high-precision environmental perception system not only accurately detects various types of obstacles (such as personnel, construction vehicles, and rocks) within the operating area, improving the efficiency of subsequent system decision-making, but also promptly detects and reports potential safety hazards such as obstacle intrusions, ensuring the efficient and safe operation of unmanned vibratory rollers.

[0003] The perception system for unmanned vibratory rollers relies on various sensors installed on the rollers, such as visual cameras, millimeter-wave radars, and lidars, to monitor the construction environment and rolling operation status in real time, thereby providing intuitive data feedback on the actual conditions at the construction site. However, due to the complexity and variability of the construction environment, the perception system exhibits extremely high uncertainty in both time and space. For example, changes in environmental conditions (weather, light, temperature) and ground conditions at the construction site pose significant challenges to accurate perception and decision-making. Therefore, the perception system for unmanned vibratory rollers typically employs advanced methods that can effectively capture the dynamic characteristics of the environment. These can be further divided into traditional perception methods based on clustering and segmentation, enhanced perception methods based on machine learning, and intelligent perception methods based on deep learning.

[0004] Traditional methods primarily rely on physical sensors such as lidar, ultrasonic sensors, and cameras. These technologies rely on direct sampling of environmental information and clustering and segmentation algorithms to provide highly accurate distance measurement and object detection. However, these methods are susceptible to environmental noise in complex scenarios, and their adaptability and robustness are often insufficient in inclement weather or complex terrain.

[0005] Machine learning-based enhanced perception methods leverage historical data for pattern recognition and prediction, enabling more intelligent analysis of environmental states and target behaviors. Through feature engineering and data-driven learning models, these methods have improved the robustness of traditional approaches and enhanced perception capabilities in unstructured environments. However, their performance depends on the quality and diversity of training data and has limitations when dealing with unknown environments.

[0006] Intelligent perception methods based on deep learning use models such as Convolutional Neural Networks (CNN) and Graph Neural Networks (GNN) to directly extract complex features from high-dimensional data such as images and point clouds, and perform real-time environmental understanding and decision-making. Compared with traditional methods and machine learning methods, deep learning technology has greater advantages in multimodal data fusion and adaptability to environmental changes, and can significantly improve the accuracy and robustness of perception systems in dynamic environments. However, its high computing requirements also place higher demands on the system's hardware performance and real-time performance.

[0007] Therefore, how to reduce the complexity of the detection network required for perception while improving the perception accuracy has become a technical problem that needs to be urgently solved by those skilled in the art. Summary of the Invention

[0008] The present invention provides an unmanned vibratory roller sensing method, an unmanned vibratory roller sensing device, an unmanned vibratory roller sensing system and a storage medium, which solve the problems of complex detection network required for perception and low perception accuracy existing in related technologies.

[0009] As a first aspect of the present invention, there is provided an unmanned vibrating roller sensing method, which comprises:

[0010] Acquire data information collected by a multi-source sensor, wherein the data information collected by the multi-source sensor includes at least image data and point cloud data, and the image data includes at least RGB images and thermal imaging images;

[0011] Preprocessing the data information collected by the multi-source sensors to obtain multi-source sensor preprocessed data;

[0012] Performing feature extraction on the multi-source sensor preprocessed data to obtain multi-source sensor feature data;

[0013] Performing feature fusion on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data;

[0014] The multi-source sensor feature fusion data is subjected to feature detection according to the target detection and classification model to obtain target perception results.

[0015] Furthermore, preprocessing the data information collected by the multi-source sensors to obtain multi-source sensor preprocessed data includes:

[0016] The image data and point cloud data are both subjected to denoising, normalization, and time synchronization processing to obtain the positional relationship of the multi-source sensor relative to the world coordinate system.

[0017] Furthermore, the image data and the point cloud data are subjected to denoising, normalization and time synchronization processing, including:

[0018] removing noise from the RGB image using a Gaussian filter method, and normalizing the color of the RGB image after noise removal using a contrast enhancement method and a white balance adjustment method to obtain an RGB normalized processed image;

[0019] removing temperature noise from the thermal imaging image according to an adaptive threshold method, and normalizing the temperature values in the thermal imaging image from which the temperature noise has been removed to obtain a normalized thermal imaging image;

[0020] removing outliers and misdetected points from the point cloud data according to a RANSAC algorithm to obtain point cloud denoised data;

[0021] The point cloud denoised data, the RGB normalized processed image and the thermal imaging normalized image are spatially aligned to obtain the point cloud denoised data, the RGB normalized processed image and the thermal imaging normalized image in the same reference coordinate system.

[0022] Furthermore, feature extraction is performed on the multi-source sensor preprocessed data to obtain multi-source sensor feature data, including:

[0023] Performing feature extraction on the image data according to a lightweight CNN to obtain image feature data;

[0024] Feature extraction is performed on the point cloud data according to PointNet to obtain point cloud feature data.

[0025] Furthermore, feature extraction is performed on the image data according to a lightweight CNN to obtain image feature data, including:

[0026] Perform a depthwise separable convolution operation on the image data to obtain convolution operation feature data, wherein the depthwise separable convolution operation includes depthwise convolution and pointwise convolution, and the expression of the depthwise convolution is:

[0027]

[0028]

[0029] in, Represents the output feature of the cth channel at position (x, y), represents the depth convolution kernel of the lth layer, and I(x+i,y+j,c) represents the value of the cth channel of the image data; represents the eigenvalue of the point-by-point convolution output, represents the weight of 1×1 convolution, C in Represents the input channel tree;

[0030] The convolution operation feature data is processed according to the ReLU activation function to obtain image feature data.

[0031] Furthermore, feature extraction is performed on the point cloud data according to PointNet to obtain point cloud feature data, including:

[0032] The point cloud data is subjected to feature extraction according to PointNet, wherein the feature extraction expression of PointNet is:

[0033] f(X)=max i∈[1,N] (φ(x i )),

[0034] Among them, φ represents the feature learning function of each point cloud, φ(x i )=W2·σ(W1·x i +b1)+b2, σ represents the activation function, W1 and W2 both represent the weight matrices of the feature learning function, b1 and b2 both represent bias terms, X represents the point cloud data, and the max operation represents the extraction of global features from the features of all point clouds.

[0035] Furthermore, feature fusion is performed on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data, including:

[0036] Converting the feature data of the RGB image, the feature data of the thermal imaging image, and the feature data of the point cloud data into feature representations of the same dimension;

[0037] Calculate a channel descriptor based on feature data of an RGB image, feature data of a thermal image, and feature data of point cloud data of the same dimension;

[0038] Calculate the attention weights of the channel descriptors of multi-source sensors according to the Softmax function;

[0039] Calculating adaptive weights of channel descriptors of multi-source sensors according to an adaptive weight allocation mechanism;

[0040] The feature fusion of multi-source sensors is performed according to their channel descriptors and their corresponding attention weights and adaptive weights to obtain multi-source sensor feature fusion data.

[0041] As another aspect of the present invention, an unmanned vibrating roller sensing device is provided for implementing the unmanned vibrating roller sensing method described above, comprising:

[0042] An acquisition data acquisition module is used to acquire data information collected by multi-source sensors, wherein the data information collected by the multi-source sensors includes at least image data and point cloud data, and the image data includes at least RGB images and thermal imaging images;

[0043] A preprocessing module, configured to preprocess the data information collected by the multi-source sensors to obtain multi-source sensor preprocessed data;

[0044] A feature extraction module is used to extract features from the multi-source sensor preprocessed data to obtain multi-source sensor feature data;

[0045] A feature fusion module is used to perform feature fusion on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data;

[0046] The target detection and classification module is used to perform feature detection on the multi-source sensor feature fusion data according to the target detection and classification model to obtain target perception results.

[0047] As another aspect of the present invention, an unmanned vibrating roller sensing system is provided, comprising: an RGB camera device, a thermal imaging device, a millimeter-wave radar device, and the unmanned vibrating roller sensing device described above, wherein the RGB camera device, the thermal imaging device, and the millimeter-wave radar device are all communicatively connected to the unmanned vibrating roller sensing device;

[0048] The RGB camera device is used to collect RGB images;

[0049] The thermal imaging device is used to collect thermal imaging images;

[0050] The millimeter wave radar device is used to collect point cloud data;

[0051] The unmanned vibrating roller sensing device is used to perform feature detection after preprocessing, feature extraction and feature fusion based on RGB images, thermal imaging images and point cloud data to obtain target perception results.

[0052] As another aspect of the present invention, a storage medium is provided, which is used to store computer instructions. When the computer instructions are loaded and executed by a processor, the unmanned vibration roller sensing method described above is implemented.

[0053] The unmanned vibratory roller perception method provided by the present invention obtains environmental information around the vibratory roller through sensors such as RGB cameras, thermal imaging devices, and millimeter-wave radars, and uses this to construct data samples to provide input for subsequent algorithms; preprocesses the acquired multi-source data, including image denoising, data normalization, and time synchronization, to ensure high data robustness and spatiotemporal consistency of data from different sensors; uses lightweight CNN and PointNet to extract features from the collected image data and point cloud data, respectively; uses a channel attention mechanism to fuse features of the extracted image and point cloud features; and uses a simplified Fast ParC-Net model architecture to detect and classify targets on the fused data. This unmanned vibratory roller perception method can not only reduce sensor redundancy and optimize system costs, but also has excellent stability and performance in severe weather conditions; the use of a fusion network based on a channel attention mechanism enables the model to automatically learn the importance of data from different modalities and highlight the features that have the greatest impact on the perception results. Therefore, the unmanned vibratory roller perception method provided by the present invention can not only be easily and quickly deployed on devices with limited computing resources, but also exhibits excellent recognition capabilities in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0055] Figure 1 This is a flow chart of the unmanned vibrating roller sensing method provided by the present invention.

[0056] Figure 2 This is a flow chart of the preprocessing method provided by the present invention.

[0057] Figure 3 This is a flow chart of the feature extraction method provided by the present invention.

[0058] Figure 4 Flowchart of the feature fusion method provided by the present invention.

[0059] Figure 5 This is a structural block diagram of the unmanned vibrating roller sensing device provided by the present invention.

[0060] Figure 6 This is a structural block diagram of the unmanned vibrating roller sensing system provided by the present invention.

[0061] Figure 7 This is a structural block diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0062] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0063] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0064] It should be noted that the terms "first," "second," and the like in the specification and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate for the embodiments of the present invention described herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0065] In this embodiment, an unmanned vibrating roller sensing method is provided. Figure 1 FIG. 1 is a flow chart of an unmanned vibrating roller sensing method according to an embodiment of the present invention. Figure 1 As shown, including:

[0066] S100: Acquire data information collected by multi-source sensors, where the data information collected by the multi-source sensors includes at least image data and point cloud data, and the image data includes at least RGB images and thermal imaging images;

[0067] In an embodiment of the present invention, the multi-source sensor may specifically include an RGB camera, a thermal imaging device, and a millimeter-wave radar device. The RGB camera is used to capture RGB images, the thermal imaging device is used to capture thermal images, and the millimeter-wave radar device is used to capture point cloud data. Therefore, the data collected by the multi-source sensor includes at least RGB images, thermal images, and point cloud data.

[0068] It should be noted that in the embodiment of the present invention, an RGB camera device is used to capture visible light images, and the thermal imaging images collected by the thermal imaging device supplement the environmental perception capability in low light or inclement weather. This fusion of multispectral data enhances the visual recognition capability of the perception system in diverse environments. At the same time, combined with millimeter-wave radar point cloud data, the long-range detection capability of the perception system is achieved. Therefore, the embodiment of the present invention acquires environmental information around the vibratory roller through the arrangement of multi-source sensors, which not only reduces sensor redundancy and optimizes system costs, but also has excellent stability and performance in inclement weather.

[0069] S200, preprocessing the data information collected by the multi-source sensors to obtain multi-source sensor preprocessed data;

[0070] In an embodiment of the present invention, preprocessing is performed on RGB images, thermal images, and point cloud data, including image denoising, data normalization, and time synchronization, to ensure high data robustness and spatiotemporal consistency of data from different sensors.

[0071] S300, performing feature extraction on the multi-source sensor preprocessed data to obtain multi-source sensor feature data;

[0072] In an embodiment of the present invention, lightweight convolutional neural networks (CNN) are used to extract features from RGB image and thermal imaging image data; and point cloud processing networks (PointNet) are used to extract spatial features from millimeter wave radar data.

[0073] S400, performing feature fusion on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data;

[0074] Specifically, in the feature fusion stage, features from different sources are aligned and fused, and an attention mechanism is used to enhance important features while suppressing noise and other unimportant background information.

[0075] S500: Perform feature detection on the multi-source sensor feature fusion data according to a target detection and classification model to obtain a target perception result.

[0076] In an embodiment of the present invention, a simplified Fast ParC-Net model architecture is used for target detection and classification. The network combines the Position Aware Circular Convolution (ParC) operation with the Squeeze-and-Excitation module to improve the expressiveness of the model and reduce the computational complexity of the model.

[0077] Therefore, the unmanned vibratory roller perception method provided by the present invention obtains environmental information around the vibratory roller through sensors such as RGB cameras, thermal imaging devices, and millimeter-wave radars, and uses this to construct data samples to provide input for subsequent algorithms; preprocesses the acquired multi-source data, including image denoising, data normalization, and time synchronization, to ensure high robustness of the data and spatiotemporal consistency of data from different sensors; uses lightweight CNN and PointNet to extract features from the collected image data and point cloud data respectively; uses a channel attention mechanism to perform feature fusion on the extracted image and point cloud features; and uses a simplified Fast ParC-Net model architecture to perform target detection and classification on the fused data. This unmanned vibratory roller perception method can not only reduce sensor redundancy and optimize system costs, but also has excellent stability and performance in severe weather conditions; the use of a fusion network based on a channel attention mechanism enables the model to automatically learn the importance of different modal data and highlight the features that have the greatest impact on the perception results. Therefore, the unmanned vibratory roller perception method provided by the present invention can not only be easily and quickly deployed on devices with limited computing resources, but also exhibits excellent recognition capabilities in complex scenarios.

[0078] In an embodiment of the present invention, preprocessing the multi-source sensor collected data information to obtain multi-source sensor preprocessed data includes:

[0079] The image data and point cloud data are both subjected to denoising, normalization, and time synchronization processing to obtain the positional relationship of the multi-source sensor relative to the world coordinate system.

[0080] It should be understood that Gaussian filtering is used to remove noise from RGB images to ensure image clarity. At the same time, the image is contrast enhanced and white balance adjusted to ensure that the color of the images collected under different lighting conditions remains consistent, thereby eliminating the impact of lighting on image quality. An adaptive threshold algorithm is used to remove temperature noise from thermal imaging data to maintain the clarity of the temperature information of the target object. At the same time, the temperature values in the data are normalized to between 0 and 1 so that they can be fused with other types of sensor data. For the point cloud noise in the millimeter-wave radar data, the RANSAC (Random Sample Consensus) algorithm is used to remove outliers and false detection points to improve the accuracy of the point cloud data. The RGB image data, thermal imaging infrared image and millimeter-wave radar point cloud data are spatially aligned to obtain the relative position relationship of each sensor relative to the world coordinate system.

[0081] Specifically, if Figure 2 As shown, the image data and point cloud data are subjected to denoising, normalization and time synchronization processing, including:

[0082] S210, removing noise from the RGB image using a Gaussian filter method, and normalizing the color of the RGB image after noise removal using a contrast enhancement method and a white balance adjustment method to obtain an RGB normalized processed image;

[0083] Specifically, for RGB images, Gaussian filtering is used to remove image noise, and contrast enhancement and white balance adjustment are used to normalize image colors to ensure that images collected under different lighting conditions can maintain consistent color performance and eliminate the impact of lighting on image quality. The expression of Gaussian filtering is:

[0084]

[0085] The expression for color normalization is:

[0086]

[0087] Among them, I(x,y) represents the RGB image to be processed, σ represents the standard deviation of the Gaussian kernel, k represents the window size of the filter, and I min and I max Represent the minimum pixel value and the maximum pixel value of the image respectively.

[0088] S220, removing temperature noise from the thermal imaging image according to an adaptive threshold method, and normalizing the temperature values in the thermal imaging image after removing the temperature noise to obtain a normalized thermal imaging image;

[0089] In this embodiment of the present invention, an adaptive threshold algorithm is used to remove temperature noise from thermal images, maintaining the clarity of the target object's temperature information. The temperature values in the thermal images are normalized to between 0 and 1 to facilitate fusion with other types of sensor data. The expressions for temperature noise removal and normalization are:

[0090]

[0091] Where T represents the threshold value that is adaptively adjusted according to the local characteristics of the image, T(x,y) represents the temperature value in the thermal imaging image, and T min and T max Represent the minimum and maximum temperature values respectively.

[0092] S230, removing outliers and misdetected points in the point cloud data according to the RANSAC algorithm to obtain point cloud denoised data;

[0093] In an embodiment of the present invention, the RANSAC algorithm is used to remove outliers and false detection points from millimeter-wave radar point cloud data to improve the accuracy of the point cloud data. The expression for denoising is shown in the following formula:

[0094]

[0095] Where d represents the distance from the point cloud to the plane. If d exceeds a certain threshold, the point cloud is considered an outlier.

[0096] S240: spatially aligning the point cloud denoised data, the RGB normalized processed image, and the thermal imaging normalized image to obtain the point cloud denoised data, the RGB normalized processed image, and the thermal imaging normalized image in the same reference coordinate system.

[0097] Specifically, the point cloud data of the millimeter-wave radar is spatially aligned with the data of the RGB camera and the thermal imaging camera to ensure that the data collected by all sensors are in the same reference coordinate system. The expression of spatial configuration is:

[0098]

[0099] Where t represents the translation matrix and K represents the camera matrix.

[0100] In an embodiment of the present invention, feature extraction is performed on the multi-source sensor preprocessing data to obtain multi-source sensor feature data, such as Figure 3 Shown, including:

[0101] S310, performing feature extraction on the image data according to a lightweight CNN to obtain image feature data;

[0102] A lightweight CNN is used to extract features from image data. First, a depth-wise separable convolution operation is performed on the input image to extract image features, where the depth-wise separable convolution can be decomposed into depth-wise convolution and point-wise convolution.

[0103] Specifically, feature extraction is performed on the image data according to the lightweight CNN to obtain image feature data, including:

[0104] Perform a depthwise separable convolution operation on the image data to obtain convolution operation feature data, wherein the depthwise separable convolution operation includes depthwise convolution and pointwise convolution, and the expression of the depthwise convolution is:

[0105]

[0106] Among them, F l depthwise (x,y,c) represents the output feature of the cth channel at position (x,y), represents the depth convolution kernel of the lth layer, I(x+i,y+j,c) represents the value of the cth channel of the image data; F l pointwise (x, y, c) represents the eigenvalues of the point-by-point convolution output, represents the weight of 1×1 convolution, C in Represents the input channel tree;

[0107] The convolution operation feature data is processed according to the ReLU activation function to obtain image feature data.

[0108] It should be understood that after the convolution operation, the ReLU activation function is used to introduce a nonlinear layer, and global average pooling is used instead of the traditional fully connected layer during downsampling to further reduce the number of parameters.

[0109] S320 , performing feature extraction on the point cloud data according to PointNet to obtain point cloud feature data.

[0110] In an embodiment of the present invention, feature extraction is performed on the point cloud data according to PointNet to obtain point cloud feature data, including:

[0111] The point cloud data is subjected to feature extraction according to PointNet, wherein the feature extraction expression of PointNet is:

[0112] f(X)=max i∈[1,N] (φ(x i )),

[0113] Among them, φ represents the feature learning function of each point cloud, φ(x i )=W2·σ'(W1·x i +b1)+b2, σ' represents an activation function, W1 and W2 both represent weight matrices of the feature learning function, b1 and b2 both represent bias terms, X represents the point cloud data, and the max operation represents extracting global features from the features of all point clouds.

[0114] In order to fuse the image features and point cloud features to improve the accuracy and robustness of the perception system, in an embodiment of the present invention, the multi-source sensor feature data is subjected to feature fusion according to the channel attention mechanism to obtain multi-source sensor feature fusion data, such as Figure 4 Shown, including:

[0115] S410, converting the feature data of the RGB image, the feature data of the thermal imaging image, and the feature data of the point cloud data into feature representations of the same dimension;

[0116] In an embodiment of the present invention, feature alignment is performed on data from different sensors, and image, thermal imaging, and point cloud data are converted into feature representations of the same dimension.

[0117] S420, calculating a channel descriptor based on feature data of the RGB image, feature data of the thermal imaging image, and feature data of the point cloud data of the same dimension;

[0118] Specifically, the calculation formula of the channel descriptor is:

[0119] z=σ'(W1′·F RGB )+σ′(W2′·F IR )+σ′(W3′·F point ),

[0120] Among them, F RGB Represents the feature data of RGB image, F IR Represents the characteristic data of thermal imaging image, F point represents the feature data of the point cloud data, W1′, W2′ and W3′ all represent learnable weight matrices, and σ′ represents the activation function.

[0121] S430, calculating the attention weights of the channel descriptors of the multi-source sensors according to the Softmax function;

[0122] In this embodiment of the present invention, the Softmax function is used to calculate the attention weight, and the calculation formula is:

[0123] a=Softmax(z).

[0124] S440, calculating adaptive weights of channel descriptors of the multi-source sensors according to an adaptive weight allocation mechanism;

[0125] The adaptive weight is calculated through the adaptive weight allocation mechanism. The expression for weight calculation is:

[0126]

[0127] β=1-α,

[0128] Among them, γ represents an adjustable parameter, z RGB Channel descriptor representing the feature data of RGB image, z IR A channel descriptor representing feature data of a thermal imaging image.

[0129] S450 , performing feature fusion of the multi-source sensors according to the channel descriptors of the multi-source sensors and their corresponding attention weights and adaptive weights to obtain multi-source sensor feature fusion data.

[0130] Specifically, the final expression of multi-source sensor feature fusion data is:

[0131] F fusion =α·(a RGB ·F RGB )+β·(a IR ·F IR )+(1-α-β)·F point .

[0132] In an embodiment of the present invention, feature detection is performed on the multi-source sensor feature fusion data according to a target detection and classification model to obtain a target perception result, including:

[0133] The ParC-Net network is used to perform target recognition and classification on the multi-source sensor feature fusion data to obtain target perception results.

[0134] For the above fusion features, the ParC-Net network is used for target recognition and classification, and the final output result is expressed as:

[0135] Output = parc-Net(F fusion ).

[0136] It should be noted that another embodiment of the multi-source sensor in the embodiment of the present invention may also include an RGB camera device, a millimeter wave radar device and a lidar device. Based on this, the target recognition and detection network can be implemented using YOLO, Vision Transformer, etc.

[0137] In summary, the unmanned vibrating roller perception method provided by the present invention uses an RGB camera device to capture visible light images, and the thermal imaging head infrared image supplements the environmental perception capability in low light or bad weather. Through the fusion of this multi-spectral data, the visual recognition capability of the perception system in diverse environments is improved. At the same time, combined with millimeter-wave radar point cloud data, the long-range detection capability of the perception system is realized. This sensor arrangement not only reduces sensor redundancy and optimizes system costs, but also has excellent stability and performance in bad weather. The use of a fusion network based on a channel attention mechanism enables the model to automatically learn the importance of different modal data and highlight the features that have the greatest impact on the perception results. At the same time, through an adaptive weight allocation mechanism, the fusion ratio of each modal feature is dynamically adjusted. For example, in low visibility conditions, the weight of the thermal imaging camera and millimeter-wave radar is increased to reduce the influence of the RGB camera, thereby improving the overall perception accuracy. The Parc-Net network is used for target recognition and classification. The network combines the advantages of convolutional neural networks in local feature extraction and the capabilities of visual transformers in global feature modeling, thereby reducing model complexity while maintaining excellent performance. Therefore, the unmanned vibratory roller perception method provided by the present invention can not only be conveniently and quickly deployed on devices with limited computing resources, but also demonstrates excellent recognition capabilities in complex scenarios.

[0138] As another embodiment of the present invention, an unmanned vibrating roller sensing device 100 is provided for implementing the unmanned vibrating roller sensing method described above, wherein: Figure 5 Shown, including:

[0139] The acquisition data acquisition module 110 is used to acquire data information collected by multi-source sensors, wherein the data information collected by the multi-source sensors includes at least image data and point cloud data, and the image data includes at least RGB images and thermal imaging images;

[0140] A preprocessing module 120 is used to preprocess the data information collected by the multi-source sensors to obtain multi-source sensor preprocessed data;

[0141] A feature extraction module 130 is configured to extract features from the multi-source sensor preprocessed data to obtain multi-source sensor feature data;

[0142] A feature fusion module 140 is configured to perform feature fusion on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data;

[0143] The target detection and classification module 150 is used to perform feature detection on the multi-source sensor feature fusion data according to the target detection and classification model to obtain a target perception result.

[0144] The unmanned vibratory roller sensing device provided by the present invention acquires environmental information around the vibratory roller through sensors such as RGB cameras, thermal imaging devices, and millimeter-wave radars, and uses this information to construct data samples to provide input for subsequent algorithms; preprocesses the acquired multi-source data, including image denoising, data normalization, and time synchronization, to ensure high data robustness and spatiotemporal consistency of data from different sensors; uses lightweight CNN and PointNet to extract features from the collected image data and point cloud data, respectively; uses a channel attention mechanism to fuse the extracted image and point cloud features; and uses a simplified Fast ParC-Net model architecture to detect and classify targets on the fused data. This unmanned vibratory roller sensing device can not only reduce sensor redundancy and optimize system costs, but also has excellent stability and performance in severe weather conditions; the use of a fusion network based on a channel attention mechanism enables the model to automatically learn the importance of different modal data and highlight the features that have the greatest impact on the perception results. Therefore, the unmanned vibratory roller sensing device provided by the present invention can not only be easily and quickly deployed on devices with limited computing resources, but also exhibits excellent recognition capabilities in complex scenarios.

[0145] Regarding the specific working principle of the unmanned vibrating roller sensing device provided by the present invention, reference can be made to the description of the unmanned vibrating roller sensing method above, which will not be repeated here.

[0146] As another embodiment of the present invention, an unmanned vibrating roller sensing system 10 is provided, wherein, as Figure 6As shown, it includes: an RGB camera device 200, a thermal imaging device 300, a millimeter wave radar device 400 and the unmanned vibrating roller sensing device 100 mentioned above, and the RGB camera device 200, the thermal imaging device 300 and the millimeter wave radar device 400 are all communicatively connected to the unmanned vibrating roller sensing device 100;

[0147] The RGB camera device 200 is used to capture RGB images;

[0148] The thermal imaging device 300 is used to collect thermal imaging images;

[0149] The millimeter wave radar device 400 is used to collect point cloud data;

[0150] The unmanned vibratory roller sensing device 100 is used to perform feature detection after preprocessing, feature extraction, and feature fusion based on RGB images, thermal imaging images, and point cloud data to obtain target perception results.

[0151] In this embodiment of the present invention, environmental information about the vibratory roller's surroundings is acquired through RGB cameras, thermal imaging devices, and millimeter-wave radar devices, and a data sample is constructed from this information. The unmanned vibratory roller sensing device 100 performs operations such as noise removal, normalization, and time alignment on the acquired data. It then extracts features from the image data and point cloud data using lightweight CNNs and PointNet, respectively. A feature fusion module based on a channel-attention mechanism is constructed, combined with an adaptive attention weight allocation mechanism to assign weights to different features and dynamically adjust the fusion ratio of each modal feature. The fused features are then used to identify and classify targets using a Parc-Net network.

[0152] Specifically, an RGB camera, a thermal imaging camera, and a millimeter-wave radar are installed simultaneously on the unmanned vibrating roller to capture visible light images and point cloud data. The fusion of multispectral data helps improve visual recognition capabilities in diverse environmental conditions. Combined with millimeter-wave radar point cloud data, the perception system achieves long-range detection capabilities, reduces sensor redundancy, optimizes system costs, and increases reliability in harsh environments. This multi-sensor data acquisition approach achieves high-precision and high-reliability perception in harsh environments.

[0153] Furthermore, a lightweight CNN and PointNet are used to extract image and point cloud features, respectively, and a fusion network based on a channel-attention mechanism is constructed to fuse these features. This enables the model to automatically learn the importance of different modal data, highlighting the features that have the greatest impact on perception. Furthermore, an adaptive weight allocation mechanism dynamically adjusts the fusion ratio of each modal feature. For example, in low-visibility conditions, the weight of thermal imaging cameras and millimeter-wave radar is increased to reduce the influence of RGB cameras, thereby improving overall perception accuracy.

[0154] Finally, the Parc-Net network is used to identify and classify input features. This network combines the advantages of convolutional neural networks in local feature extraction with the ability of visual transformers in global feature modeling. This allows for excellent performance with low complexity, making it easy and fast to deploy on devices with limited computing resources.

[0155] The specific working principle of the unmanned vibrating roller sensing system provided by the present invention can be referred to the description of the unmanned vibrating roller sensing method above, which will not be repeated here.

[0156] As another embodiment of the present invention, a storage medium is provided, which is used to store computer instructions. When the computer instructions are loaded and executed by a processor, the unmanned vibration roller sensing method described above is implemented.

[0157] In an embodiment of the present invention, a non-transitory computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions can execute the unmanned vibrating roller sensing method in any of the above-mentioned method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); the storage medium can also include a combination of the above-mentioned types of memory.

[0158] As another embodiment of the present invention, an electronic device is provided, which includes a memory and a processor, the processor is communicatively connected to the memory, the memory is used to store a computer program, and the processor is used to load and execute the computer program to implement the unmanned vibration roller sensing method described above.

[0159] like Figure 7As shown, the electronic device 70 may include: at least one processor 71, such as a CPU (Central Processing Unit), at least one communication interface 73, a memory 74, and at least one communication bus 72. The communication bus 72 is used to realize the connection and communication between these components. The communication interface 73 may include a display screen (Display), a keyboard (Keyboard), and the optional communication interface 73 may also include a standard wired interface and a wireless interface. The memory 74 may be a high-speed RAM memory (Random Access Memory, volatile random access memory) or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 74 may optionally be at least one storage device located away from the aforementioned processor 71. The memory 74 stores application programs, and the processor 71 calls the program code stored in the memory 74 to execute any of the above method steps.

[0160] The communication bus 72 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The communication bus 72 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0161] Among them, the memory 74 may include a volatile memory (English: volatile memory), such as a random-access memory (English: random-access memory, abbreviated: RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk drive (English: hard disk drive, abbreviated: HDD) or a solid-state drive (English: solid-state drive, abbreviated: SSD); the memory 74 may also include a combination of the above types of memory.

[0162] The processor 71 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP.

[0163] The processor 71 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0164] Optionally, the memory 74 is also used to store program instructions. The processor 71 can call the program instructions to implement the present invention. Figure 1 The unmanned vibrating roller sensing method shown in the embodiment.

[0165] It will be understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present invention, and the present invention is not limited thereto. Those skilled in the art will appreciate that various modifications and improvements can be made without departing from the spirit and substance of the present invention, and such modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. An unmanned vibrating roller sensing method, characterized in that: include: Acquire data information collected by a multi-source sensor, wherein the data information collected by the multi-source sensor includes at least image data and point cloud data, and the image data includes at least RGB images and thermal imaging images; Preprocessing the data information collected by the multi-source sensors to obtain multi-source sensor preprocessed data; Performing feature extraction on the multi-source sensor preprocessed data to obtain multi-source sensor feature data; Performing feature fusion on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data; Performing feature detection on the multi-source sensor feature fusion data according to a target detection and classification model to obtain a target perception result; The multi-source sensor feature data is subjected to feature fusion according to the channel attention mechanism to obtain multi-source sensor feature fusion data, including: Converting the feature data of the RGB image, the feature data of the thermal imaging image, and the feature data of the point cloud data into feature representations of the same dimension; The channel descriptor is calculated based on the feature data of the RGB image, the feature data of the thermal imaging image, and the feature data of the point cloud data of the same dimension. The calculation formula of the channel descriptor is: ， in, Represents the feature data of the RGB image, Represents the characteristic data of the thermal imaging image, Represents the feature data of point cloud data, 、 and Both represent learnable weight matrices, represents the activation function; Calculate the attention weights of the channel descriptors of multi-source sensors according to the Softmax function; The adaptive weights of the channel descriptors of the multi-source sensors are calculated according to the adaptive weight allocation mechanism. The adaptive weights of the channel descriptors of the multi-source sensors are expressed as follows: ， ， in, represents an adjustable parameter, A channel descriptor representing the feature data of an RGB image. A channel descriptor representing feature data of a thermal imaging image; The multi-source sensor feature fusion is performed according to the channel descriptors of the multi-source sensors and their corresponding attention weights and adaptive weights to obtain the multi-source sensor feature fusion data. The expression of the multi-source sensor feature fusion data is: 。 2. The unmanned vibrating roller sensing method according to claim 1, characterized in that: Preprocessing the data collected by the multi-source sensors to obtain multi-source sensor preprocessed data includes: The image data and point cloud data are both subjected to denoising, normalization, and time synchronization processing to obtain the positional relationship of the multi-source sensor relative to the world coordinate system.

3. The unmanned vibrating roller sensing method according to claim 2, characterized in that: The image data and point cloud data are both subjected to denoising, normalization, and time synchronization processing, including: removing noise from the RGB image using a Gaussian filter method, and normalizing the color of the RGB image after noise removal using a contrast enhancement method and a white balance adjustment method to obtain an RGB normalized processed image; removing temperature noise from the thermal imaging image according to an adaptive threshold method, and normalizing the temperature values in the thermal imaging image from which the temperature noise has been removed to obtain a normalized thermal imaging image; removing outliers and misdetected points from the point cloud data according to a RANSAC algorithm to obtain point cloud denoised data; The point cloud denoised data, the RGB normalized processed image and the thermal imaging normalized image are spatially aligned to obtain the point cloud denoised data, the RGB normalized processed image and the thermal imaging normalized image in the same reference coordinate system.

4. The unmanned vibrating roller sensing method according to claim 1, characterized in that: Extracting features from the multi-source sensor preprocessed data to obtain multi-source sensor feature data includes: Performing feature extraction on the image data according to a lightweight CNN to obtain image feature data; Feature extraction is performed on the point cloud data according to PointNet to obtain point cloud feature data.

5. The unmanned vibrating roller sensing method according to claim 4, characterized in that: Performing feature extraction on the image data according to a lightweight CNN to obtain image feature data includes: Perform a depthwise separable convolution operation on the image data to obtain convolution operation feature data, wherein the depthwise separable convolution operation includes depthwise convolution and pointwise convolution, and the expression of the depthwise convolution is: ， ， in, Indicates location Previous The output features of the channel, Indicates the The depth convolution kernel of the layer, Indicates the image data The value of the channel; represents the eigenvalue of the point-by-point convolution output, express The weights of the convolution, Represents the input channel tree; The convolution operation feature data is processed according to the ReLU activation function to obtain image feature data.

6. The unmanned vibrating roller sensing method according to claim 4, characterized in that: Perform feature extraction on the point cloud data according to PointNet to obtain point cloud feature data, including: The point cloud data is subjected to feature extraction according to PointNet, wherein the feature extraction expression of PointNet is: ， in, represents the feature learning function of each point cloud, , represents the activation function, and Both represent the weight matrix of the feature learning function, and Both represent bias terms, Represents the point cloud data, and the max operation represents extracting global features from the features of all point clouds.

7. An unmanned vibrating roller sensing device, used to implement the unmanned vibrating roller sensing method according to any one of claims 1 to 6, characterized in that: include: An acquisition data acquisition module is used to acquire data information collected by multi-source sensors, wherein the data information collected by the multi-source sensors includes at least image data and point cloud data, and the image data includes at least RGB images and thermal imaging images; A preprocessing module, configured to preprocess the data information collected by the multi-source sensors to obtain multi-source sensor preprocessed data; A feature extraction module is used to extract features from the multi-source sensor preprocessed data to obtain multi-source sensor feature data; A feature fusion module is used to perform feature fusion on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data; The target detection and classification module is used to perform feature detection on the multi-source sensor feature fusion data according to the target detection and classification model to obtain target perception results.

8. An unmanned vibrating roller sensing system, characterized in that: include: An RGB camera device, a thermal imaging device, a millimeter-wave radar device, and the unmanned vibrating roller sensing device according to claim 7, wherein the RGB camera device, the thermal imaging device, and the millimeter-wave radar device are all communicatively connected to the unmanned vibrating roller sensing device; The RGB camera device is used to collect RGB images; The thermal imaging device is used to collect thermal imaging images; The millimeter wave radar device is used to collect point cloud data; The unmanned vibrating roller sensing device is used to perform feature detection after preprocessing, feature extraction and feature fusion based on RGB images, thermal imaging images and point cloud data to obtain target perception results.

9. A storage medium, characterized in that: Used to store computer instructions, when the computer instructions are loaded and executed by a processor to implement the unmanned vibration roller sensing method described in any one of claims 1 to 6.

Images