Unmanned vibration roller sensing method, device and system and storage medium

Abstract

The invention relates to the technical field of unmanned vibration roller sensing, and particularly discloses an unmanned vibration roller sensing method, device and system and a storage medium, and the method comprises the steps: obtaining data information collected by a multi-source sensor, the data information collected by the multi-source sensor at least comprises image data and point cloud data, the image data at least comprises an RGB image and a thermal imaging image; preprocessing the data information acquired by the multi-source sensor to obtain preprocessed data of the multi-source sensor; performing feature extraction on the preprocessed data of the multi-source sensor to obtain feature data of the multi-source sensor; performing feature fusion on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data; and performing feature detection on the multi-source sensor feature fusion data according to a target detection and classification model to obtain a target sensing result. According to the unmanned vibration roller sensing method provided by the invention, the sensing precision can be improved while the complexity of a detection network required for sensing can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of unmanned vibratory roller sensing technology, and in particular to an unmanned vibratory roller sensing method, an unmanned vibratory roller sensing device, an unmanned vibratory roller sensing system and a storage medium. Background Art

[0002] With the rapid development of infrastructure construction, the demand for construction automation continues to increase. As an important part of intelligent construction equipment, the technology of the perception system of unmanned vibratory rollers is becoming increasingly mature. Accurate environmental perception is the basis for the efficient operation of unmanned rollers. The high-precision environmental perception system can not only accurately detect multiple types of obstacles (such as personnel, construction vehicles, stones, etc.) in the operating area, improve the efficiency of subsequent system decision-making, but also promptly discover and report obstacles that may cause safety hazards, and ensure the efficient and safe operation of unmanned vibratory rollers.

[0003] The perception system of unmanned vibratory rollers relies on various sensors installed on unmanned vibratory rollers, such as visual cameras, millimeter-wave radars, and lidars, to monitor the construction environment and rolling operation status in real time, thereby providing intuitive data feedback for the actual situation at the construction site. However, due to the complexity and variability of the construction environment, the perception system exhibits extremely high uncertainty in time and space dimensions. For example, changes in environmental conditions (weather, light, temperature) and ground conditions at the construction site pose huge challenges to accurate perception and decision-making. Therefore, the perception system of unmanned vibratory rollers usually adopts advanced methods that can effectively capture the dynamic characteristics of the environment, which can be further subdivided into traditional perception methods based on clustering segmentation, enhanced perception methods based on machine learning, and intelligent perception methods based on deep learning.

[0004] Among them, traditional methods mainly include means based on physical sensors such as lidar, ultrasonic sensors, and cameras. These technologies rely on direct sampling of environmental information and clustering segmentation algorithms to provide high-precision distance measurement and object detection. However, these methods are easily affected by environmental noise in complex scenes, and their adaptability and robustness are often insufficient in severe weather or complex terrain conditions.

[0005] Enhanced perception methods based on machine learning can perform more intelligent analysis of environmental states and target behaviors by using historical data for pattern recognition and prediction. Through feature engineering and data-driven learning models, such methods have improved the robustness of traditional methods to a certain extent and enhanced perception capabilities in unstructured environments. However, their performance depends on the quality and diversity of training data and has limitations when dealing with unknown environments.

[0006] Intelligent perception methods based on deep learning use models such as Convolutional Neural Networks (CNN) and Graph Neural Networks (GNN) to directly extract complex features from high-dimensional data such as images and point clouds, and perform real-time environmental understanding and decision-making. Compared with traditional methods and machine learning methods, deep learning technology has greater advantages in multimodal data fusion and adaptability to environmental changes, and can significantly improve the accuracy and robustness of perception systems in dynamic environments. However, its high computing requirements also place higher demands on the hardware performance and real-time performance of the system.

[0007] Therefore, how to reduce the complexity of the detection network required for perception while improving the perception accuracy has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0008] The present invention provides an unmanned vibratory roller sensing method, an unmanned vibratory roller sensing device, an unmanned vibratory roller sensing system and a storage medium, which solve the problems of complex detection network required for sensing and low sensing accuracy existing in the related art.

[0009] As a first aspect of the present invention, there is provided an unmanned vibrating roller sensing method, which comprises:

[0010] Acquire data information collected by a multi-source sensor, wherein the data information collected by the multi-source sensor includes at least image data and point cloud data, and the image data includes at least RGB image and thermal imaging image;

[0011] Preprocessing the data information collected by the multi-source sensors to obtain multi-source sensor preprocessing data;

[0012] Performing feature extraction on the multi-source sensor preprocessing data to obtain multi-source sensor feature data;

[0013] Performing feature fusion on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data;

[0014] The multi-source sensor feature fusion data is subjected to feature detection according to the target detection and classification model to obtain a target perception result.

[0015] Furthermore, preprocessing the data information collected by the multi-source sensors to obtain preprocessed data of the multi-source sensors includes:

[0016] The image data and point cloud data are both subjected to denoising, normalization and time synchronization processing to obtain the positional relationship of the multi-source sensor relative to the world coordinate system.

[0017] Furthermore, the image data and the point cloud data are subjected to denoising, normalization and time synchronization processing, including:

[0018] The noise of the RGB image is removed according to the Gaussian filtering method, and the color of the RGB image after the noise is removed is normalized by the contrast enhancement method and the white balance adjustment method to obtain an RGB normalized processed image;

[0019] Removing temperature noise from the thermal imaging image according to an adaptive threshold method, and normalizing the temperature value in the thermal imaging image after the temperature noise is removed to obtain a normalized thermal imaging image;

[0020] Remove outliers and misdetected points in the point cloud data according to the RANSAC algorithm to obtain point cloud denoised data;

[0021] The point cloud denoised data, the RGB normalized processed image and the thermal imaging normalized image are spatially aligned to obtain the point cloud denoised data, the RGB normalized processed image and the thermal imaging normalized image in the same reference coordinate system.

[0022] Further, feature extraction is performed on the multi-source sensor preprocessing data to obtain multi-source sensor feature data, including:

[0023] Performing feature extraction on the image data according to a lightweight CNN to obtain image feature data;

[0024] Feature extraction is performed on the point cloud data according to PointNet to obtain point cloud feature data.

[0025] Furthermore, feature extraction is performed on the image data according to the lightweight CNN to obtain image feature data, including:

[0026] A depthwise separable convolution operation is performed on the image data to obtain convolution operation feature data, wherein the depthwise separable convolution operation includes depthwise convolution and pointwise convolution, and the expression of the depthwise convolution is:

[0027]

[0028]

[0029] in, Represents the output feature of the cth channel at position (x, y), represents the depth convolution kernel of the lth layer, and I(x+i,y+j,c) represents the value of the cth channel of the image data; represents the eigenvalue of the point-by-point convolution output, represents the weight of 1×1 convolution, C in Represents the input channel tree;

[0030] The convolution operation feature data is processed according to the ReLU activation function to obtain image feature data.

[0031] Further, feature extraction is performed on the point cloud data according to PointNet to obtain point cloud feature data, including:

[0032] The point cloud data is subjected to feature extraction according to PointNet, wherein the feature extraction expression of PointNet is:

[0033] f(X)=max i∈[1,N] (φ(x i )),

[0034] Among them, φ represents the feature learning function of each point cloud, φ(x i )=W2·σ(W1·x i +b1)+b2, σ represents the activation function, W1 and W2 both represent the weight matrix of the feature learning function, b1 and b2 both represent bias terms, X represents the point cloud data, and the max operation represents the extraction of global features from the features of all point clouds.

[0035] Further, the multi-source sensor feature data is subjected to feature fusion according to the channel attention mechanism to obtain multi-source sensor feature fusion data, including:

[0036] Converting the feature data of the RGB image, the feature data of the thermal imaging image, and the feature data of the point cloud data into feature representations of the same dimension;

[0037] Calculate a channel descriptor based on feature data of an RGB image, feature data of a thermal imaging image, and feature data of point cloud data of the same dimension;

[0038] Calculate the attention weights of the channel descriptors of multi-source sensors according to the Softmax function;

[0039] Calculating adaptive weights of channel descriptors of multi-source sensors according to an adaptive weight allocation mechanism;

[0040] The feature fusion of the multi-source sensors is performed according to the channel descriptors of the multi-source sensors and their corresponding attention weights and adaptive weights to obtain the multi-source sensor feature fusion data.

[0041] As another aspect of the present invention, an unmanned vibrating roller sensing device is provided, which is used to implement the unmanned vibrating roller sensing method described above, and includes:

[0042] A data acquisition module is used to acquire data information collected by multi-source sensors, wherein the data information collected by the multi-source sensors includes at least image data and point cloud data, and the image data includes at least RGB images and thermal imaging images;

[0043] A preprocessing module, used to preprocess the data information collected by the multi-source sensors to obtain multi-source sensor preprocessing data;

[0044] A feature extraction module, used to extract features from the multi-source sensor preprocessing data to obtain multi-source sensor feature data;

[0045] A feature fusion module is used to perform feature fusion on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data;

[0046] The target detection and classification module is used to perform feature detection on the multi-source sensor feature fusion data according to the target detection and classification model to obtain the target perception result.

[0047] As another aspect of the present invention, there is provided an unmanned vibrating roller sensing system, which comprises: an RGB camera device, a thermal imaging device, a millimeter wave radar device and the unmanned vibrating roller sensing device described above, wherein the RGB camera device, the thermal imaging device and the millimeter wave radar device are all in communication connection with the unmanned vibrating roller sensing device;

[0048] The RGB camera device is used to collect RGB images;

[0049] The thermal imaging device is used to collect thermal imaging images;

[0050] The millimeter wave radar device is used to collect point cloud data;

[0051] The unmanned vibratory roller sensing device is used to perform feature detection after preprocessing, feature extraction and feature fusion based on RGB images, thermal imaging images and point cloud data to obtain target perception results.

[0052] As another aspect of the present invention, a storage medium is provided, which is used to store computer instructions. When the computer instructions are loaded and executed by a processor, the unmanned vibration roller sensing method described above is implemented.

[0053] The unmanned vibrating roller perception method provided by the present invention obtains environmental information around the vibrating roller through sensors such as RGB cameras, thermal imaging devices and millimeter wave radars, and uses this to construct data samples to provide input for subsequent algorithms; preprocesses the acquired multi-source data, including image denoising, data normalization and time synchronization, to ensure the high robustness of the data and the spatiotemporal consistency of data from different sensors; uses lightweight CNN and PointNet to extract features from the collected image data and point cloud data respectively; uses a channel attention mechanism to fuse the extracted image and point cloud features; and uses a simplified Fast ParC-Net model architecture to detect and classify targets for the fused data. This unmanned vibrating roller perception method can not only reduce sensor redundancy and optimize system costs, but also has excellent stability and performance in severe weather; uses a fusion network based on a channel attention mechanism, so that the model can automatically learn the importance of different modal data and highlight the features that have the greatest impact on the perception results. Therefore, the unmanned vibrating roller perception method provided by the present invention can not only be conveniently and quickly deployed on devices with limited computing resources, but also exhibit excellent recognition capabilities in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0055] Figure 1 A flow chart of the unmanned vibrating roller sensing method provided by the present invention.

[0056] Figure 2 A flow chart of the preprocessing method provided by the present invention.

[0057] Figure 3 This is a flow chart of the feature extraction method provided by the present invention.

[0058] Figure 4 A flow chart of the feature fusion method provided by the present invention.

[0059] Figure 5 This is a structural block diagram of the unmanned vibrating roller sensing device provided by the present invention.

[0060] Figure 6 This is a structural block diagram of the unmanned vibrating roller sensing system provided by the present invention.

[0061] Figure 7 This is a structural block diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0062] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0063] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.

[0064] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0065] In this embodiment, an unmanned vibration roller sensing method is provided. Figure 1 is a flow chart of an unmanned vibration roller sensing method provided according to an embodiment of the present invention. Figure 1 As shown, including:

[0066] S100, acquiring data information collected by a multi-source sensor, wherein the data information collected by the multi-source sensor includes at least image data and point cloud data, and the image data includes at least RGB images and thermal imaging images;

[0067] In the embodiment of the present invention, the multi-source sensor may specifically include an RGB camera device, a thermal imaging device, and a millimeter wave radar device, wherein the RGB camera device is used to collect RGB images, the thermal imaging device is used to collect thermal imaging images, and the millimeter wave radar device is used to collect point cloud data. Therefore, the data information collected by the multi-source sensor includes at least RGB images, thermal imaging images, and point cloud data.

[0068] It should be noted that in the embodiment of the present invention, the RGB camera device can capture visible light images, and the thermal imaging images collected by the thermal imaging device supplement the environmental perception ability in low light or bad weather. The fusion of this multi-spectral data improves the visual recognition ability of the perception system in a variety of environments. At the same time, combined with millimeter-wave radar point cloud data, the long-distance detection capability of the perception system is realized. Therefore, the embodiment of the present invention obtains environmental information around the vibrating roller through the arrangement of multi-source sensors, which can not only reduce sensor redundancy and optimize system costs, but also has excellent stability and performance in bad weather.

[0069] S200, preprocessing the data information collected by the multi-source sensors to obtain multi-source sensor preprocessed data;

[0070] In the embodiment of the present invention, preprocessing is performed on RGB images, thermal imaging images and point cloud data respectively, including image denoising, data normalization and time synchronization, to ensure high robustness of the data and spatiotemporal consistency of data from different sensors.

[0071] S300, performing feature extraction on the multi-source sensor preprocessing data to obtain multi-source sensor feature data;

[0072] In the embodiment of the present invention, lightweight convolutional neural networks (CNN) are used to extract features from RGB image and thermal imaging image data; and point cloud processing network (PointNet) is used to extract spatial features from millimeter wave radar data.

[0073] S400, performing feature fusion on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data;

[0074] Specifically, in the feature fusion stage, features from different sources are aligned and fused. The attention mechanism is used to enhance important features while suppressing noise and other unimportant background information.

[0075] S500: Perform feature detection on the multi-source sensor feature fusion data according to a target detection and classification model to obtain a target perception result.

[0076] In an embodiment of the present invention, a simplified Fast ParC-Net model architecture is used for target detection and classification. The network combines the position-aware circular convolution (ParC) operation with the Squeeze-and-Excitation module to improve the expressiveness of the model and reduce the computational complexity of the model.

[0077] Therefore, the unmanned vibration roller perception method provided by the present invention obtains the environmental information around the vibration roller through sensors such as RGB camera devices, thermal imaging devices and millimeter wave radars, and constructs data samples based on this to provide input for subsequent algorithms; preprocesses the acquired multi-source data, including image denoising, data normalization and time synchronization, to ensure the high robustness of the data and the spatiotemporal consistency of different sensor data; lightweight CNN and PointNet are used to extract features from the collected image data and point cloud data respectively; the channel attention mechanism is used to fuse the extracted image and point cloud features; the simplified Fast ParC-Net model architecture is used for target detection and classification of the fused data. This unmanned vibration roller perception method can not only reduce sensor redundancy and optimize system costs, but also has excellent stability and performance in bad weather; the fusion network based on the channel attention mechanism is used, so that the model can automatically learn the importance of different modal data and highlight the features that have the greatest impact on the perception results. Therefore, the unmanned vibration roller perception method provided by the present invention can not only be conveniently and quickly deployed on devices with limited computing resources, but also show excellent recognition capabilities in complex scenes.

[0078] In an embodiment of the present invention, preprocessing the multi-source sensor collected data information to obtain multi-source sensor preprocessed data includes:

[0079] The image data and point cloud data are both subjected to denoising, normalization and time synchronization processing to obtain the positional relationship of the multi-source sensor relative to the world coordinate system.

[0080] It should be understood that Gaussian filtering is used to remove noise from RGB images to ensure image clarity. At the same time, the image is contrast enhanced and white balance adjusted to ensure that the color of the image collected under different lighting conditions remains consistent, thereby eliminating the impact of lighting on image quality. An adaptive threshold algorithm is used to remove temperature noise from thermal imaging data to maintain the clarity of the temperature information of the target object. At the same time, the temperature value in the data is normalized to between 0 and 1 so that it can be fused with other types of sensor data. For the point cloud noise in the millimeter-wave radar data, the RANSAC (Random Sample Consensus) algorithm is used to remove outliers and misdetected points to improve the accuracy of the point cloud data. The RGB image data, thermal imaging infrared image, and millimeter-wave radar point cloud data are spatially aligned to obtain the relative position relationship of each sensor relative to the world coordinate system.

[0081] Specifically, Figure 2 As shown, the image data and the point cloud data are both subjected to denoising, normalization and time synchronization processing, including:

[0082] S210, removing noise from the RGB image according to a Gaussian filtering method, and normalizing the color of the RGB image after the noise is removed by a contrast enhancement method and a white balance adjustment method to obtain an RGB normalized processed image;

[0083] Specifically, for RGB images, the image noise is removed by Gaussian filtering, and the image color is normalized using contrast enhancement and white balance adjustment methods to ensure that images collected under different lighting conditions can maintain consistent color performance and eliminate the impact of lighting on image quality. The expression of Gaussian filtering is:

[0084]

[0085] The expression for color normalization is:

[0086]

[0087] Where I(x,y) represents the RGB image to be processed, σ represents the standard deviation of the Gaussian kernel, k represents the window size of the filter, and I min and I max Represent the minimum pixel value and the maximum pixel value of the image respectively.

[0088] S220, removing the temperature noise of the thermal imaging image according to an adaptive threshold method, and normalizing the temperature value in the thermal imaging image after the temperature noise is removed to obtain a normalized thermal imaging image;

[0089] In the embodiment of the present invention, an adaptive threshold algorithm is used to remove temperature noise in the thermal imaging image to maintain the clarity of the temperature information of the target object. The temperature value in the thermal imaging image is normalized to between 0 and 1 so as to be fused with other types of sensor data. The expressions for temperature noise filtering and normalization processing are:

[0090]

[0091] Where T represents the threshold value that is adaptively adjusted according to the local characteristics of the image, T(x,y) represents the temperature value in the thermal imaging image, and T min and T max Represent the minimum temperature and maximum temperature respectively.

[0092] S230, removing outliers and misdetected points in the point cloud data according to the RANSAC algorithm to obtain point cloud denoised data;

[0093] In the embodiment of the present invention, the RANSAC algorithm is used to remove outliers and misdetected points from the millimeter wave radar point cloud data to improve the accuracy of the point cloud data. The expression of the denoising process is shown in the following formula:

[0094]

[0095] Among them, d represents the distance from the point cloud to the plane. If d exceeds a certain threshold, the point cloud is considered to be an outlier.

[0096] S240, spatially aligning the point cloud denoised data, the RGB normalized processed image and the thermal imaging normalized image to obtain the point cloud denoised data, the RGB normalized processed image and the thermal imaging normalized image in the same reference coordinate system.

[0097] Specifically, the point cloud data of the millimeter-wave radar is spatially aligned with the data of the RGB camera and the thermal imaging camera to ensure that the data collected by all sensors are in the same reference coordinate system. The expression of the spatial configuration is:

[0098]

[0099] Among them, t represents the translation matrix and K represents the camera matrix.

[0100] In an embodiment of the present invention, feature extraction is performed on the multi-source sensor preprocessing data to obtain multi-source sensor feature data, such as Figure 3 As shown, including:

[0101] S310, extracting features from the image data according to a lightweight CNN to obtain image feature data;

[0102] A lightweight CNN is used to extract features from image data. First, a depth-wise separable convolution operation is performed on the input image to extract image features, where the depth-wise separable convolution can be decomposed into depth-wise convolution and point-wise convolution.

[0103] Specifically, feature extraction is performed on the image data according to the lightweight CNN to obtain image feature data, including:

[0104] A depthwise separable convolution operation is performed on the image data to obtain convolution operation feature data, wherein the depthwise separable convolution operation includes depthwise convolution and pointwise convolution, and the expression of the depthwise convolution is:

[0105]

[0106] Among them, F l depthwise (x,y,c) represents the output feature of the cth channel at position (x,y), represents the depth convolution kernel of the lth layer, I(x+i,y+j,c) represents the value of the cth channel of the image data; F l pointwise (x, y, c) represents the eigenvalues ​​of the point-by-point convolution output, represents the weight of 1×1 convolution, C in Represents the input channel tree;

[0107] The convolution operation feature data is processed according to the ReLU activation function to obtain image feature data.

[0108] It should be understood that after the convolution operation, a nonlinear layer is introduced using the ReLU activation function, while global average pooling is used instead of the traditional fully connected layer during downsampling to further reduce the amount of parameters.

[0109] S320 , performing feature extraction on the point cloud data according to PointNet to obtain point cloud feature data.

[0110] In an embodiment of the present invention, feature extraction is performed on the point cloud data according to PointNet to obtain point cloud feature data, including:

[0111] The point cloud data is subjected to feature extraction according to PointNet, wherein the feature extraction expression of PointNet is:

[0112] f(X)=max i∈[1,N] (φ(x i )),

[0113] Among them, φ represents the feature learning function of each point cloud, φ(x i )=W2·σ'(W1·x i +b1)+b2, σ' represents the activation function, W1 and W2 both represent the weight matrix of the feature learning function, b1 and b2 both represent bias terms, X represents the point cloud data, and the max operation represents the extraction of global features from the features of all point clouds.

[0114] In order to fuse the image features and the point cloud features to improve the accuracy and robustness of the perception system, in an embodiment of the present invention, the multi-source sensor feature data is subjected to feature fusion according to the channel attention mechanism to obtain multi-source sensor feature fusion data, such as Figure 4 As shown, including:

[0115] S410, converting the feature data of the RGB image, the feature data of the thermal imaging image, and the feature data of the point cloud data into feature representations of the same dimension;

[0116] In an embodiment of the present invention, feature alignment is performed on data from different sensors, and image, thermal imaging and point cloud data are converted into feature representations of the same dimension.

[0117] S420, calculating a channel descriptor according to feature data of the RGB image, feature data of the thermal imaging image, and feature data of the point cloud data of the same dimension;

[0118] Specifically, the calculation formula of the channel descriptor is:

[0119] z=σ'(W1′·F RGB )+σ′(W2′·F IR )+σ′(W3′·F point ),

[0120] Among them, F RGB Represents the feature data of the RGB image, F IR Represents the characteristic data of the thermal imaging image, F point represents the feature data of the point cloud data, W1′, W2′ and W3′ all represent learnable weight matrices, and σ′ represents the activation function.

[0121] S430, calculating the attention weights of the channel descriptors of the multi-source sensors according to the Softmax function;

[0122] In this embodiment of the present invention, the Softmax function is used to calculate the attention weight, and the calculation formula is:

[0123] a=Softmax(z).

[0124] S440, calculating the adaptive weights of the channel descriptors of the multi-source sensors according to the adaptive weight allocation mechanism;

[0125] The adaptive weight is calculated through the adaptive weight allocation mechanism. The expression for weight calculation is:

[0126]

[0127] β=1-α,

[0128] Among them, γ represents an adjustable parameter, z RGB Channel descriptor representing the feature data of RGB image, z IR A channel descriptor representing feature data of a thermal imaging image.

[0129] S450 , performing feature fusion of the multi-source sensors according to the channel descriptors of the multi-source sensors and their corresponding attention weights and adaptive weights to obtain multi-source sensor feature fusion data.

[0130] Specifically, the final expression of multi-source sensor feature fusion data is:

[0131] F fusion =α·(a RGB ·F RGB )+β·(a IR ·F IR )+(1-α-β)·F point .

[0132] In an embodiment of the present invention, feature detection is performed on the multi-source sensor feature fusion data according to a target detection and classification model to obtain a target perception result, including:

[0133] The multi-source sensor feature fusion data is subjected to target recognition and classification according to the ParC-Net network to obtain target perception results.

[0134] For the above fusion features, the ParC-Net network is used for target recognition and classification, and the final output result is expressed as:

[0135] Output = parc-Net(F fusion ).

[0136] It should be noted that another embodiment of the multi-source sensor in the embodiment of the present invention may also include an RGB camera device, a millimeter wave radar device and a lidar device. Based on this, the target recognition and detection network can be implemented using YOLO, Vison Transformer, etc.

[0137] In summary, the unmanned vibrating roller perception method provided by the present invention uses an RGB camera device to capture visible light images, and the thermal imaging head infrared image supplements the environmental perception ability in low light or bad weather. The visual recognition ability of the perception system in a variety of environments is improved through the fusion of this multi-spectral data. At the same time, combined with millimeter-wave radar point cloud data, the long-distance detection capability of the perception system is realized. This sensor arrangement method can not only reduce sensor redundancy and optimize system cost, but also has excellent stability and performance in bad weather. The fusion network based on the channel attention mechanism is adopted, so that the model can automatically learn the importance of different modal data and highlight the features that have the greatest impact on the perception results. At the same time, through the adaptive weight allocation mechanism, the fusion ratio of each modal feature is dynamically adjusted, for example, the weight of the thermal imaging camera and the millimeter-wave radar is increased under low visibility conditions, and the influence of the RGB camera is reduced, thereby improving the overall perception accuracy. The Parc-Net network is used for target recognition and classification. The network combines the advantages of convolutional neural networks in local feature extraction and the ability of visual Transformer in global feature modeling, thereby reducing the complexity of the model while maintaining excellent performance. Therefore, the unmanned vibratory roller perception method provided by the present invention can not only be conveniently and quickly deployed on devices with limited computing resources, but also demonstrates excellent recognition capabilities in complex scenarios.

[0138] As another embodiment of the present invention, an unmanned vibrating roller sensing device 100 is provided for implementing the unmanned vibrating roller sensing method described above, wherein: Figure 5 As shown, including:

[0139] The acquisition data acquisition module 110 is used to acquire the data information acquired by the multi-source sensor, wherein the data information acquired by the multi-source sensor includes at least image data and point cloud data, and the image data includes at least RGB image and thermal imaging image;

[0140] A preprocessing module 120 is used to preprocess the data information collected by the multi-source sensors to obtain multi-source sensor preprocessed data;

[0141] A feature extraction module 130 is used to extract features from the multi-source sensor preprocessed data to obtain multi-source sensor feature data;

[0142] A feature fusion module 140 is used to perform feature fusion on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data;

[0143] The target detection and classification module 150 is used to perform feature detection on the multi-source sensor feature fusion data according to the target detection and classification model to obtain a target perception result.

[0144] The unmanned vibrating roller sensing device provided by the present invention obtains environmental information around the vibrating roller through sensors such as RGB camera devices, thermal imaging devices and millimeter wave radars, and constructs data samples based on this information to provide input for subsequent algorithms; preprocesses the acquired multi-source data, including image denoising, data normalization and time synchronization, to ensure the high robustness of the data and the spatiotemporal consistency of data from different sensors; uses lightweight CNN and PointNet to extract features from the collected image data and point cloud data respectively; uses a channel attention mechanism to fuse the extracted image and point cloud features; and uses a simplified Fast ParC-Net model architecture to detect and classify targets for the fused data. This unmanned vibrating roller sensing device can not only reduce sensor redundancy and optimize system costs, but also has excellent stability and performance in severe weather; uses a fusion network based on a channel attention mechanism, so that the model can automatically learn the importance of different modal data and highlight the features that have the greatest impact on the perception results. Therefore, the unmanned vibrating roller sensing device provided by the present invention can not only be conveniently and quickly deployed on devices with limited computing resources, but also exhibit excellent recognition capabilities in complex scenarios.

[0145] Regarding the specific working principle of the unmanned vibration roller sensing device provided by the present invention, reference can be made to the description of the unmanned vibration roller sensing method mentioned above, which will not be repeated here.

[0146] As another embodiment of the present invention, an unmanned vibrating roller sensing system 10 is provided, wherein: Figure 6As shown, it includes: an RGB camera device 200, a thermal imaging device 300, a millimeter wave radar device 400 and the unmanned vibration roller sensing device 100 mentioned above, and the RGB camera device 200, the thermal imaging device 300 and the millimeter wave radar device 400 are all connected to the unmanned vibration roller sensing device 100 for communication;

[0147] The RGB camera device 200 is used to collect RGB images;

[0148] The thermal imaging device 300 is used to collect thermal imaging images;

[0149] The millimeter wave radar device 400 is used to collect point cloud data;

[0150] The unmanned vibratory roller sensing device 100 is used to perform feature detection after preprocessing, feature extraction and feature fusion based on RGB images, thermal imaging images and point cloud data to obtain target perception results.

[0151] In the embodiment of the present invention, the environmental information of the vibrating roller is obtained through the RGB camera device, the thermal imaging device and the millimeter wave radar device, and the data sample is constructed based on this. The unmanned vibrating roller sensing device 100 performs noise removal, normalization, time alignment and other operations on the obtained data, and uses lightweight CNN and PointNet to extract features for the image data and point cloud data respectively; constructs a feature fusion module based on the channel attention mechanism, and combines the adaptive attention weight allocation mechanism to allocate weights to different features, and dynamically adjust the fusion ratio of each modal feature; for the fused features, the Parc-Net network is used to identify and classify the target.

[0152] Specifically, RGB cameras, thermal imaging cameras and millimeter-wave radars are installed on the unmanned vibrating roller to obtain visible light images and point cloud data. The fusion of multi-spectral data helps improve visual recognition capabilities under different environmental conditions. At the same time, combined with millimeter-wave radar point cloud data, the long-distance detection capability of the perception system is realized, sensor redundancy is reduced, system costs are optimized, and reliability in harsh environments is increased. Through this multi-sensor data acquisition method, high-precision and high-reliability perception in harsh environments is achieved.

[0153] In addition, the image and point cloud features are extracted respectively through lightweight CNN and PointNet, and a fusion network based on the channel attention mechanism is constructed to fuse the image and point cloud features. This enables the model to automatically learn the importance of different modal data, highlight the features that have the greatest impact on the perception results, and dynamically adjust the fusion ratio of each modal feature through an adaptive weight allocation mechanism. For example, in low visibility conditions, the weights of thermal imaging cameras and millimeter-wave radars are increased to reduce the impact of RGB cameras, thereby improving the overall perception accuracy.

[0154] Finally, the Parc-Net network is used to identify and classify the input features. This network combines the advantages of convolutional neural networks in local feature extraction and the ability of visual transformers in global feature modeling, making it have excellent performance under low complexity conditions, so that it can be easily and quickly deployed on devices with limited computing resources.

[0155] The specific working principle of the unmanned vibration roller sensing system provided by the present invention can be referred to the description of the unmanned vibration roller sensing method mentioned above, which will not be repeated here.

[0156] As another embodiment of the present invention, a storage medium is provided, which is used to store computer instructions. When the computer instructions are loaded and executed by a processor, the unmanned vibration roller sensing method described above is implemented.

[0157] In an embodiment of the present invention, a non-transitory computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions can execute the unmanned vibration roller sensing method in any of the above method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memories.

[0158] As another embodiment of the present invention, an electronic device is provided, which includes a memory and a processor, wherein the processor is communicatively connected to the memory, the memory is used to store a computer program, and the processor is used to load and execute the computer program to implement the unmanned vibration roller sensing method described above.

[0159] like Figure 7As shown, the electronic device 70 may include: at least one processor 71, such as a CPU (Central Processing Unit), at least one communication interface 73, a memory 74, and at least one communication bus 72. The communication bus 72 is used to realize the connection and communication between these components. The communication interface 73 may include a display screen (Display) and a keyboard (Keyboard), and the optional communication interface 73 may also include a standard wired interface and a wireless interface. The memory 74 may be a high-speed RAM memory (Random Access Memory, volatile random access memory) or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 74 may optionally be at least one storage device located away from the aforementioned processor 71. The memory 74 stores an application program, and the processor 71 calls the program code stored in the memory 74 to execute any of the above method steps.

[0160] The communication bus 72 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The communication bus 72 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0161] Among them, the memory 74 may include a volatile memory (English: volatile memory), such as a random access memory (English: random-access memory, abbreviated: RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk drive (English: hard disk drive, abbreviated: HDD) or a solid-state drive (English: solid-state drive, abbreviated: SSD); the memory 74 may also include a combination of the above types of memory.

[0162] The processor 71 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP.

[0163] The processor 71 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0164] Optionally, the memory 74 is also used to store program instructions. The processor 71 can call the program instructions to implement the present invention. Figure 1 The unmanned vibrating roller sensing method shown in the embodiment.

[0165] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. An unmanned vibrating roller sensing method, characterized in that: include: Acquire data information collected by a multi-source sensor, wherein the data information collected by the multi-source sensor includes at least image data and point cloud data, and the image data includes at least RGB image and thermal imaging image; Preprocessing the data information collected by the multi-source sensors to obtain multi-source sensor preprocessing data; Performing feature extraction on the multi-source sensor preprocessing data to obtain multi-source sensor feature data; Performing feature fusion on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data; The multi-source sensor feature fusion data is subjected to feature detection according to the target detection and classification model to obtain a target perception result.

2. The unmanned vibrating roller sensing method according to claim 1 is characterized in that: Preprocessing the data information collected by the multi-source sensors to obtain multi-source sensor preprocessed data includes: The image data and point cloud data are both subjected to denoising, normalization and time synchronization processing to obtain the positional relationship of the multi-source sensor relative to the world coordinate system.

3. The unmanned vibrating roller sensing method according to claim 2 is characterized in that: The image data and point cloud data are both subjected to denoising, normalization and time synchronization processing, including: The noise of the RGB image is removed according to the Gaussian filtering method, and the color of the RGB image after the noise is removed is normalized by the contrast enhancement method and the white balance adjustment method to obtain an RGB normalized processed image; Removing temperature noise from the thermal imaging image according to an adaptive threshold method, and normalizing the temperature value in the thermal imaging image after the temperature noise is removed to obtain a normalized thermal imaging image; Remove outliers and misdetected points in the point cloud data according to the RANSAC algorithm to obtain point cloud denoised data; The point cloud denoised data, the RGB normalized processed image and the thermal imaging normalized image are spatially aligned to obtain the point cloud denoised data, the RGB normalized processed image and the thermal imaging normalized image in the same reference coordinate system.

4. The unmanned vibrating roller sensing method according to claim 1, characterized in that: Extracting features from the multi-source sensor preprocessed data to obtain multi-source sensor feature data includes: Performing feature extraction on the image data according to a lightweight CNN to obtain image feature data; Feature extraction is performed on the point cloud data according to PointNet to obtain point cloud feature data.

5. The unmanned vibrating roller sensing method according to claim 4 is characterized in that: Performing feature extraction on the image data according to the lightweight CNN to obtain image feature data includes: A depthwise separable convolution operation is performed on the image data to obtain convolution operation feature data, wherein the depthwise separable convolution operation includes depthwise convolution and pointwise convolution, and the expression of the depthwise convolution is: Among them, F l depthwise (x,y,c) represents the output feature of the cth channel at position (x,y), represents the depth convolution kernel of the lth layer, I(x+i,y+j,c) represents the value of the cth channel of the image data; F l pointwise (x, y, c) represents the eigenvalues ​​of the point-by-point convolution output, represents the weight of 1×1 convolution, C in Represents the input channel tree; The convolution operation feature data is processed according to the ReLU activation function to obtain image feature data.

6. The unmanned vibrating roller sensing method according to claim 4, characterized in that: Perform feature extraction on the point cloud data according to PointNet to obtain point cloud feature data, including: The point cloud data is subjected to feature extraction according to PointNet, wherein the feature extraction expression of PointNet is: f(X)=max i∩[1,N] (φ(x i )), Among them, φ represents the feature learning function of each point cloud, φ(x i )=W2·σ(W1·x i +b1)+b2, σ represents the activation function, W1 and W2 both represent the weight matrix of the feature learning function, b1 and b2 both represent bias terms, X represents the point cloud data, and the max operation represents the extraction of global features from the features of all point clouds.

7. The unmanned vibrating roller sensing method according to claim 1, characterized in that: The multi-source sensor feature data is subjected to feature fusion according to the channel attention mechanism to obtain multi-source sensor feature fusion data, including: Converting the feature data of the RGB image, the feature data of the thermal imaging image, and the feature data of the point cloud data into feature representations of the same dimension; Calculate a channel descriptor based on feature data of an RGB image, feature data of a thermal imaging image, and feature data of point cloud data of the same dimension; Calculate the attention weights of the channel descriptors of multi-source sensors according to the Softmax function; Calculating adaptive weights of channel descriptors of multi-source sensors according to an adaptive weight allocation mechanism; The feature fusion of the multi-source sensors is performed according to the channel descriptors of the multi-source sensors and their corresponding attention weights and adaptive weights to obtain the multi-source sensor feature fusion data.

8. An unmanned vibrating roller sensing device, used to implement the unmanned vibrating roller sensing method according to any one of claims 1 to 7, characterized in that: include: A data acquisition module is used to acquire data information collected by multi-source sensors, wherein the data information collected by the multi-source sensors includes at least image data and point cloud data, and the image data includes at least RGB images and thermal imaging images; A preprocessing module, used to preprocess the data information collected by the multi-source sensors to obtain multi-source sensor preprocessing data; A feature extraction module, used to extract features from the multi-source sensor preprocessing data to obtain multi-source sensor feature data; A feature fusion module is used to perform feature fusion on the multi-source sensor feature data according to a channel attention mechanism to obtain multi-source sensor feature fusion data; The target detection and classification module is used to perform feature detection on the multi-source sensor feature fusion data according to the target detection and classification model to obtain the target perception result.

9. An unmanned vibrating roller sensing system, characterized in that: include: An RGB camera device, a thermal imaging device, a millimeter wave radar device and the unmanned vibrating roller sensing device according to claim 8, wherein the RGB camera device, the thermal imaging device and the millimeter wave radar device are all communicatively connected to the unmanned vibrating roller sensing device; The RGB camera device is used to collect RGB images; The thermal imaging device is used to collect thermal imaging images; The millimeter wave radar device is used to collect point cloud data; The unmanned vibratory roller sensing device is used to perform feature detection after preprocessing, feature extraction and feature fusion based on RGB images, thermal imaging images and point cloud data to obtain target perception results.

10. A storage medium, characterized in that: Used to store computer instructions, when the computer instructions are loaded and executed by a processor, to implement the unmanned vibration roller sensing method described in any one of claims 1 to 7.

Images