Auxiliary driving method suitable for underground coal mine transport locomotive

Abstract

The invention relates to an auxiliary driving method suitable for an underground coal mine transport locomotive. The method comprises the steps that space environment information of the advancing direction of an underground coal mine transport locomotive is obtained in real time, a walking front target laser point cloud and a walking front target image are loaded into an obstacle detection model so that corresponding obstacle state information can be output through the obstacle detection model, the walking front target image is loaded into a track detection model, and the track detection model outputs the corresponding obstacle state information. And outputting corresponding track shape information through the track detection model, and generating driving prompt information when the obstacle state information is that an obstacle influencing advancing exists and / or the track shape information is in a curved track state. According to the invention, the space environment state in the advancing direction of the underground coal mine transport locomotive can be effectively detected, the operation of the underground coal mine transport locomotive is regulated and controlled, and the stability and reliability of the operation of the underground coal mine transport locomotive are improved.

Description

Technical Field

[0001] The invention relates to an assisted driving method, in particular to an assisted driving method suitable for underground coal mine transport locomotives. Background Art

[0002] In order to meet the needs of unmanned driving, a vehicle perception system needs to be installed on unmanned vehicles to use the wheel perception system to perceive the surrounding environment and provide support for further path planning, motion decision-making, etc. Compared with unmanned driving on the ground or open-pit mines, underground driving tunnels have the characteristics of small space, uneven lighting, high temperature, high humidity, and uneven floor plates in some tunnels. In addition, there are explosive substances such as gas and coal dust. Therefore, the underground environment will limit the performance of vehicle perception devices.

[0003] At present, deep learning-based target detection in coal mines mostly uses single sensors, such as visual cameras or single lidars. Specifically, visual cameras can obtain rich information about the surrounding environment and objects, but due to the lack of depth features, they cannot obtain accurate three-dimensional spatial information and are easily affected by the external environment. LiDARs have good robustness and can accurately obtain three-dimensional spatial information of objects, but their resolution is low and they cannot obtain rich object information.

[0004] Due to the complex environment, low light and high dust content in coal mines, traditional vision-based image recognition algorithms are difficult to accurately and efficiently detect obstacles, personnel and track directions in front of coal mine underground transport locomotives. In addition, due to the difficulty in extracting features from the laser radar point cloud data in coal mines, it may lead to false detection and missed detection, which will seriously affect the accuracy of detecting obstacles and personnel in front. It can be seen that the existing technology cannot effectively detect the spatial environment status of coal mine underground transport locomotives, which in turn leads to the inability to improve the stability and reliability of coal mine underground transport locomotives. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an assisted driving method suitable for underground coal mine transport locomotives, which can effectively detect the spatial environment state in the forward direction of the underground coal mine transport locomotive, and regulate the operation of the underground coal mine transport locomotive, thereby improving the stability and reliability of the operation of the underground coal mine transport locomotive.

[0006] According to the technical solution provided by the present invention, an assisted driving method suitable for underground coal mine transport locomotives, the assisted driving method comprising:

[0007] Real-time acquisition of spatial environment information of the forward direction of the underground transport locomotive in the coal mine, wherein the acquired spatial environment information includes a laser point cloud of the target ahead and an image of the target ahead, and the laser point cloud of the target ahead and the image of the target ahead are in a time-aligned state;

[0008] Loading the target laser point cloud and the target image ahead of the vehicle into the obstacle detection model, so as to output corresponding obstacle status information through the obstacle detection model, wherein the obstacle status information includes whether there is an obstacle that affects the vehicle ahead or not;

[0009] Loading the target image ahead of the walking into the track detection model, so as to output corresponding track shape information through the track detection model, wherein the track shape information includes a curved track state or a straight track state;

[0010] When the obstacle status information indicates that there is an obstacle that affects the forward movement and / or the track shape information indicates a curved track state, driving prompt information is generated, wherein the driving prompt information includes deceleration prompt information for controlling the deceleration of the coal mine underground transport locomotive or stop prompt information for controlling the coal mine underground transport locomotive to stop.

[0011] When using the obstacle detection model to detect obstacles, it includes:

[0012] Performing point cloud feature extraction on the target laser point cloud ahead of walking, so as to generate corresponding point cloud features ahead of walking after the point cloud feature extraction, and generating point cloud query features, point cloud key features and point cloud value features based on the point cloud features ahead of walking;

[0013] Performing image feature extraction on the walking ahead target image to generate corresponding walking ahead image features after the image feature extraction, and generating image query features, image key features and image features based on the walking ahead image features;

[0014] Performing point cloud-image feature fusion on the point cloud query feature, the image value feature, and the image key feature to generate corresponding point cloud-image fusion features after the point cloud-image feature fusion;

[0015] Performing image-point cloud feature fusion on the image query feature, the point cloud value feature, and the point cloud key feature to generate corresponding image-point cloud fusion features after the image-point cloud feature fusion;

[0016] Performing feature remapping processing on the point cloud-image fusion features and the image-point cloud fusion features to generate point cloud image remapping features after the feature remapping processing;

[0017] Perform feature stitching on the point cloud image remapping features, the walking front point cloud features, and the walking front image features to form a point cloud image fusion stitching feature after feature stitching;

[0018] The point cloud image fusion and splicing features are detected and classified, and obstacle detection and classification information is generated after detection and classification, and obstacle status information is generated based on the obstacle detection and classification information.

[0019] The obstacle detection model includes a point cloud feature extraction network and an image feature extraction network, wherein:

[0020] The point cloud feature extraction network is connected with the point cloud linear transformation module and the feature splicing module, and the image feature extraction network is connected with the image linear transformation module and the feature splicing module;

[0021] The point cloud linear transformation module and the image linear transformation module are adaptively connected to the point cloud-image feature cross-focus linear attention fusion module and the image-point cloud feature cross-focus linear attention fusion module respectively;

[0022] The point cloud-image feature cross-focus linear attention fusion module and the image-point cloud feature cross-focus linear attention fusion module are connected to the feature splicing module through the feature remapping module.

[0023] The feature stitching module is connected to the obstacle detection head;

[0024] When performing obstacle detection, it includes:

[0025] The point cloud feature extraction network is used to extract point cloud features from the target laser point cloud ahead of the walking object, and the corresponding point cloud features ahead of the walking object are generated after the point cloud feature extraction;

[0026] The image feature extraction network is used to extract image features of the target image ahead of the walking object, and the corresponding image features ahead of the walking object are generated after the image feature extraction;

[0027] The point cloud linear transformation module is used to perform point cloud linear transformation on the point cloud features ahead of walking, so as to generate point cloud query features, point cloud key features and point cloud value features after the point cloud linear transformation;

[0028] The image linear transformation module is used to perform image linear transformation on the image features ahead of the walking path, so as to generate image query features, image key features and image value features after the image linear transformation;

[0029] Loading the point cloud query features, image key features, and image features into the point cloud-image feature cross-focus linear attention fusion module to generate corresponding point cloud-image fusion features using the point cloud-image feature cross-focus linear attention fusion module;

[0030] Loading the image query features, point cloud value features, and point cloud key features into the image-point cloud feature cross-focus linear attention fusion module, so as to generate corresponding image-point cloud fusion features using the image-point cloud feature cross-focus linear attention fusion module;

[0031] Using a feature remapping module to perform feature remapping processing on the image-point cloud fusion feature and the point cloud-image fusion feature, so as to generate a point cloud image remapping feature after the feature remapping processing;

[0032] The feature stitching module is used to perform feature stitching on the point cloud image remapping features, the walking front point cloud features and the walking front image features, so as to generate the point cloud image fusion stitching features after feature stitching;

[0033] The obstacle detection head is used to detect and classify the point cloud image fusion and splicing features to generate detection and classification information after detection and classification.

[0034] The point cloud-image feature cross-focus linear attention fusion module includes a query feature processing unit and a key feature processing unit, wherein:

[0035] The query feature processing unit receives the point cloud query feature and is connected to the first input terminal of the matrix multiplier, wherein the query feature processing unit at least performs activation and focusing processing on the point cloud query feature to generate a point cloud post-activation and focusing query feature after the activation and focusing processing;

[0036] The key feature processing unit receives the image key feature and is connected to the second input terminal of the matrix multiplier, wherein the key feature processing unit at least performs activation and focusing processing on the image feature to generate an image activated and focused key feature after the activation and focusing processing;

[0037] The third input terminal of the matrix multiplier receives the image value feature, and the output terminal of the matrix multiplier is connected to the feature point remapping module, wherein the matrix multiplier first multiplies the image value feature with the key feature after image activation and focus, and then multiplies it with the query feature after point cloud activation and focus;

[0038] The feature point remapping module is connected to the feature map flattening processing module through a depthwise separable convolution, the feature map flattening processing module is connected to a first input end of a feature element adder, and a second input end of the feature element adder receives a point cloud query feature;

[0039] The output of the feature element adder is connected to the attention fusion fully connected layer.

[0040] The query feature processing unit includes a query feature Mish activation function and a query feature focusing function, wherein:

[0041] The received point cloud query features are processed in sequence using the query feature Mish activation function and the query feature focusing function, and after being processed by the query feature focusing function, a point cloud activated and focused query feature is generated.

[0042] The track detection model includes a track detection network and a track shape judgment module, wherein:

[0043] The lane detection network is built based on the multi-task single-stage network YOLOP;

[0044] A lane line detection network is used to receive a target image ahead of the walking object, and a lane line detection network is used to perform track detection on the target object ahead of the walking object, so as to generate at least a binary track mask image after the track detection;

[0045] Based on the binary track mask image, the track shape judgment module determines the shape of the track in the current forward target image and outputs the track shape information.

[0046] Based on the binary track mask image, the track shape judgment module determines the shape of the track in the current forward target image, including:

[0047] Extracting a single-side track region in the binary track mask image, and performing polynomial fitting on the single-side track region to generate a single-side track fitting curve after polynomial fitting;

[0048] Based on the shape of the unilateral track fitting curve, the shape of the track in the current forward target image is determined and the track shape information is output.

[0049] When obtaining the laser point cloud of the target ahead, it includes:

[0050] Acquire the time-aligned walking forward source laser point cloud, and perform point cloud data processing on the acquired walking forward source laser point cloud to generate walking forward laser point cloud processed data after point cloud data processing, wherein:

[0051] Point cloud data processing includes point cloud preprocessing and point cloud segmentation processing performed in sequence, wherein after point cloud preprocessing is performed on the forward source laser point cloud, corresponding laser point cloud preprocessed data is generated, and the point cloud preprocessing at least includes point cloud source data denoising processing;

[0052] The laser point cloud preprocessed data is subjected to point cloud segmentation processing, so as to remove the ground point cloud in the laser point cloud preprocessed data after the point cloud segmentation processing, and generate the corresponding walking front target laser point cloud.

[0053] When acquiring the target image ahead of you, it includes:

[0054] A time-aligned walking forward source image is acquired, and image preprocessing is performed on the acquired walking forward source image to generate a walking forward target image after image preprocessing, wherein:

[0055] The image preprocessing at least includes image enhancement processing and / or image denoising processing performed on the source image ahead of the walking.

[0056] When performing image enhancement processing on the source image ahead of walking, it includes:

[0057] Convert the source image in front of you from RGB color space to HSV color space;

[0058] The converted brightness V is enhanced to generate an enhanced brightness V after the enhancement process, wherein:

[0059] The enhancement processing of the brightness V at least includes the enhancement processing using the Retinex algorithm fused with multi-scale gradient decreasing domain guided filtering, and the local contrast enhancement using the restricted contrast adaptive histogram equalization method;

[0060] Based on the enhanced brightness V, the saturation S is corrected and equalized to generate a corrected and equalized saturation S after the correction and equalization processing;

[0061] Based on the corrected balanced saturation S, enhanced brightness V, and converted chroma H, a conversion from HSV color space to RGB color space is performed to generate an enhanced image ahead of the walking path after the conversion;

[0062] The enhanced image ahead of walking is subjected to image denoising to generate a target image ahead of walking after the image denoising.

[0063] The advantages of the present invention are as follows: in order to obtain the laser point cloud of the target ahead, the collected laser point cloud of the source ahead can be processed for point cloud data, so as to denoise and remove redundancy of the source laser power ahead; in order to obtain the image of the target ahead, the collected source image ahead can be enhanced and denoised, so as to reduce the influence of low light and high dust content in coal mines, and improve the generalization ability of obstacle detection and track detection.

[0064] The detection of the front of the coal mine underground transport locomotive based on the laser point cloud and the image of the target in front of the coal mine can effectively detect the spatial environment state of the forward direction of the coal mine underground transport locomotive compared with the detection method of a single sensor in the prior art, and can improve the obstacle detection accuracy and system robustness in the complex environment of the coal mine.

[0065] By generating driving prompt information through obstacle status information and track shape information, the operation of underground coal mine transport locomotives can be regulated and controlled to improve the stability and reliability of underground coal mine transport locomotive operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of an embodiment of the assisted driving of the present invention.

[0067] Figure 2 The present invention is a flowchart of an embodiment of performing image enhancement processing on a source image ahead of walking.

[0068] Figure 3 The figure is a schematic diagram of an embodiment of the present invention for performing obstacle detection using an obstacle detection model.

[0069] Figure 4 A schematic diagram of an embodiment of the point cloud-image feature cross-focus linear attention fusion module of the present invention. DETAILED DESCRIPTION

[0070] The present invention will be further described below in conjunction with specific drawings and embodiments.

[0071] In order to effectively detect the spatial environment state of the forward direction of the coal mine underground transport locomotive, regulate the operation of the coal mine underground transport locomotive, and improve the stability and reliability of the operation of the coal mine underground transport locomotive, the present invention provides an assisted driving method suitable for the coal mine underground transport locomotive. Specifically, the assisted driving method includes:

[0072] Real-time acquisition of spatial environment information of the forward direction of the underground transport locomotive in the coal mine, wherein the acquired spatial environment information includes a laser point cloud of the target ahead and an image of the target ahead, and the laser point cloud of the target ahead and the image of the target ahead are in a time-aligned state;

[0073] Loading the target laser point cloud and the target image ahead of the vehicle into the obstacle detection model, so as to output corresponding obstacle status information through the obstacle detection model, wherein the obstacle status information includes whether there is an obstacle that affects the vehicle ahead or not;

[0074] Loading the target image ahead of the walking into the track detection model, so as to output corresponding track shape information through the track detection model, wherein the track shape information includes a curved track state or a straight track state;

[0075] When the obstacle status information indicates that there is an obstacle that affects the forward movement and / or the track shape information indicates a curved track state, driving prompt information is generated, wherein the driving prompt information includes deceleration prompt information for controlling the deceleration of the coal mine underground transport locomotive or stop prompt information for controlling the coal mine underground transport locomotive to stop.

[0076] It should be noted that the present invention is suitable for an assisted driving method of a coal mine underground transport locomotive, specifically refers to regulating the running state of a coal mine transport locomotive to improve the stability and reliability of the operation of the coal mine underground transport locomotive, wherein regulating the forward state of the coal mine underground transport locomotive generally refers to prompting the coal mine underground transport locomotive to slow down or stop, so as to improve the safety and reliability of the transportation process of the coal mine underground transport locomotive. It can be understood that the coal mine underground transport locomotive is generally driven by a driver. Under normal circumstances, the driver can drive the coal mine underground transport locomotive forward in a normal manner. When it is identified that there are obstacles that affect the forward movement and / or the track shape is a curved track state, a driving prompt information is generated so that the driver can regulate the forward state of the coal mine underground transport locomotive according to the driving prompt information. If the driving prompt information is a deceleration prompt information, the driver is prompted to decelerate the coal mine underground transport locomotive. If the driving prompt information is a stop prompt information, the driver is prompted to stop the coal mine underground transport locomotive.

[0077] In order to generate driving prompt information so that the driver can control the forward state of the coal mine underground transport locomotive, the spatial environment information of the forward direction of the coal mine underground transport locomotive should be obtained in real time. In one embodiment of the present invention, the spatial environment information obtained in real time includes at least a laser point cloud of the target ahead and an image of the target ahead, wherein the acquired laser point cloud of the target ahead and the image of the target ahead should be in a time-aligned state. Thereafter, based on the laser point cloud of the target ahead and the image of the target ahead, it is possible to detect and judge whether there are obstacles affecting the forward direction of the coal mine underground transport locomotive and the shape of the track in the forward direction, and then determine whether to generate driving prompt information so that the driver can control the forward state of the coal mine underground transport locomotive.

[0078] In order to obtain the laser point cloud of the target in front of the vehicle, a laser radar should be installed at the front of the underground mine transport locomotive. At the same time, an image acquisition device should be installed at the front of the underground mine transport locomotive to use the image acquisition device to obtain the image of the target in front of the vehicle. In the specific implementation, the laser radar can use an intrinsically safe laser radar, and the image acquisition device can use an explosion-proof camera, such as Figure 1 As shown, the intrinsically safe laser radar and explosion-proof camera should be installed at the front of the coal mine underground transport locomotive. The location and assembly method of the intrinsically safe laser radar and explosion-proof camera can be selected according to needs, and the specific method shall be based on the requirement that the intrinsically safe laser radar and explosion-proof camera can comprehensively and effectively collect the scene and target information in front of the coal mine underground transport locomotive.

[0079] After the intrinsically safe laser radar and explosion-proof camera are installed at the front of the underground transport locomotive in the coal mine, the AutoWare calibration toolbox can be used to calibrate the external and internal parameters of the intrinsically safe laser radar and explosion-proof camera; in the ROS environment, the time synchronization tool message_filters can be used to align the point cloud data collected by the intrinsically safe laser radar with the image collected by the explosion-proof camera, so that the laser point cloud of the target in front of the walking and the image of the target in front of the walking are in a time-aligned state.

[0080] It is understandable that the point cloud data collected by the intrinsically safe laser radar will have problems such as a large number of noise points and redundant point clouds. Therefore, the collected point cloud data cannot be directly used as the laser point cloud of the target ahead. Otherwise, the accuracy of generating driving prompt information will be poor. The following example illustrates the method and process of obtaining the laser point cloud of the target ahead.

[0081] In one embodiment of the present invention, when obtaining the laser point cloud of the target ahead of walking, it includes:

[0082] Acquire the time-aligned walking forward source laser point cloud, and perform point cloud data processing on the acquired walking forward source laser point cloud to generate walking forward laser point cloud processed data after point cloud data processing, wherein:

[0083] Point cloud data processing includes point cloud preprocessing and point cloud segmentation processing performed in sequence, wherein after point cloud preprocessing is performed on the forward source laser point cloud, corresponding laser point cloud preprocessed data is generated, and the point cloud preprocessing at least includes point cloud source data denoising processing;

[0084] The laser point cloud preprocessed data is subjected to point cloud segmentation processing, so as to remove the ground point cloud in the laser point cloud preprocessed data after the point cloud segmentation processing, and generate the corresponding walking front target laser point cloud.

[0085] Figure 1 An embodiment of obtaining the laser point cloud of the target ahead of walking is shown in FIG. Figure 1 It can be seen that when obtaining the target laser point cloud in front of the walking object, the source laser point cloud in front of the walking object should be collected by the intrinsically safe laser radar first. It can be seen that the source laser point cloud in front of the walking object is collected by the intrinsically safe laser radar. After that, the data timestamps of the source laser point cloud in front of the walking object and the source image in front of the walking object are aligned, and after the data timestamps are aligned, the point cloud data is processed. Among them, the source image in front of the walking object specifically refers to the image directly collected by the explosion-proof camera, and the data timestamp alignment is the time alignment mentioned above.

[0086] Figure 1An embodiment of point cloud data processing is shown in FIG. 1 , in which point cloud data processing may include point cloud preprocessing and point cloud segmentation processing, wherein when point cloud data processing is performed on the walking front source laser point cloud, point cloud preprocessing is performed first, and laser point cloud preprocessed data is generated after the point cloud preprocessing, and the point cloud preprocessing is at least point cloud data denoising processing, that is, the point cloud preprocessing of the walking front source laser point cloud at least includes point cloud data denoising processing on the walking front source laser point cloud.

[0087] When the laser radar adopts an intrinsically safe laser radar, the laser point cloud of the walking front source collected by the intrinsically safe laser radar is a three-dimensional point cloud. In one embodiment of the present invention, during the point cloud preprocessing, statistical filtering is used to perform point cloud denoising, and then, through filtering is used to filter the redundant point clouds corresponding to the X-axis and the Z-axis in the three-dimensional point cloud. Specifically:

[0088] Statistical filtering is a statistical method used to remove noise and outliers in the laser point cloud of the walking front source. By analyzing the neighborhood distribution characteristics of each point cloud point, point cloud points that obviously deviate from the neighborhood average are identified as outliers and removed. Methods for point cloud denoising using statistical filtering include:

[0089] Step 1: Import the laser point cloud of the source ahead of the walk and record it as P = {p1, p2, ..., p n}, where P is the laser point cloud of the source in front of the walk, p i represents the i-th point cloud point, n is the number of point cloud points in the laser point cloud of the source ahead, and the point cloud point p i is a three-dimensional data;

[0090] Step 2: For each point cloud point, calculate the Euclidean distance between the point cloud point and the remaining n-1 point cloud points, select k smaller neighboring points based on the calculated Euclidean distance, and calculate the average distance between the point cloud point and the k neighboring points;

[0091] Specifically, the number of domain points k can generally be selected to be 30 to 50; since the Euclidean distance between the current point cloud point and other point cloud points has been calculated, after selecting k domain points, the average distance of the Euclidean distances of the k corresponding domain points can be calculated, that is, the calculated average distance is the average of the k Euclidean distances. In addition, after selecting k neighborhood points, the current point cloud point and the corresponding k neighborhood points are used to construct a point cloud point set. At this time, the average distance calculated above is the average distance of the current point cloud point set.

[0092] Step 3: Count the average distance of all point cloud point sets, and calculate the global mean and standard deviation based on the average distance of all point cloud point sets, and calculate the outlier threshold based on the calculated global mean and standard deviation;

[0093] By adopting the method of step 2, n point cloud point sets can be obtained, and the corresponding average distances of the n point cloud point sets can be counted. Thereafter, the corresponding global average and standard deviation can be calculated using the n average distances. After calculating the global average and standard deviation, the outlier threshold can be calculated. For example, the outlier threshold can be the sum of the global average and the standard deviation of 2. The method and process of calculating the global average and standard deviation can be consistent with the prior art.

[0094] Step 4: for each point cloud point set, compare the Euclidean distance of k+1 neighboring points in the point cloud point set with the outlier threshold. If the Euclidean distance is greater than the outlier threshold, the corresponding neighboring point is regarded as an outlier.

[0095] Step 5: All outliers are determined in the manner of step 4, and then the determined outliers are deleted from the forward source laser point cloud P.

[0096] It should be noted that through filtering is a basic filtering method commonly used to process point cloud data. Its main function is to limit the range of the point cloud in a specified dimension, thereby filtering out points that are not within the range, and realizing regional focusing or noise removal of the data. Since the width of the tunnel where the underground transport locomotive of the coal mine is located is relatively narrow, the Y-axis direction may not be filtered. From the above description, it can be seen that the present invention uses through filtering to mainly filter the redundant point clouds corresponding to the X-axis and Z-axis in the three-dimensional point cloud. Specifically, according to the characteristics of through filtering, it can be seen that the filtering threshold of the X-axis and the filtering threshold of the Z-axis should be set, wherein the X-axis is the length direction of the tunnel where the underground transport locomotive of the coal mine is located, that is, the forward direction of the underground transport locomotive of the coal mine, the Y-axis is the width direction of the tunnel where the underground transport locomotive of the coal mine is located, and the Z-axis is the height direction of the tunnel where the underground transport locomotive of the coal mine is located.

[0097] It is understandable that a longer detection distance will make the point cloud sparse. Since the transportation speed of underground coal mine transport locomotives is slow and the braking distance is short, in one embodiment of the present invention, it is only necessary to perceive the spatial environment within a certain distance in front of the underground coal mine transport locomotive. Therefore, when the maximum detection distance of the intrinsically safe laser radar is 50 meters, combined with the actual spatial range of the tunnel where the underground coal mine transport locomotive is located and the volume of obstacles, the Z-axis filter threshold can be set to 2.0 meters, and the X-axis filter threshold can be set to 40 meters.

[0098] It should be understood that after determining the filter threshold of the X-axis and the filter threshold of the Z-axis, the technical means commonly used in the technical field can be used to implement through-filtering, so that the laser point cloud pre-processed data can be obtained after through-filtering.

[0099] In specific implementation, after obtaining the laser point cloud preprocessed data, point cloud segmentation processing should be performed on the laser point cloud preprocessed data. Among them, the point cloud segmentation processing mainly removes the ground point cloud in the laser point cloud preprocessed data. Thereafter, a laser point cloud of the walking target ahead can be formed based on removing the remaining point cloud.

[0100] From the above description, it can be seen that the walking front source laser point cloud is a three-dimensional point cloud. In addition, the laser point cloud preprocessed data obtained by the above point cloud preprocessing is also a three-dimensional point cloud. The walking front source laser point cloud and the laser point cloud preprocessed data generally include ground point clouds, and the ground is usually one of the most common and stable objects in most environments. Therefore, the ground point cloud needs to be segmented to reduce the amount of point cloud data.

[0101] In specific implementation, the existing commonly used point cloud segmentation methods can be used for point cloud segmentation processing, such as the random sampling consistency algorithm can be used to segment and remove the ground point cloud, and then the target laser point cloud ahead can be formed based on the remaining point cloud. Specifically, when the random sampling consistency algorithm is used for ground point cloud segmentation, the point cloud points of the laser point cloud preprocessed data are mainly used for ground fitting, and it is determined whether the point cloud points are ground point clouds. The method and process of the random sampling consistency algorithm for ground point cloud segmentation can be consistent with the existing technology, which will not be repeated here.

[0102] Since the tunnel where the underground transport locomotive is located has low ambient light intensity and a large amount of dust, the forward source image captured by the explosion-proof camera cannot be directly used as the forward target image. In order to obtain the forward target image, the forward source image should be preprocessed, and the image quality can be improved after preprocessing, that is, the quality of the forward target image should be higher than the image quality of the forward source image, thereby improving the accuracy and reliability of the assisted driving of the present invention.

[0103] In one embodiment of the present invention, when obtaining a target image ahead of walking, the process includes:

[0104] A time-aligned walking forward source image is acquired, and image preprocessing is performed on the acquired walking forward source image to generate a walking forward target image after image preprocessing, wherein:

[0105] The image preprocessing at least includes image enhancement processing and / or image denoising processing performed on the source image ahead of the walking.

[0106] Specifically, after aligning the data timestamps of the image data collected by the explosion-proof camera, a source image of the walking front can be obtained. Thereafter, the source image of the walking front can be image preprocessed, and after image preprocessing, a target image of the walking front can be obtained. In specific implementation, the image preprocessing can be at least image enhancement processing and / or image denoising processing. The following takes the image preprocessing as image enhancement processing and image denoising processing performed in sequence as an example to specifically describe the method and process of image preprocessing.

[0107] In one embodiment of the present invention, when performing image enhancement processing on the source image ahead of walking, it includes:

[0108] Convert the source image in front of you from RGB color space to HSV color space;

[0109] The converted brightness V is enhanced to generate an enhanced brightness V after the enhancement process, wherein:

[0110] The enhancement processing of the brightness V at least includes the enhancement processing using the Retinex algorithm fused with multi-scale gradient decreasing domain guided filtering, and the local contrast enhancement using the restricted contrast adaptive histogram equalization method;

[0111] Based on the enhanced brightness V, the saturation S is corrected and equalized to generate a corrected and equalized saturation S after the correction and equalization processing;

[0112] Based on the corrected balanced saturation S, enhanced brightness V, and converted chroma H, a conversion from HSV color space to RGB color space is performed to generate an enhanced image ahead of the walking path after the conversion;

[0113] The enhanced image ahead of walking is subjected to image denoising to generate a target image ahead of walking after the image denoising.

[0114] Figure 2 An embodiment of image enhancement processing for a source image ahead of walking is shown in FIG. The source image ahead of walking is generally an RGB image. Figure 2 It can be seen that when performing image enhancement processing on the source image in front of the walking, the source image in front of the walking should be converted from the RGB color space to the HSV color space. The following example illustrates the method of converting the source image in front of the walking from the RGB color space to the HSV color space. Specifically,

[0115] V←max

[0116]

[0117] Among them, r, g, and b are the grayscale distribution values ​​of a pixel in the source image in front of the walking direction in the RGB color space, with a value range of 0 to 1. Max and min are the maximum and minimum values ​​of r, g, and b respectively. V is the brightness of the HSV color space, S is the saturation of the HSV color space, and H is the chromaticity of the HSV color space.

[0118] When converting from RGB color space to HSV space, for brightness V, the maximum value among r, g, and b is taken as brightness V. In the above conversion, “←” represents the same meaning, which can be referred to here.

[0119] It should be noted that the brightness V component of the image includes two parts: one part is the influence of the reflected light on the surface of the object (i.e., the reflection component), and the other part is the brightness change caused by the change of the ambient light source or the lighting condition (i.e., the illumination component). Among them, the illumination component mainly reflects the change of the ambient light, while the reflection component contains the detailed information of the image. The Retinex algorithm using the fusion multi-scale gradient decreasing domain guided filter is used to decompose the reflection part in the brightness V, estimate the illumination component, and then use the estimated illumination component to simultaneously decompose the illumination part at multiple scales, so as to calculate the reflection component. This parallel processing method improves the estimation accuracy of the illumination and reflection components by fusing information of different scales, and achieves the effect of detail enhancement. The method and process of enhancing the brightness V using the Retinex algorithm fusion multi-scale gradient decreasing domain guided filter can be consistent with the prior art and will not be repeated here.

[0120] After enhancing the brightness V using the Retinex algorithm fused with multi-scale gradient decreasing domain guided filtering, the initial enhanced brightness V can be obtained. At this time, the local contrast and details of the image can be improved, thereby improving the clarity and contrast of the low-light image. For the initial enhanced brightness V, the limited contrast adaptive histogram equalization method is used to perform local contrast enhancement to improve image details. Specifically, after local contrast enhancement, the enhanced brightness V can be obtained.

[0121] It should be noted that when performing local contrast enhancement, the image is divided into multiple local areas (called grid blocks), and a histogram equalization operation is performed on each area, while limiting the contrast to avoid excessive enhancement. The method and process of using the limited contrast adaptive histogram equalization method to perform local contrast enhancement are consistent with the prior art and will not be repeated here.

[0122] When the brightness V is enhanced, the saturation S will change. The saturation S needs to be corrected and balanced. During the correction and balance processing, the following are obtained:

[0123] S'=S+t×(V'-V)×μ

[0124] Among them, S' is the corrected balanced saturation, V' is the enhanced brightness, V is the brightness before enhancement processing, t is a constant; μ is the adjustment coefficient, which is calculated as follows:

[0125]

[0126] Where (x, y) is the position coordinate of each pixel during the correction equalization process, Ω is the neighborhood window of θ×θ centered at (x, y); (i, j) is the coordinate of the pixel in the neighborhood window Ω; are the mean brightness and saturation in the neighborhood window Ω respectively; δ V (x,y),δ S (x, y) are the brightness variance and saturation variance of the calibration point respectively, and the size of the domain window Ω can generally be 5×5.

[0127] After obtaining the corrected balanced saturation S and enhanced brightness V in the above manner, the HSV color space-RGB color space conversion can be performed based on the corrected balanced saturation S, enhanced brightness V and the converted chromaticity H to generate an enhanced image of the walking front after the conversion. Specifically, when performing the HSV color space-RGB color space conversion, there is:

[0128]

[0129] d=v×(1-s)

[0130] z=v×(1-fs)

[0131]

[0132] Among them, h, s, v represent the hue, saturation, and brightness of a pixel in the HSV color space; mod is the modulus operation; This is the floor operation.

[0133] The above-mentioned enhanced image of the walking ahead can be subjected to image denoising to generate a walking ahead target image after the image denoising. In specific implementation, the image denoising can adopt existing commonly used denoising methods, such as using a Gamma correction algorithm to perform multi-scale image enhancement on the enhanced image of the walking ahead, mainly to improve the image detail level and contrast, especially for image enhancement under low light conditions, and finally obtain the walking ahead target image. Specifically, the method and process of performing image denoising on the enhanced image of the walking ahead using the Gamma correction algorithm can be consistent with the existing technology and will not be repeated here.

[0134] In addition, the operating environment of underground transport locomotives in coal mines contains a large amount of dust, which causes the image to contain a large amount of noise. The present invention can also use Daubechies wavelet transform to perform denoising on the enhanced image. For example, after multi-scale image enhancement is performed on the enhanced image ahead of walking by the Gamma correction algorithm, a re-enhanced image ahead of walking can be obtained. Thereafter, the re-enhanced image ahead of walking is denoised using Daubechies wavelet transform, and the denoised image is used as the target image ahead of walking.

[0135] It should be noted that due to the good frequency domain localization properties and resolution ability of the Daubechies wavelet transform, it is very suitable for denoising and removing the influence of irregular interference signals (such as dust) in the image. The method and process of using the Daubechies wavelet transform for image denoising can be consistent with the existing technology. For example, db4 can be used as the wavelet basis function, the number of decomposition layers is set to 5 layers, and the soft threshold method is used for denoising. The process of using the Daubechies wavelet transform for denoising will not be repeated here.

[0136] In one embodiment of the present invention, when obstacle detection is performed using an obstacle detection model, the method includes:

[0137] Performing point cloud feature extraction on the target laser point cloud ahead of walking, so as to generate corresponding point cloud features ahead of walking after the point cloud feature extraction, and generating point cloud query features, point cloud key features and point cloud value features based on the point cloud features ahead of walking;

[0138] Performing image feature extraction on the walking ahead target image to generate corresponding walking ahead image features after the image feature extraction, and generating image query features, image key features and image features based on the walking ahead image features;

[0139] Performing point cloud-image feature fusion on the point cloud query feature, the image value feature, and the image key feature to generate corresponding point cloud-image fusion features after the point cloud-image feature fusion;

[0140] Performing image-point cloud feature fusion on the image query feature, the point cloud value feature, and the point cloud key feature to generate corresponding image-point cloud fusion features after the image-point cloud feature fusion;

[0141] Performing feature remapping processing on the point cloud-image fusion features and the image-point cloud fusion features to generate point cloud image remapping features after the feature remapping processing;

[0142] Perform feature stitching on the point cloud image remapping features, the walking front point cloud features, and the walking front image features to form a point cloud image fusion stitching feature after feature stitching;

[0143] The point cloud image fusion and splicing features are detected and classified, and obstacle detection and classification information is generated after detection and classification, and obstacle status information is generated based on the obstacle detection and classification information.

[0144] From the above description, it can be seen that when using the obstacle detection model for obstacle detection, the laser point cloud of the target ahead and the image of the target ahead should be loaded into the obstacle detection image. In the obstacle detection model, the laser point cloud of the target ahead and the image of the target ahead are processed. Figure 3 The method and process of obstacle detection by the obstacle detection model are specifically described.

[0145] In one embodiment of the present invention, the obstacle detection model includes a point cloud feature extraction network and an image feature extraction network, wherein:

[0146] The point cloud feature extraction network is connected with the point cloud linear transformation module and the feature splicing module, and the image feature extraction network is connected with the image linear transformation module and the feature splicing module;

[0147] The point cloud linear transformation module and the image linear transformation module are adaptively connected to the point cloud-image feature cross-focus linear attention fusion module and the image-point cloud feature cross-focus linear attention fusion module respectively;

[0148] The point cloud-image feature cross-focus linear attention fusion module and the image-point cloud feature cross-focus linear attention fusion module are connected to the feature splicing module through the feature remapping module.

[0149] The feature stitching module is connected to the obstacle detection head;

[0150] When performing obstacle detection, it includes:

[0151] The point cloud feature extraction network is used to extract point cloud features from the target laser point cloud ahead of the walking object, and the corresponding point cloud features ahead of the walking object are generated after the point cloud feature extraction;

[0152] The image feature extraction network is used to extract image features of the target image ahead of the walking object, and the corresponding image features ahead of the walking object are generated after the image feature extraction;

[0153] The point cloud linear transformation module is used to perform point cloud linear transformation on the point cloud features ahead of walking, so as to generate point cloud query features, point cloud key features and point cloud value features after the point cloud linear transformation;

[0154] The image linear transformation module is used to perform image linear transformation on the image features ahead of the walking path, so as to generate image query features, image key features and image value features after the image linear transformation;

[0155] Loading the point cloud query features, image key features, and image features into the point cloud-image feature cross-focus linear attention fusion module to generate corresponding point cloud-image fusion features using the point cloud-image feature cross-focus linear attention fusion module;

[0156] Loading the image query features, point cloud value features, and point cloud key features into the image-point cloud feature cross-focus linear attention fusion module, so as to generate corresponding image-point cloud fusion features using the image-point cloud feature cross-focus linear attention fusion module;

[0157] Using a feature remapping module to perform feature remapping processing on the image-point cloud fusion feature and the point cloud-image fusion feature, so as to generate a point cloud image remapping feature after the feature remapping processing;

[0158] The feature stitching module is used to perform feature stitching on the point cloud image remapping features, the walking front point cloud features and the walking front image features, so as to generate the point cloud image fusion stitching features after feature stitching;

[0159] The obstacle detection head is used to detect and classify the point cloud image fusion and splicing features to generate detection and classification information after detection and classification.

[0160] Figure 3 In the method, a point cloud feature extraction network is used to receive the laser point cloud of the target in front of the walking object. Thereafter, the point cloud feature extraction network is used to extract features of the laser point cloud of the target in front of the walking object, thereby extracting the point cloud features in front of the walking object. In the specific implementation, the point cloud feature extraction network can adopt the PointPillars network. The PointPillars network is an efficient three-dimensional target detection network designed for point cloud data in autonomous driving scenarios. The method and process of point cloud feature extraction using the PointPillars network can be consistent with the existing technology. The method of implementing point cloud feature extraction using the PointPillars network can refer to the paper: Lang AH, Vora S, Caesar H, et al. Pointpillars: Fast encoders for object detection from point clouds [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 12697-12705; of course, the point cloud feature extraction network can also adopt other forms, which will not be given examples one by one here.

[0161] Figure 3In the method, an image feature extraction network is used to receive a target image in front of the walking object. Thereafter, the image feature extraction network is used to extract features of the target image in front of the walking object, thereby extracting features of the image in front of the walking object. In the specific implementation, the image feature extraction network can adopt a ResNet50 network. ResNet50 (Residual Network) is a deep convolutional neural network. The core innovation of ResNet50 is the introduction of a residual block. By adding skip connections, the gradient vanishing problem in the deep network is solved. The method and process of image feature extraction using the ResNet50 network can be consistent with the prior art, such as the reference paper: He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778. Of course, the image feature extraction network can also adopt other forms, which will not be listed here one by one.

[0162] Figure 3 In the method, the point cloud linear transformation module can be used to perform linear transformation on the point cloud features ahead of the walking object, so that point cloud query features, point cloud key features and point cloud value features can be generated after the linear transformation; at the same time, the image linear transformation module can be used to perform linear transformation on the image features ahead of the walking object, so that image query features, image key features and image value features can be generated after the linear transformation.

[0163] During specific implementation, the point cloud linear transformation module and the image linear transformation module can adopt the commonly used methods in the field of this technology, specifically based on being able to achieve linear transformation and obtain corresponding query features, key features and value features. For example, in the attention mechanism, Q (Query), K (Key) and V (Value) are obtained from the input features through simple linear transformation. Specifically, the input features will pass through three independent fully connected layers (also called linear layers), each layer is responsible for generating Q, K or V. These fully connected layers contain learnable weight matrices that can map the input features to different spaces, thereby achieving linear transformation.

[0164] Specifically, the query feature represents the current area or feature point of concern, which can be regarded as the "focus" or "point of interest"; the key feature represents all possible candidate areas or feature points, which can be regarded as "reference points" or "potential related areas"; the value feature represents the actual content or features of each position, which can be regarded as an "information carrier", that is, after determining that a certain area is related to the query feature, the key feature provides the specific information of the area, and the value feature is the key part used to generate the output because it contains the actual feature value.

[0165] Figure 3 In , the point cloud query feature, the image key feature and the image feature are loaded into the point cloud-image feature cross-focus linear attention fusion module, so as to generate the corresponding point cloud-image fusion feature by using the point cloud-image feature cross-focus linear attention fusion module. It can be seen that the point cloud-image fusion feature realizes the fusion of the point cloud and the image. In addition, the image query feature, the point cloud value feature and the point cloud key feature are loaded into the image-point cloud feature cross-focus linear attention fusion module, so as to generate the corresponding image-point cloud fusion feature by using the image-point cloud feature cross-focus linear attention fusion module.

[0166] It can be understood that the point cloud-image feature cross-focused linear attention fusion module and the image-point cloud feature cross-focused linear attention fusion module use different types of features when performing point cloud-image feature fusion. In specific implementation, the image-point cloud feature cross-focused linear attention fusion module and the point cloud-image feature cross-focused linear attention fusion module can adopt the same structural form, which will be explained in detail below.

[0167] Figure 3 In the figure, the point cloud-image fusion feature is the fusion feature output by the point cloud-image feature cross-focus linear attention fusion module. Similarly, the image-point cloud fusion feature is the fusion feature output by the image-point cloud feature cross-focus linear attention fusion module. Both the point cloud-image fusion feature and the image-point cloud fusion feature are loaded into the feature remapping module. After that, the feature remapping module is used to perform feature remapping and generate point cloud image remapping features.

[0168] Specifically, the feature remapping module can adopt the existing commonly used structural forms, such as the attention mechanism module, to spatially align and semantically enhance the point cloud-image fusion features and image-point cloud fusion features, to ensure that the features of different modalities are consistent at the same spatial scale and highlight important areas; the feature mapping module reallocates feature weights by dynamically adjusting the sampling point positions, and finally generates point cloud image remapping features, thereby improving the obstacle detection model's perception ability and detection accuracy for complex environments.

[0169] Figure 3In the embodiment, the output end of the feature remapping module is connected to the feature splicing module to load the point cloud image remapping features into the feature splicing module. The feature splicing module also receives the walking front point cloud features and the walking front image features. Figure 3 In the figure, the connection method of the feature stitching module with the point cloud linear transformation module and the image linear transformation module is shown. It can be understood that this connection is mainly to obtain the point cloud features and image features of the walking front. The feature stitching module can generate the point cloud image fusion stitching features.

[0170] Specifically, the feature splicing module can adopt the existing commonly used form, and can specifically realize that different features (of the same size) will splice multiple input tensors together along the channel dimension to form a new tensor, that is, the walking forward point cloud features, the walking forward image features and the point cloud image remapping features are spliced ​​in the channel dimension, and the point cloud image fusion splicing features are generated.

[0171] Figure 3 In the process, the point cloud image fusion splicing features are sent to the obstacle detection head for positioning and classification. The obstacle detection head can adopt the existing commonly used forms, such as including multiple convolutional layers and detection and classification fully connected layers. The form adopted by the obstacle detection head is based on the positioning and classification of obstacle detection. Specifically, the obstacle detection head can predict the position, size and direction of the obstacle in the space in front of the underground transport locomotive in the coal mine through regression processing; thereafter, after classification processing, the detection and classification information output by the obstacle detection head may include the 3D position, size, direction and category of the obstacle.

[0172] Specifically, the categories of obstacles may include miners, locomotives and / or irregular stones, the 3D position of the obstacle specifically refers to the spatial position coordinates of the obstacle in front of the underground coal mine transport locomotive, the size of the obstacle specifically refers to the physical size of the obstacle, the direction of the obstacle specifically refers to the direction of the obstacle in front of the underground coal mine transport locomotive, and the confidence is the probability that the obstacle is of the current category. In specific implementation, when the obstacle detection head outputs multiple obstacle categories at the same time, the current obstacle can be determined by the confidence of each category.

[0173] As can be seen from the above description, based on the detection classification information output by the obstacle detection classification, the obstacle state information can be further determined. For example, the category, 3D position, size and direction of the obstacle can be determined based on the detection classification information, and then the position state with the coal mine underground transport locomotive workshop can be judged, thereby determining whether the obstacle will affect the normal movement of the coal mine underground transport locomotive. When it is judged that a collision will occur, the current obstacle can be an obstacle that affects the forward movement, otherwise, the current obstacle is an obstacle that does not affect the forward movement. Therefore, the obstacle state information output by the obstacle detection model can be: there is an obstacle that affects the forward movement or there is no obstacle that affects the forward movement.

[0174] From the above description, it can be seen that the point cloud-image feature cross-focus linear attention fusion module and the image-point cloud feature cross-focus linear attention fusion module can adopt the same structural form. The following takes the point cloud-image feature cross-focus linear attention fusion module as an example and combines Figure 4 Provide specific instructions.

[0175] In one embodiment of the present invention, the point cloud-image feature cross-focus linear attention fusion module includes a query feature processing unit and a key feature processing unit, wherein:

[0176] The query feature processing unit receives the point cloud query feature and is connected to the first input terminal of the matrix multiplier, wherein the query feature processing unit at least performs activation and focusing processing on the point cloud query feature to generate a point cloud post-activation and focusing query feature after the activation and focusing processing;

[0177] The key feature processing unit receives the image key feature and is connected to the second input terminal of the matrix multiplier, wherein the key feature processing unit at least performs activation and focusing processing on the image feature to generate an image activated and focused key feature after the activation and focusing processing;

[0178] The third input terminal of the matrix multiplier receives the image value feature, and the output terminal of the matrix multiplier is connected to the feature point remapping module, wherein the matrix multiplier first multiplies the image value feature with the key feature after image activation and focus, and then multiplies it with the query feature after point cloud activation and focus;

[0179] The feature point remapping module is connected to the feature map flattening processing module through a depthwise separable convolution, the feature map flattening processing module is connected to a first input end of a feature element adder, and a second input end of the feature element adder receives a point cloud query feature;

[0180] The output of the feature element adder is connected to the attention fusion fully connected layer.

[0181] Figure 4An embodiment of a query feature processing unit is shown in FIG. 1 , specifically, the query feature processing unit includes a query feature Mish activation function and a query feature focusing function, wherein:

[0182] The received point cloud query features are processed in sequence using the query feature Mish activation function and the query feature focusing function, and after being processed by the query feature focusing function, a point cloud activated and focused query feature is generated.

[0183] Figure 4 In the figure, Mish1 is the query feature Mish activation function, and JJS1 is the query feature focusing function. The query feature Mish activation function and the query feature focusing function can adopt existing commonly used forms. Among them, the Mish activation function can maintain the gradient flow in a wider input range, avoid the gradient vanishing problem, and increase the nonlinear expression ability of the feature; the focusing function gives priority to the key point information in the feature to reduce the interference of background noise and redundant features.

[0184] Figure 4 An embodiment of a key feature processing unit is also shown in Figure 4 It can be seen that the key feature processing unit can adopt the same form as the query feature processing unit, that is, the key feature processing unit also includes a key feature Mish activation function and a key feature focusing function. In the figure, Mish2 is the key feature Mish activation function, and JJS2 is the key feature focusing function. The key feature processing unit receives the image key feature, and processes it in turn using the key feature Mish activation function and the key feature focusing function, and generates the key feature after the image is activated and focused through the key feature focusing function.

[0185] Figure 4 In the process, the query feature focusing function and the key feature focusing function are connected to the matrix multiplier. Since the image value feature needs to retain the original semantic information as much as possible to provide rich details for use after the matrix operation, if it is activated or focused, the feature may lose some details, resulting in the loss of information in the fusion result. Therefore, the present invention directly connects the image value feature to the third input terminal of the matrix multiplier.

[0186] The matrix multiplier can adopt the existing commonly used multiplier. Figure 4 The JZCF in the figure is a matrix multiplier. The matrix multiplier transmits the result of the matrix multiplication calculation to the feature point mapping module. After that, the result of the matrix multiplier calculation is processed in turn by the depth-separable convolution and feature map flattening processing modules. After that, the corresponding elements are accumulated by the feature element adder. Figure 4The YSJF in the figure is the feature element adder, which is connected to the attention fusion fully connected layer. The attention fusion fully connected layer serves as the output layer of the point cloud-image feature cross-focused linear attention fusion module, that is, the point cloud-image fusion feature can be output through the attention fusion fully connected layer.

[0187] The working process of the point cloud-image feature cross-focused linear attention fusion module is as follows: first, the query feature processing unit receives the point cloud query features, performs activation and focusing processing on the point cloud query features, and strengthens the important parts of the point cloud features through the attention mechanism, thereby generating point cloud activated and focused query features to capture spatial key information.

[0188] The key feature processing unit receives the image key feature, performs similar activation and focusing processing on the received image key feature, and generates the image activated and focused key feature, aiming to enhance the global feature representation capability through the local details of the image.

[0189] Subsequently, the query features after activation and focusing of the point cloud and the key features after activation and focusing of the image are input into the matrix multiplier. The matrix multiplier first multiplies the image value features with the key features after activation and focusing of the image to establish the intrinsic relationship between the image features, and then multiplies them with the query features after activation and focusing of the point cloud, thereby realizing the cross-correlation and fusion of the point cloud and image features.

[0190] The fused features are passed from the output of the matrix multiplier to the feature point remapping module, which redistributes the features in space and channels to adjust the key areas of feature expression. Next, the remapped features enter the depthwise separable convolution module, which decomposes the spatial convolution and channel convolution operations, greatly reducing the computational complexity while enhancing the spatial expression capabilities of the features.

[0191] After the depthwise separable convolution, the features are passed to the feature map flattening processing module, which maps the high-dimensional features into a low-dimensional form for subsequent operations. Subsequently, the flattened results are fused with the original point cloud query features through the feature element adder, which further retains the key feature information of the point cloud and the image and enhances the complementarity. Finally, the fused features are passed to the attention fusion fully connected layer, which further performs nonlinear transformation and fusion of the features to obtain a deep joint representation across modalities. It can be seen that through multiple activations of focus and linear attention calculations, efficient complementarity and deep fusion of point cloud and image features are achieved, enhancing multimodal perception capabilities.

[0192] right Figure 3 And the above obstacle detection model, the method for constructing the obstacle detection model may include:

[0193] Constructing an obstacle detection basic model and constructing an obstacle detection basic model training data set for training the obstacle detection basic model, wherein the obstacle detection basic model training data set includes a plurality of training samples, and any training sample includes a training laser point cloud and a training image;

[0194] The model training parameters of the obstacle detection basic model are configured, and the obstacle detection basic model training data set is used to perform model training on the obstacle detection basic model. When the training reaches the target training state, an obstacle detection model is formed based on the obstacle detection basic model.

[0195] It is understandable that the constructed obstacle detection basic model should be consistent with the above obstacle detection model. The situation of the obstacle detection basic model can be referred to Figure 3 As well as the description of the above obstacle detection model, it will not be repeated here.

[0196] The obstacle detection basic model training data set can be constructed in a manner commonly used in the technical field. The following example illustrates the method of constructing the obstacle detection basic model training data set. Specifically, 6000 frames of images and 6000 frames of laser point cloud data are collected, totaling 12,000 frames, as sample source data. Specifically, the sample source data should cover a variety of complex lane environments, and one frame of image corresponds to one frame of laser point cloud data. The correspondence between image and laser point cloud data specifically refers to being collected at the same time and field of view and satisfying the time alignment state.

[0197] In order to ensure the validity and accuracy of the data, the fusion annotation tool is used to accurately annotate the key elements in the laser point cloud data. During the annotation, three types of objects are focused on: workers, electric locomotives, and irregular stones. In particular, the annotation of workers, as the key object of detection, has become the top priority of this annotation work. After detailed annotation work, the result of one annotation can be: a total of 10,800 3D worker instances, 4,670 3D electric locomotive instances, and 8,900 3D irregular stone instances are annotated. Specifically, the 3D here specifically refers to the annotation box in the laser point cloud data as a 3D annotation box, and the annotation information includes spatial position, size, and direction. Among them, the spatial position refers to the center point coordinates of the target object in the point cloud data, usually expressed as x, y, and z, which defines the specific position of the object in three-dimensional space; the size refers to the length, width, and height of the target object, which is used to describe the spatial occupancy range of the object; the direction refers to the yaw angle of the annotation box, which is used to describe the directional attributes of the target in three-dimensional space.

[0198] During specific implementation, since the labeled objects are workers, electric locomotives, and irregular stones, during reasoning, the corresponding deceleration prompt information or stop prompt information can be output based on the type of obstacles identified and the spatial position between the obstacles and the underground transport locomotive in the coal mine, depending on the impact on the forward movement of the underground transport locomotive in the coal mine.

[0199] After completing the annotation of the laser point cloud data, the image also needs to be annotated. After the annotation is completed, the annotation format is converted to the KITTI dataset format. Specifically, the method of annotating the image can be consistent with the prior art, which will not be repeated here. It should be noted that when annotating the image, the annotation method is a two-dimensional standard, that is, the annotation box in the image is a 2D annotation box, and the annotation target must be annotated with the corresponding point cloud aligned with the timestamp. The standard information includes the position, width and height of the target object on the image plane, where the position is represented by the upper left corner coordinates of the annotation box, and the width and height define the boundary range of the annotation box in the image. This annotation method can intuitively describe the distribution of the target object in the image.

[0200] Specifically, after the above-mentioned annotation, a training sample is formed based on a frame of annotated laser point cloud data and a corresponding frame of image. Of course, the 6000 frames of laser point cloud data and 6000 frames of images formed after annotation can all be used as corresponding training samples. At this time, 6000 training samples can be obtained. In order to ensure the generalization performance and robustness of the obstacle detection model, the laser point cloud data can be partially processed by the above-mentioned point cloud data, and the above-mentioned image preprocessing can be performed on some of the above-mentioned images. For example, the ratio of the number of laser point cloud data that has undergone point cloud data preprocessing to the number of laser point cloud data that has not undergone point cloud data preprocessing can be 73: the ratio of the number of images that have undergone image preprocessing to the number of images that have not undergone image preprocessing should also be 7:3, thereby generating corresponding training samples. Of course, the proportion is divided into training samples, verification samples and test samples, which can be selected according to needs.

[0201] Configure the model training parameters of the obstacle detection basic model, which may include: using the Adam optimizer, the batch size is 1, the maximum training round is 80 rounds, the initial learning rate is set to 0.005, and the OneCycle strategy is used to dynamically adjust the learning rate; of course, the model training parameters may also include other necessary parameters or optional parameters, which can be selected according to needs to meet the training requirements of the obstacle detection basic model.

[0202] After the obstacle detection basic model is trained using all the training samples in the obstacle detection basic model training data set, it is a round of training. After that, the training loss should be calculated. In one embodiment of the present invention, the loss function used to calculate the training loss can be a mixed loss, which can be specifically:

[0203]

[0204] Among them, L total is the total loss of model training, L FL is the classification loss, L SmoothL1 is the regression loss, L BCE is the direction prediction loss, a1 is the weight of the classification loss, a2 is the weight of the regression loss, a3 is the weight of the direction prediction loss, M is the number of training samples in the obstacle detection basic model training data set, and λ i is the factor used to balance positive and negative samples, p t,i,j is the predicted probability of the target category. D is the number of categories, and NM is the number of target regression information.

[0205] Specifically, the initial values ​​of the weight a1 of the classification loss, the weight a2 of the regression loss, and the weight a3 of the direction prediction loss should be set according to the characteristics of the task and the importance of each task. Usually, the classification task is crucial for the learning of the obstacle detection model. Therefore, the initial value of the weight a1 of the classification loss can be set to a higher value of 1.0 to ensure that the obstacle detection model can effectively distinguish the foreground and background; and the initial values ​​of the weight a2 of the regression loss and the weight a3 of the direction prediction loss can be set to smaller values, such as 0.1. This is because the impact on the final result may be small in the early stage of training, especially when the positioning and direction information of the target in the data is more complex, and too high weights may lead to deviations in the training process.

[0206] During the model training process, an adaptive adjustment method can be used to achieve dynamic adjustment of weights. In this case, a dynamic adjustment strategy based on the rate of change of the loss function is used to automatically adjust the weights by monitoring the changes of each loss function during the model training process, so as to balance the contribution of each task during the model training process and ultimately improve the overall performance. In terms of specific implementation, a weight update mechanism based on performance feedback can be used to dynamically adjust the weight of each loss as the training process progresses, thereby optimizing the overall performance of the model. The method and process of the weight update mechanism based on performance feedback can be consistent with the existing technology and will not be repeated here.

[0207] Factor λ to balance positive and negative samples i The initial value of can be set to 0.25, and a dynamic adjustment strategy is adopted during the training process. The specific adjustment strategy can be set as follows: after several training cycles, monitor the changes in the classification errors of positive and negative samples in the loss function. If the classification error of negative samples is significantly reduced, the weight of negative samples is gradually reduced, and vice versa. Through this dynamic adjustment, the model can gradually optimize the sample weights during the training process and improve the classification accuracy of the model.

[0208] In the classification loss, pt,i,j represents the predicted probability of the target category. Specifically, the obstacle detection basic model predicts the probability value of the i-th sample in the j-th category under the i-th training sample; the predicted probability of the target category p t,i,j It reflects the confidence of the obstacle detection basic model in this category. In specific implementation, the original output of the obstacle detection basic model network is converted into a probability distribution by using the softmax activation function in the obstacle detection head at the output layer of the obstacle detection basic model. That is, the predicted probability p of the target category can be obtained through the softmax activation function in the obstacle detection head. t,i,j .

[0209] When the target regression information includes spatial position, size, category and direction, the value of the target regression information quantity NM is 6, that is, the regression loss is mainly responsible for the two corresponding regression components of spatial position and size. The number of categories D is specifically related to the labeling of training samples in the obstacle detection basic model training data set. From the above description, it can be seen that when labeling, the labeled objects include people, electric locomotives and irregular stones. At this time, the number of categories D is 3.

[0210] In the regression loss, both the predicted value and the true value are the target components in the regression task, where represents the value of the i-th sample predicted by the obstacle detection basic model on the j-th regression component, β i,j Represents the true value of the i-th sample in the j-th regression component.

[0211] In the task of 3D object detection, the regression component may include the regression value of the spatial position, size, direction and category of the target. When determining the true value of the regression component, it can be annotated manually or automatically according to the spatial position, size, direction and category of the actual target during the data annotation stage and used as the corresponding true value. The basic obstacle detection model outputs the corresponding prediction value by learning the relationship between the input and regression components of the training data samples. That is, the predicted value can be directly obtained through the obstacle detection head In practical applications, the predicted value of the regression component target should be consistent with the actual physical properties of the target object, and the category regressor represents the category information to which the target belongs.

[0212] For the regression loss L SmoothL1 , To calculate the predicted value and the true value β i,j The error between them is:

[0213]

[0214] During the training process, gt_sampling (ground truth sampling, a technique for joint enhancement of image and point cloud data, is commonly used in multimodal (such as image and point cloud) detection tasks. Its purpose is to improve the generalization ability of the model by enhancing data diversity, balancing sample categories or processing unbalanced data) is used to perform joint enhancement of images and point clouds to ensure the balance of training data. After each round of training, the corresponding total model training loss L is calculated. total When the total loss of model training is L total After stabilizing or reaching the maximum number of training rounds mentioned above, it can be considered that the target training state has been reached. After that, the total loss L of the model training can be calculated. total An optimal obstacle detection basic model is determined, and the selected obstacle detection basic model is configured as the obstacle detection model of the present invention.

[0215] In one embodiment of the present invention, the track detection model includes a lane detection network and a track shape judgment module, wherein:

[0216] The lane detection network is built based on the multi-task single-stage network YOLOP;

[0217] A lane line detection network is used to receive a target image ahead of the walking object, and a lane line detection network is used to perform track detection on the target object ahead of the walking object, so as to generate at least a binary track mask image after the track detection;

[0218] Based on the binary track mask image, the track shape judgment module determines the shape of the track in the current forward target image and outputs the track shape information.

[0219] It should be noted that the multi-task single-stage network YOLOP is a commonly used multi-task single-stage network, which includes 1 shared encoder and 3 decoders for specific tasks, wherein the decoders of each task are target detection part, drivable area segmentation, lane line segmentation (wherein the drivable area segmentation and lane line segmentation are finally merged into lane line detection). In order to meet the track detection of the present invention, only the lane line segmentation task in the multi-task single-stage network YOLOP can be selected, and the construction of the lane line detection network can be realized. The task of selecting and configuring the multi-task single-stage network YOLOP and the method and process of constructing the lane line detection network can be consistent with the prior art and will not be repeated here.

[0220] Since the lane detection network only selects the lane detection task in the multi-task single-stage network YOLOP, the lane detection network has a lightweight design and an efficient decoder, and can quickly and accurately complete rail detection in complex environments, meeting real-time and precision requirements. Furthermore, according to the characteristics of the lane detection task in the multi-task single-stage network YOLOP, after loading the target image in front of the walking into the lane detection network, at least a binary track mask image can be output. Thereafter, the track shape judgment module outputs the track shape information according to the shape of the track in the current target image in front of the walking in the binary mask image. Specifically, in the binary mask image, the pixels of the track part are 1, and the pixels of the rest are 0.

[0221] In one embodiment of the present invention, based on the binary track mask image, the track shape judgment module determines the shape of the track in the current forward target image, including:

[0222] Extracting a single-side track region in the binary track mask image, and performing polynomial fitting on the single-side track region to generate a single-side track fitting curve after the polynomial fitting;

[0223] Based on the shape of the unilateral track fitting curve, the shape of the track in the current forward target image is determined and the track shape information is output.

[0224] It should be noted that the underground transport locomotive of a coal mine runs on the track in the underground tunnel of the coal mine. The track of the underground transport locomotive of a coal mine generally has track areas on both sides, that is, the underground transport locomotive of a coal mine runs on double tracks, and the binary track mask generally includes track areas on both sides. Since the shapes of the track areas on both sides are generally the same, when determining the track shape, only the single-side track area in the binary track mask can be used.

[0225] During specific implementation, the commonly used technical means in the field of this technology can be used to extract the single-sided track area in the binary track mask image. Thereafter, the extracted single-sided track area can be subjected to polynomial fitting, and a single-sided track fitting curve can be generated after the polynomial fitting. Specifically, during the polynomial fitting, for the extracted single-sided track area, the boundary points on one side of the single-sided track area are extracted, and the least squares method is used to perform polynomial fitting on all the boundary points. The method and process of extracting boundary points and performing polynomial fitting on all boundary points using the least squares method can be consistent with the prior art and will not be repeated here.

[0226] For the obtained single-sided track fitting curve, the shape of the single-sided track fitting curve can be determined. If the curvature of the single-sided track fitting curve is close to 0, then the single-sided track fitting curve is close to a first-order linear function. At this time, it can be determined that the shape of the track in the current walking front target image is a straight track. If the shape of the single-sided track fitting curve has obvious nonlinear changes, such as a quadratic curve shape, then it can be determined that the shape of the track in the current walking front target image is a curved track. In specific implementation, when it is determined that the shape of the track in the current walking front target image is a straight track, the output track shape information is a straight track state; when it is determined that the shape of the track in the current walking front target image is a curved track, the output track shape information is a curved track state.

[0227] It can be understood that in order to obtain a track detection model, a track detection basic model corresponding to the track detection model should be constructed, and the track detection basic model should be trained. The situation of the track detection basic model can refer to the description of the above track detection model. When performing model training, a track detection basic model training data set should be constructed. The following examples illustrate how to construct a track detection basic model training data set and how to train the track detection basic model.

[0228] When constructing a training data set for the basic model of track detection, a feasible approach is to collect images of the rails in campus drainage holes and / or mines to obtain rail images. The acquired rail images should cover straight rails and curved rails. The acquired rail images should be collected in a low-light and dusty environment, which is highly representative and challenging. The number of acquired rail images can be selected as needed, such as 7,000.

[0229] The above-mentioned rail images are processed and corresponding training samples are formed. The processing of rail images should at least include data annotation and image size specification adjustment. When annotating data, high-resolution image annotation tools are used for pixel-level annotation to ensure that the boundaries of the rails in the rail images are clear and accurate; when adjusting the image size specification, the size of the rail images is unified to 640×640 pixels. In addition, the number of corresponding training samples of straight rails and curved rails in the training data set of the track detection basic model should be evenly distributed to ensure that good detection performance can be achieved under different track forms. Specifically, the image size of the walking ahead target image should meet the multiple of 32, and the image size of the walking ahead target image can be selected as needed.

[0230] During the training phase, the model training parameters should also be configured, such as using the Adam optimizer, setting the initial learning rate to 0.001, and using the cosine annealing strategy to dynamically adjust the learning rate. The loss function uses a hybrid loss form that combines cross entropy and Dice Loss, uses a batch size of 32, and a maximum training round of 500.

[0231] Based on the model parameters configured above, the track detection basic model is trained using the track detection basic model training data set until the target training state is reached. Generally, when the training reaches the set maximum training rounds or the value of the loss function tends to be stable, it can be considered that the target training state is reached. The method and process of model training for the track detection basic model can refer to the existing commonly used forms, which will not be described in detail here.

Claims

1. An assisted driving method suitable for underground coal mine transport locomotives, characterized in that: The assisted driving method comprises: Real-time acquisition of spatial environment information of the forward direction of the underground transport locomotive in the coal mine, wherein the acquired spatial environment information includes a laser point cloud of the target ahead and an image of the target ahead, and the laser point cloud of the target ahead and the image of the target ahead are in a time-aligned state; Loading the target laser point cloud and the target image ahead of the vehicle into the obstacle detection model, so as to output corresponding obstacle status information through the obstacle detection model, wherein the obstacle status information includes whether there is an obstacle that affects the vehicle ahead or not; Loading the target image ahead of the walking into the track detection model, so as to output corresponding track shape information through the track detection model, wherein the track shape information includes a curved track state or a straight track state; When the obstacle status information indicates that there is an obstacle that affects the forward movement and / or the track shape information indicates a curved track state, driving prompt information is generated, wherein the driving prompt information includes deceleration prompt information for controlling the deceleration of the coal mine underground transport locomotive or stop prompt information for controlling the coal mine underground transport locomotive to stop.

2. The auxiliary driving method suitable for underground coal mine transport locomotive according to claim 1, characterized in that: When using the obstacle detection model to detect obstacles, it includes: Performing point cloud feature extraction on the target laser point cloud ahead of walking, so as to generate corresponding point cloud features ahead of walking after the point cloud feature extraction, and generating point cloud query features, point cloud key features and point cloud value features based on the point cloud features ahead of walking; Performing image feature extraction on the walking ahead target image to generate corresponding walking ahead image features after the image feature extraction, and generating image query features, image key features and image features based on the walking ahead image features; Performing point cloud-image feature fusion on the point cloud query feature, the image value feature, and the image key feature to generate corresponding point cloud-image fusion features after the point cloud-image feature fusion; Performing image-point cloud feature fusion on the image query feature, the point cloud value feature, and the point cloud key feature to generate corresponding image-point cloud fusion features after the image-point cloud feature fusion; Performing feature remapping processing on the point cloud-image fusion features and the image-point cloud fusion features to generate point cloud image remapping features after the feature remapping processing; Perform feature stitching on the point cloud image remapping features, the walking front point cloud features, and the walking front image features to form a point cloud image fusion stitching feature after feature stitching; The point cloud image fusion and splicing features are detected and classified, and obstacle detection and classification information is generated after detection and classification, and obstacle status information is generated based on the obstacle detection and classification information.

3. The auxiliary driving method for underground coal mine transport locomotive according to claim 2, characterized in that: The obstacle detection model includes a point cloud feature extraction network and an image feature extraction network. in, The point cloud feature extraction network is connected with the point cloud linear transformation module and the feature splicing module, and the image feature extraction network is connected with the image linear transformation module and the feature splicing module; The point cloud linear transformation module and the image linear transformation module are adaptively connected to the point cloud-image feature cross-focus linear attention fusion module and the image-point cloud feature cross-focus linear attention fusion module respectively; The point cloud-image feature cross-focus linear attention fusion module and the image-point cloud feature cross-focus linear attention fusion module are connected to the feature splicing module through the feature remapping module. The feature stitching module is connected to the obstacle detection head; When performing obstacle detection, it includes: The point cloud feature extraction network is used to extract point cloud features from the target laser point cloud ahead of the walking object, and the corresponding point cloud features ahead of the walking object are generated after the point cloud feature extraction; The image feature extraction network is used to extract image features of the target image ahead of the walking object, and the corresponding image features ahead of the walking object are generated after the image feature extraction; The point cloud linear transformation module is used to perform point cloud linear transformation on the point cloud features ahead of walking, so as to generate point cloud query features, point cloud key features and point cloud value features after the point cloud linear transformation; The image linear transformation module is used to perform image linear transformation on the image features ahead of the walking path, so as to generate image query features, image key features and image value features after the image linear transformation; Loading the point cloud query features, image key features, and image features into the point cloud-image feature cross-focus linear attention fusion module to generate corresponding point cloud-image fusion features using the point cloud-image feature cross-focus linear attention fusion module; Loading the image query features, point cloud value features, and point cloud key features into the image-point cloud feature cross-focus linear attention fusion module, so as to generate corresponding image-point cloud fusion features using the image-point cloud feature cross-focus linear attention fusion module; Using a feature remapping module to perform feature remapping processing on the image-point cloud fusion feature and the point cloud-image fusion feature, so as to generate a point cloud image remapping feature after the feature remapping processing; The feature stitching module is used to perform feature stitching on the point cloud image remapping features, the walking front point cloud features and the walking front image features, so as to generate the point cloud image fusion stitching features after feature stitching; The obstacle detection head is used to detect and classify the point cloud image fusion and splicing features to generate detection and classification information after detection and classification.

4. The auxiliary driving method for underground coal mine transport locomotive according to claim 3 is characterized in that: The point cloud-image feature cross-focus linear attention fusion module includes a query feature processing unit and a key feature processing unit. in, The query feature processing unit receives the point cloud query feature and is connected to the first input terminal of the matrix multiplier, wherein the query feature processing unit at least performs activation and focusing processing on the point cloud query feature to generate a point cloud post-activation and focusing query feature after the activation and focusing processing; The key feature processing unit receives the image key feature and is connected to the second input terminal of the matrix multiplier, wherein the key feature processing unit at least performs activation and focusing processing on the image feature to generate an image activated and focused key feature after the activation and focusing processing; The third input terminal of the matrix multiplier receives the image value feature, and the output terminal of the matrix multiplier is connected to the feature point remapping module, wherein the matrix multiplier first multiplies the image value feature with the key feature after image activation and focus, and then multiplies it with the query feature after point cloud activation and focus; The feature point remapping module is connected to the feature map flattening processing module through a depthwise separable convolution, the feature map flattening processing module is connected to a first input end of a feature element adder, and a second input end of the feature element adder receives a point cloud query feature; The output of the feature element adder is connected to the attention fusion fully connected layer.

5. The auxiliary driving method for underground coal mine transport locomotive according to claim 4, characterized in that: The query feature processing unit includes a query feature Mish activation function and a query feature focusing function, wherein: The received point cloud query features are processed in sequence using the query feature Mish activation function and the query feature focusing function, and after being processed by the query feature focusing function, a point cloud activated and focused query feature is generated.

6. The auxiliary driving method for underground coal mine transport locomotive according to claim 1, characterized in that: The track detection model includes a lane detection network and a track shape judgment module, wherein: The lane detection network is built based on the multi-task single-stage network YOLOP; A lane line detection network is used to receive a target image ahead of the walking object, and a lane line detection network is used to perform track detection on the target object ahead of the walking object, so as to generate at least a binary track mask image after the track detection; Based on the binary track mask image, the track shape judgment module determines the shape of the track in the current forward target image and outputs the track shape information.

7. The auxiliary driving method for underground coal mine transport locomotive according to claim 6, characterized in that: Based on the binary track mask image, the track shape judgment module determines the shape of the track in the current forward target image, including: Extracting a single-side track region in the binary track mask image, and performing polynomial fitting on the single-side track region to generate a single-side track fitting curve after polynomial fitting; Based on the shape of the unilateral track fitting curve, the shape of the track in the current forward target image is determined and the track shape information is output.

8. The auxiliary driving method for underground coal mine transport locomotive according to any one of claims 1 to 7, characterized in that: When obtaining the laser point cloud of the target ahead, it includes: Acquire the time-aligned walking forward source laser point cloud, and perform point cloud data processing on the acquired walking forward source laser point cloud to generate walking forward laser point cloud processed data after point cloud data processing, wherein: Point cloud data processing includes point cloud preprocessing and point cloud segmentation processing performed in sequence, wherein after point cloud preprocessing is performed on the forward source laser point cloud, corresponding laser point cloud preprocessed data is generated, and the point cloud preprocessing at least includes point cloud source data denoising processing; The laser point cloud preprocessed data is subjected to point cloud segmentation processing, so as to remove the ground point cloud in the laser point cloud preprocessed data after the point cloud segmentation processing, and generate the corresponding walking front target laser point cloud.

9. The auxiliary driving method for underground coal mine transport locomotive according to any one of claims 1 to 7, characterized in that: When acquiring the target image ahead of you, it includes: A time-aligned walking forward source image is acquired, and image preprocessing is performed on the acquired walking forward source image to generate a walking forward target image after image preprocessing, wherein: The image preprocessing at least includes image enhancement processing and / or image denoising processing performed on the source image ahead of the walking.

10. The auxiliary driving method for underground coal mine transport locomotive according to claim 9, characterized in that: When performing image enhancement processing on the source image ahead of walking, it includes: Convert the source image in front of you from RGB color space to HSV color space; The converted brightness V is enhanced to generate an enhanced brightness V after the enhancement process, wherein: The enhancement processing of the brightness V at least includes the enhancement processing using the Retinex algorithm fused with multi-scale gradient decreasing domain guided filtering, and the local contrast enhancement using the restricted contrast adaptive histogram equalization method; Based on the enhanced brightness V, the saturation S is corrected and equalized to generate a corrected and equalized saturation S after the correction and equalization processing; Based on the corrected balanced saturation S, enhanced brightness V, and converted chroma H, a conversion from HSV color space to RGB color space is performed to generate an enhanced image ahead of the walking path after the conversion; The enhanced image ahead of walking is subjected to image denoising to generate a target image ahead of walking after the image denoising.

Images