Automatic driving control method and device for trackless rubber-tyred vehicle

Abstract

The invention provides an automatic driving control method and device for a trackless rubber-tyred vehicle, and the method comprises the steps: collecting an infrared image and a visible light image in the driving direction of the trackless rubber-tyred vehicle, carrying out the image registration of the infrared image and the visible light image, and obtaining a fusion image; acquiring point cloud data in a view range of the driving direction of the trackless rubber-tyred vehicle, and identifying obstacle information in the view range based on the fused image and the point cloud data; determining a driving range of the trackless rubber-tyred vehicle based on the obstacle information in the visual field range; identifying indication information of traffic signs in the visual field range, and determining a driving strategy of the trackless rubber-tyred vehicle in the driving range based on the indication information, so that the trackless rubber-tyred vehicle drives according to the driving strategy; wherein the driving strategy comprises speed control, driving path planning and a steering angle. According to the invention, the trackless rubber-tyred vehicle can automatically avoid obstacles and simultaneously meet the driving requirements, so that the reliability of automatic driving control of the trackless rubber-tyred vehicle during underground driving is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an automatic driving control method and device for a trackless rubber-tyred vehicle. Background Art

[0002] The rubber-tyred trackless vehicle is an auxiliary means of transportation in coal mines. It has the characteristics of large load-bearing capacity, flexible use and high climbing ability. It is currently widely used in production and transportation in coal mines. Since the working environment of the rubber-tyred trackless vehicle is dim and dusty, when the personnel operate improperly, it is easy to cause coal mine safety accidents and major losses of personnel and equipment. In order to achieve coal mine safety management, automatic driving technology is usually applied to rubber-tyred vehicles. However, the existing automatic driving technology of rubber-tyred trackless vehicles usually uses cameras to obtain video images around the driving area, and laser radar is used as the surrounding obstacle avoidance alarm sensing device. It does not identify objects that affect driving in the entire coal mine underground environment. Therefore, the existing automatic driving technology of rubber-tyred trackless vehicles still has the problem of low reliability of automatic driving control. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide an automatic driving control method and device for a rubber-tyred trackless vehicle, which can enable the rubber-tyred vehicle to automatically avoid obstacles while complying with the driving requirements of traffic signs, thereby improving the reliability of automatic driving control of the rubber-tyred vehicle traveling underground.

[0004] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0005] In a first aspect, an embodiment of the present invention provides an automatic driving control method for a trackless rubber-tyred vehicle, comprising:

[0006] Collecting an infrared image and a visible light image of the travel direction of the rubber-tyred trackless vehicle, performing image registration on the infrared image and the visible light image, and obtaining a fused image;

[0007] Collecting point cloud data within the field of view of the rubber-tyred trackless vehicle in the driving direction, and identifying obstacle information within the field of view based on the fused image and the point cloud data;

[0008] Determining a drivable range of the rubber-tyred trackless vehicle based on obstacle information within the field of view;

[0009] Identify the indication information of the traffic signs within the field of vision, and determine the driving strategy of the rubber-tyred trackless vehicle within the drivable range based on the indication information, so that the rubber-tyred trackless vehicle drives according to the driving strategy; wherein the driving strategy includes speed control, driving path planning and steering angle.

[0010] Furthermore, the embodiment of the present invention provides a first possible implementation manner of the first aspect, wherein the step of performing image registration on the infrared image and the visible light image to obtain a fused image includes:

[0011] The infrared image and the visible light image are subjected to feature matching and image fusion based on a preset feature matching algorithm to obtain the fused image.

[0012] Further, the embodiment of the present invention provides a second possible implementation of the first aspect, wherein the point cloud data within the field of view includes point cloud data of the road surface of the coal mine underground, point cloud data of the road top and road top attachments, point cloud data of rock walls on both sides of the road and rock wall attachments, and point cloud data of obstacles in the road;

[0013] The step of identifying obstacle information within the field of view based on the fused image and the point cloud data includes:

[0014] The fused image and the point cloud data are input into a pre-trained obstacle recognition network to obtain obstacle information within the field of view; wherein the obstacle recognition network is trained based on pre-collected fused image samples and point cloud data samples.

[0015] Furthermore, an embodiment of the present invention provides a third possible implementation of the first aspect, wherein the rubber-tyred trackless vehicle is provided with a laser radar and an image sensor, the laser radar is used to collect the point cloud data, the image sensor is used to collect the infrared image and the visible light image, and the training step of the obstacle recognition network includes:

[0016] Inputting the fused image samples and the point cloud data samples into the obstacle recognition network, and performing data enhancement on the fused images and the point cloud data in the fused image samples and the point cloud data samples;

[0017] Extracting image feature information from the enhanced fused image, and generating visual features corresponding to the point cloud data based on the setting position of the laser radar, the internal and external parameters of the image sensor, and the image feature information;

[0018] A four-dimensional spatial feature matrix is ​​generated based on the visual features, three-dimensional target detection and point cloud semantic segmentation are performed based on the four-dimensional spatial feature matrix, and training loss is calculated and back-propagated until the obstacle recognition network training is completed.

[0019] Furthermore, the embodiment of the present invention provides a fourth possible implementation of the first aspect, wherein the step of generating a four-dimensional spatial feature matrix based on the visual features includes:

[0020] voxelize the visual features to obtain dense voxel features and point cloud data with enhanced data dimension;

[0021] Extracting light voxel features from the point cloud data after the data dimension enhancement to obtain semantic voxel features;

[0022] The dense voxel features and the semantic voxel features are input into the three-dimensional bird's-eye view for scattering to generate a four-dimensional spatial feature matrix.

[0023] Furthermore, the embodiment of the present invention provides a fifth possible implementation of the first aspect, wherein the step of identifying the indication information of the traffic sign within the field of view includes:

[0024] A signboard image is segmented and extracted from the visible light image, a feature vector of the signboard image is extracted based on a preset feature extraction algorithm and a pre-constructed bag-of-words model, and the feature vector is input into a trained traffic sign recognition model to obtain the indication information of the traffic sign; wherein the traffic sign recognition model is trained based on a sample image marked with the indication information of the traffic sign.

[0025] Furthermore, the embodiment of the present invention provides a sixth possible implementation of the first aspect, wherein the training step of the traffic sign recognition model includes:

[0026] Acquire a sample image including a traffic sign, and perform filtering processing on the sample image;

[0027] Performing image segmentation on the sample image after filtering to extract a signboard image;

[0028] Extracting image features from the signboard image based on a preset feature extraction algorithm, and constructing a bag-of-words model based on a preset clustering algorithm and the image features;

[0029] Determine the feature vector of the sample image based on the bag-of-words model, input the feature vector into a neural network model for model training, and obtain a trained traffic sign recognition model.

[0030] Furthermore, an embodiment of the present invention provides a seventh possible implementation of the first aspect, wherein the step of determining a driving strategy based on the indication information and the drivable range includes:

[0031] Acquiring driving requirement information of the rubber-tyred trackless vehicle, and establishing a driving environment model based on the obstacle information and the instruction information;

[0032] A plurality of groups of feasible driving strategies corresponding to the driving environment model are set based on the driving requirement information, and an optimal driving strategy is selected from the feasible driving strategies based on a reinforcement learning algorithm.

[0033] In a second aspect, an embodiment of the present invention further provides an automatic driving control device for a trackless rubber-tyred vehicle, comprising:

[0034] A first acquisition module is used to acquire an infrared image and a visible light image of the travel direction of the rubber-tyred trackless vehicle, and to perform image registration on the infrared image and the visible light image to obtain a fused image;

[0035] A second acquisition module is used to acquire point cloud data within the field of view of the rubber-tyred trackless vehicle in the driving direction, and identify obstacle information within the field of view based on the fused image and the point cloud data;

[0036] A first determination module, used to determine the drivable range of the rubber-tyred trackless vehicle based on obstacle information within the field of view;

[0037] The second determination module is used to identify the indication information of the traffic signs within the field of view, and determine the driving strategy of the rubber-tyred trackless vehicle within the drivable range based on the indication information, so that the rubber-tyred trackless vehicle can travel according to the driving strategy; wherein the driving strategy includes speed control, driving path planning and steering angle.

[0038] In a third aspect, an embodiment of the present invention provides a rubber-tyred trackless vehicle, comprising: a laser radar, an image sensor, and a controller, wherein the controller comprises a processor and a storage device;

[0039] The storage device stores a computer program, and when the computer program is executed by the processor, the method according to any one of the first aspects is executed.

[0040] An embodiment of the present invention provides an automatic driving control method and device for a trackless rubber-tyred vehicle, the method comprising: collecting an infrared image and a visible light image of the traveling direction of the trackless rubber-tyred vehicle, performing image registration on the infrared image and the visible light image to obtain a fused image; collecting point cloud data within a field of view of the traveling direction of the trackless rubber-tyred vehicle, identifying obstacle information within the field of view based on the fused image and the point cloud data; determining a drivable range of the trackless rubber-tyred vehicle based on the obstacle information within the field of view; identifying indication information of traffic signs within the field of view, and determining a driving strategy of the trackless rubber-tyred vehicle within the drivable range based on the indication information, so that the trackless rubber-tyred vehicle travels according to the driving strategy; wherein the driving strategy includes speed control, driving path planning and steering angle. The present invention can identify obstacle information within the field of view of the trackless rubber-tyred vehicle based on the fusion image of the infrared image and the visible light image and the point cloud data, and can identify objects that affect the driving of the trackless rubber-tyred vehicle. By determining the corresponding driving strategy according to the drivable range within the field of view of the trackless rubber-tyred vehicle and the traffic sign indication information, the trackless rubber-tyred vehicle can automatically avoid obstacles while meeting the driving requirements of traffic signs, thereby improving the reliability of automatic driving control of the trackless rubber-tyred vehicle driving underground.

[0041] Other features and advantages of the embodiments of the present invention will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned techniques of the embodiments of the present invention.

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flow chart of an automatic driving control method for a trackless rubber-tyred vehicle provided by an embodiment of the present invention is shown;

[0045] Figure 2a A schematic diagram of a visual feature extraction process provided by an embodiment of the present invention is shown;

[0046] Figure 2b A schematic diagram of extracting voxel features provided by an embodiment of the present invention is shown;

[0047] Figure 2cA schematic diagram of a GLPFN network structure provided by an embodiment of the present invention is shown;

[0048] Figure 2d A 3D target detection flow chart provided by an embodiment of the present invention is shown;

[0049] Figure 2e A point cloud semantic segmentation flow chart provided by an embodiment of the present invention is shown;

[0050] Figure 2f An example pattern of a sign provided by an embodiment of the present invention is shown;

[0051] Figure 3a A schematic diagram of a two-way single-lane travel of a rubber-tyred trackless vehicle provided by an embodiment of the present invention is shown;

[0052] Figure 3b A schematic diagram of a trackless rubber-tyred vehicle running on a curve provided by an embodiment of the present invention is shown;

[0053] Figure 3c A schematic diagram of a trackless rubber-tyred vehicle meeting provided by an embodiment of the present invention is shown;

[0054] Figure 4 A schematic structural diagram of an automatic driving control device for a rubber-tyred trackless vehicle provided in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be described below in conjunction with the drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0056] At present, rubber-tyred trackless vehicles are widely used in production and transportation in coal mines as an auxiliary transportation method in coal mines due to their large load-bearing capacity, flexible use, and high climbing ability. However, the dark and dusty working environment of rubber-tyred vehicles and problems with personnel operation may lead to the occurrence of coal mine safety accidents and significant losses of personnel and equipment, which is also an important aspect of coal mine safety management. The application of autonomous driving technology to rubber-tyred vehicles is an important way to solve this problem. Among them, the intelligent perception of the surrounding driving environment is the first problem to be solved. The existing autonomous driving technology mainly uses camera data to obtain video images around the driving area, and laser radar is used as a surrounding obstacle avoidance alarm perception device. Although there are also methods that use the fusion of cameras and laser radars as perception data for judgment, there is no corresponding perception strategy for multiple types of objects that affect driving in the entire environment, and a method that can give specific driving decisions.

[0057] In order to improve the above problems, an embodiment of the present invention provides an automatic driving control method and device for a trackless rubber-tyred vehicle, and the embodiment of the present invention is introduced in detail below.

[0058] This embodiment provides an automatic driving control method for a trackless rubber-tyred vehicle. Figure 1 The flowchart of the automatic driving control method of the trackless rubber-tyred vehicle shown in the figure mainly includes the following steps:

[0059] Step S102, collecting an infrared image and a visible light image of the travel direction of the rubber-tyred trackless vehicle, performing image registration on the infrared image and the visible light image, and obtaining a fused image;

[0060] The rubber-tyred trackless vehicle is provided with an image sensor, which may include an infrared camera and an RGB camera, so as to collect infrared images and visible light images in front in real time during the travel of the rubber-tyred trackless vehicle.

[0061] Based on the image registration algorithm, the infrared image and the visible light image are registered and fused into a composite image, which is recorded as a fused image.

[0062] In a specific embodiment, feature matching and image fusion can be performed on the infrared image and the visible light image based on a preset feature matching algorithm to obtain a fused image. The preset feature matching algorithm can be any one of the feature matching algorithms such as the ORB algorithm (Oriented Fast and Rotated Brief), the FAST algorithm (Features from Accelerated Segments Test), the SIFT algorithm, and the SURF feature matching algorithm, and the SURF feature matching algorithm is preferred. The infrared image and the visible light image are fused according to the registration information. Since the lights of the opposite vehicle often cause glare when the vehicles meet in the underground tunnel in the visible light image, the field of vision cannot be seen clearly, which affects the imaging effect of the image. By registering and fusing the infrared image with the visible light image, not only the impact of the glare is reduced, but also the image details are enhanced. The fused image can not only have more image features, but also have a more staggered sense of hierarchy.

[0063] Step S104, collecting point cloud data within the field of view of the rubber-tyred trackless vehicle in the driving direction, and identifying obstacle information within the field of view based on the fused image and the point cloud data;

[0064] Determine the field of view data of the rubber-tyred trackless vehicle in the driving direction, identify obstacle information within the field of view, and determine the drivable range of the rubber-tyred trackless vehicle according to the influence of the obstacles, so as to determine the corresponding driving strategy within the drivable range of the rubber-tyred trackless vehicle.

[0065] The driving environment recognition task elements = [field of view, target 1 (car), target 2 (person), target 3 (other objects), target 4 (traffic sign)]; target 1 (car), target 2 (person), target 3 (other objects) can be calculated by the spatial coordinates of the car (referring to the length, width, height, and center coordinate information), the person, and other objects within the field of view, and the three together constitute the obstacle information. Among them, the length, width, and height of the car can be determined based on the type of rubber-wheeled vehicle used in the underground mine and the size data of the type of rubber-wheeled vehicle, such as: a certain model of rubber-wheeled vehicle has a fixed length, width, and height; the length, width, and height of a person can also be calculated based on the average length, width, and height of the space occupied by workers wearing work clothes and equipment in the coal mine, which is also a priori data.

[0066] The field of view can be determined by the rock wall background and ground surface of the laser radar. The calculation formula for the field of view can be:

[0067] Field of view range = point cloud range (top = tunnel top and tunnel top attachments, left and right = rock wall and rock wall attachments, tunnel road surface, depth = point cloud clear data range depth); the depth in this formula is the preset point cloud clear range depth on a straight road, such as 50 meters, and is the depth range that the point cloud can detect at a turn or intersection.

[0068] The fused image and point cloud data are subjected to target detection and semantic segmentation to identify the location and size information of obstacles within the field of view. For example, the fused image and / or point cloud data can be input into a pre-trained neural network model to obtain information about obstacles within the field of view.

[0069] Step S106, determining the drivable range of the rubber-tyred trackless vehicle based on obstacle information within the field of view;

[0070] The obstacle information may include the position, size and speed information of the obstacle in front of the rubber-tyred trackless vehicle. Whether the obstacle will hinder the normal travel of the rubber-tyred trackless vehicle can be determined based on the position, size and speed information of the obstacle, thereby obtaining the drivable range of the rubber-tyred trackless vehicle.

[0071] The above-mentioned drivable space range is calculated by excluding obstacle targets from the field of view in the rubber-tyred vehicle driving environment. The calculation formula for the drivable range of the rubber-tyred vehicle can be: drivable range of the rubber-tyred vehicle=field of view-target (obstacle).

[0072] Step S108, identifying the indication information of the traffic signs within the field of vision, and determining the driving strategy of the rubber-tyred trackless vehicle within the drivable range based on the indication information, so that the rubber-tyred trackless vehicle can travel according to the driving strategy;

[0073] The driving strategy includes speed control, driving route planning and steering angle. The traffic signs may include traffic signs, traffic cones and traffic lights. For example, when the traffic signs indicate that the vehicle is prohibited from passing, the rubber-tyred trackless vehicle cannot perform the forward task. Since there are many types of signs in the driving regulations for rubber-tyred vehicles, such as signs prohibiting vehicles of more than 5t from passing, unloading points and U-turn points, the content information indicated by the traffic signs is very important for indicating driving strategies.

[0074] The automatic driving control method for the above-mentioned trackless rubber-tyred vehicle provided in the present embodiment can identify objects that affect the driving of the trackless rubber-tyred vehicle by identifying obstacle information within the field of view of the trackless rubber-tyred vehicle based on the fused image of the infrared image and the visible light image and the point cloud data, and can determine the corresponding driving strategy according to the drivable range within the field of view of the trackless rubber-tyred vehicle and the traffic sign indication information, so that the trackless rubber-tyred vehicle can automatically avoid obstacles while meeting the driving requirements of traffic signs, thereby improving the reliability of automatic driving control of the trackless rubber-tyred vehicle driving underground.

[0075] In one embodiment, the point cloud data within the above-mentioned field of view includes point cloud data of the road surface of the coal mine underground tunnel, point cloud data of the tunnel roof and its attachments, point cloud data of the rock walls on both sides of the tunnel and its attachments, and point cloud data of obstacles in the tunnel; this embodiment provides a specific implementation method for identifying obstacle information within the field of view based on fused images and point cloud data: the fused image and point cloud data are input into a pre-trained obstacle recognition network to obtain obstacle information within the field of view; wherein the obstacle recognition network is trained based on pre-collected fused image samples and point cloud data samples.

[0076] The fused image and point cloud data are processed based on the pre-trained obstacle recognition network to achieve semantic segmentation and three-dimensional target detection.

[0077] In one embodiment, the above-mentioned trackless rubber-tyred vehicle is provided with a laser radar and an image sensor, the laser radar is used to collect point cloud data, and the image sensor is used to collect infrared images and visible light images. This embodiment provides training steps for an obstacle recognition network, which can be specifically performed with reference to the following steps:

[0078] Step (1): inputting the fused image samples and the point cloud data samples into the obstacle recognition network, and performing data enhancement on the fused image and the point cloud data in the fused image samples and the point cloud data samples;

[0079] The obstacle recognition network may include an image autoencoder and a decoder backbone (such as a U-Net network, FCN, SegNet, etc.). The fused image samples include multiple fused images of the trackless rubber-tyred vehicle driving environment obtained by pre-collection and registration, and the point cloud data samples include point cloud data within the field of view of the trackless rubber-tyred vehicle driving direction corresponding to the multiple fused image samples. The fused image samples and point cloud data samples are both marked with the location and size information of the obstacles.

[0080] After the fused image samples and point cloud data samples are input into the obstacle recognition network, the obstacle recognition network performs data enhancement on the fused image and point cloud data. The enhanced fused image is M∈R N×3 , the enhanced point cloud data is P∈R N×4 .

[0081] Step (2): extracting image feature information from the enhanced fused image, and generating visual features corresponding to the point cloud data based on the setting position of the laser radar, the internal and external parameters of the image sensor, and the image feature information;

[0082] See Figure 2a The schematic diagram of the visual feature extraction process is shown in N×3 Perform image feature extraction, extract low-level image information (RGB pixels) X∈R based on the extracted image features and the intrinsic and extrinsic parameters of the image sensor C ’×H’×W’ , based on the original point cloud data P∈R N×4 And image information X∈R C’×H’×W’ Perform visual feature extraction and extract visual features P'∈R N×(4+3+C’) .

[0083] The enhanced fused image M∈R N×3 Input U-Net network or ResNet18 network for image feature extraction. The extracted image feature information (i.e., the extracted feature map) can be expressed as X∈R C’×H’×W’ , where C, H, W, C', H' and W' are the channels of the fused image (RGB, default is 3), the height of the fused image, the width of the fused image, the extracted feature map channels (default is 64), the height of the feature map (default is H / 2) and the width of the feature map (default is W / 2). The U-Net network is an improved Encoder and Decoder structure based on ResNet18, which makes the entire network feature extraction capability stronger and the receptive field larger.

[0084] The enhanced point cloud data P∈R N×4 And the extracted image feature information X∈R C’×H’×W’Input visual feature sampler, which can store the setting position of lidar, intrinsic parameters of image sensor and extrinsic parameters relative to lidar. Visual feature sampler extracts low-level image information (RGB pixels) X∈R based on pre-fusion algorithm C ’×H’×W’ and high-level semantic feature vectors to enhance the feature dimension of lidar point cloud data, from the original point cloud data P∈R N×4 Generate visual features P'∈R N×(4+3+C’) Step (3): Generate a four-dimensional spatial feature matrix based on visual features, perform three-dimensional target detection and point cloud semantic segmentation based on the four-dimensional spatial feature matrix, calculate the training loss and back-propagate the loss until the obstacle recognition network training is completed.

[0085] In a specific implementation, the visual features are voxelized to obtain dense voxel features and point cloud data with enhanced data dimensions;

[0086] See Figure 2b The schematic diagram of extracting voxel features shown in the figure voxelizes the data in the visual feature P' to generate dense voxel features V∈R K×M×(4+3+C’) , K is the maximum voxel value, M is the maximum number of points in each voxel. The corresponding coordinate position of the voxel is C∈R K×3 , each point in the voxel passes through x c ,y c , z c , x p ,y p , z p To expand (decorate), where the c subscript (i.e. x c ,y c , z c The subscript c) indicates that the distance between each point in the voxel and the arithmetic mean (center of mass) of the points within the voxel is calculated, and the subscript p (i.e., x p ,y p , z p The subscript p) indicates that the offset of each point to the voxel center (physical center) is calculated. Output enhanced point cloud data (i.e. point cloud data after data dimension enhancement) U∈R 4+3+C’+6 dimension.

[0087] Extract light voxel features from the point cloud data after data dimension enhancement to obtain semantic voxel features; see Figure 2c The GLPFN network structure diagram shown in the figure will enhance the point cloud data U∈R 4+3+C’+6Input the Global-Aware and Local-Aware pillar feature network (GLPFN), which uses the QKV self-attention mechanism to learn contextual features and improve the light voxel feature extraction network (GLPFN) to extract light voxel features and generate C l The semantic voxel features of dimension B∈R K×Cl .

[0088] The dense voxel features and semantic voxel features are input into the three-dimensional bird's-eye view for scattering, generating a four-dimensional spatial feature matrix. Figure 2d The 3D target detection flow chart shown in the figure converts the previously generated dense voxel features C∈R K×3 Input to the 3D bird’s eye view (BEV) and scatter to generate a 4D spatial feature matrix π∈R Cl×X×Y×Z , where X, Y, and Z represent the number of grids along the x / y / z axis, respectively.

[0089] 3D object detection: Figure 2d As shown, the four-dimensional space feature matrix π∈R Cl×X×Y×Z Input BEV feature extractor to further extract BEV features Ω∈R 6Cl×(X / 2)×(Y / 2) , the purpose is to obtain local and global semantic information at the same time, and obtain multi-scale information through the pyramid structure. 6Cl×(X / 2)×(Y / 2) Input a multi-head position and size regressor, and apply multiple detection heads to regress the position and size of three-dimensional objects. The output feature map of the multi-head network is F∈R T ×8×(X / 2)×(Y / 2) , where 8 represents the target's log(x), log(y), log(z), length, width, height, cos(θ), sin(θ) (θ is the direction angle of the target). After decoding, the 3D target detection result output is represented as O d ∈R Q×7 , where 7 represents the coordinate (x, y) value of the object center, the z value and the size (length, width, height) of the object, the direction θ and its heat map.

[0090] Point cloud semantic segmentation: In order to obtain local and contextual features at different levels of abstraction, semantic features can be combined from two aspects. The first part is the invariant features of points, see e.g. Figure 2e The point cloud semantic segmentation flowchart shown in the figure uses T-Net to transform the invariant features of the extracted points and output R∈R N×(4+3+C’) ; The output four-dimensional spatial feature matrix π∈R of the three-dimensional bird's-eye view (BEV) scattering Cl×X×Y×ZAs the input of the second part of point cloud semantic segmentation, the surrounding context semantic feature information of each point position is obtained, and the 4D spatial feature matrix π∈R is generated according to the voxel coordinates of each point Cl×X×Y×Z The surrounding features of the sampling point, cascading the point features and the surrounding context to form a fusion feature G∈R N×(4+3+C’+Cl) . Apply a multi-layer perceptron to classify the points and form a predicted category tensor O s ∈R N×(S+1) , where S is the number of categories.

[0091] Calculate the adaptive total training loss and the target detection training loss Loss detection According to a certain weight, the classification loss Loss cls and regression loss Loss loc Add and perform adaptive training. The training loss calculation formula for target detection is:

[0092] Loss detection =w cls Loss cls +w loc Loss loc

[0093] Among them, w cls is the classification loss Loss cls The corresponding weight, w loc is the regression loss Loss loc Corresponding weights. Calculate the segmentation loss of semantic segmentation, segmentation loss Loss segment It refers to the negative log-likelihood loss between the prediction and the true value. For the input prediction result x and the true value y, the NLL loss can be calculated using the logarithmic probability of the result distribution p(y|x) predicted by the model and the true value y, NLL(p(y|x), y) = -logp(y|x).

[0094] Back propagation loss uses two losses, target detection and semantic segmentation, to calculate the back propagation loss and converge the network quickly. Back propagation loss Loss back The calculation formula is:

[0095] Loss back =Loss detection +Loss segment

[0096] The weight and other parameter information of the obstacle recognition network after this round of training is saved. When the number of training times of the obstacle recognition network reaches a preset number, the training is completed to obtain the trained obstacle recognition network. In one embodiment, the performance of the trained obstacle recognition network can also be evaluated based on the verification sample to detect the obstacle recognition accuracy of the obstacle recognition network.

[0097] By identifying obstacles in the driving environment based on point cloud data and fused images, the image data and point cloud data at the same time are complementary, the expression of the environment is richer and the differences of objects are more obvious. Even in the underground mine with very complex working conditions and very dim lighting, good obstacle recognition effects can still be obtained. Compared with obstacle recognition using only point cloud data, obstacle objects and their status can be more easily discovered.

[0098] In one embodiment, this embodiment provides a specific implementation method for identifying the indication information of traffic signs within a field of view: segmenting and extracting a signboard image from a visible light image, extracting a feature vector of the signboard image based on a preset feature extraction algorithm and a pre-built bag-of-words model, and inputting the feature vector into a trained traffic sign recognition model to obtain the indication information of the traffic sign; wherein the traffic sign recognition model is trained based on sample images marked with the indication information of the traffic sign.

[0099] Due to poor lighting in coal mines, the quality of real-life images is low, the number and types are uneven, and the distribution is extremely scattered. It is difficult to collect traffic sign images, and the signs may be placed irregularly, deformed, or partially blocked. If the deep learning algorithm commonly used for image recognition is used (i.e., the image is directly input into the deep learning model for model training), the actual scene data samples available for learning may not meet the samples required for model learning.

[0100] Perform wavelet filtering and adjust the brightness of the visible light image, see Figure 2f In the example of the sign shown, since the traffic sign usually includes a border and a background of a specific color, the sign image can be segmented and extracted from the filtered visible light image based on the RGB difference relationship and the HSV color model (i.e., the image only includes the sign), and image features are extracted from the sign image based on a preset feature extraction algorithm. According to the extracted image features, all corresponding word vectors are obtained from the pre-built bag-of-words model to obtain the feature vector corresponding to the sign image. The preset feature extraction algorithm can be an image matching recognition algorithm such as a sift feature extraction algorithm or a surf feature extraction algorithm.

[0101] The above-mentioned traffic sign recognition model can be a GRNN network (generalized regression neural network), which is trained by the feature vector of the sample image marked with the indication information of the traffic sign. The feature vector corresponding to the extracted sign image is input into the trained traffic sign recognition model, and the matching calculation is performed with the real scene of the coal mine, so that the indication information of the traffic sign in the visible light image can be obtained. The indication information of the traffic sign can include the setting location of the traffic sign and the indication meaning of the traffic sign, and the indication meaning can include traffic indication information (such as no passage or U-turn), traffic requirement information (such as speed limit information), safety reminder information and location restriction information.

[0102] This embodiment processes the image first and constructs a feature vector of the image based on a feature extraction algorithm. Even if the visible light image collected underground in a coal mine is of poor light or the sign in the image is partially obscured and deformed, the indication information of the traffic sign in the image can still be accurately identified.

[0103] In one embodiment, the present embodiment provides the training steps of a traffic sign recognition model: obtaining a sample image including a traffic sign, and filtering the sample image; performing image segmentation on the filtered sample image to extract a signboard image; extracting image features from the signboard image based on a preset feature extraction algorithm, and constructing a bag-of-words model of the sample image based on a preset clustering algorithm and image features; determining a feature vector of the sample image based on the bag-of-words model, and inputting the feature vector into a neural network model for model training to obtain a trained traffic sign recognition model.

[0104] The above-mentioned sample images may include a training set and a test set. The original images in the training set are subjected to wavelet filtering and brightness adjustment. The RGB difference relationship and HSV color model are used to segment the foreground image of the signboard. Features are extracted from the images in the training set, and the SURF features of the images in the training set are calculated. The SURF features of the images are clustered into 50 categories using k-means clustering, that is, from the dictionary to K word bags. The feature vector of each image is calculated based on the word bag. A training set word bag (also called a word bag model) is constructed. After the word bag is calculated, the word bag needs to be saved. In the subsequent test set and when predicting a new image, the feature vector of the image is determined based on the word bag. The feature vector of each image in the training set is determined by the word bag, and the feature vector of each image in the training set is calculated. The feature vector is input into the traffic sign recognition model (GRNN neural network) to train the model. During the training process, the wolf pack algorithm can be used to optimize the spread parameter of the traffic sign recognition model to obtain the optimal GRNN neural network. When the traffic sign recognition model is trained, the image segmentation in the test set is used to extract the sign image, and the surf feature in the sign image is extracted. According to the feature extraction result, the feature vector of the test set image is determined from the bag of words, and the feature vector of the test set image is input into the trained traffic sign recognition model to identify the indication information of the traffic sign in the image. The identified indication information is compared with the real indication information annotated in the test set image, and the recognition accuracy of the traffic sign recognition model on the test set can be obtained.

[0105] In one embodiment, this embodiment provides a specific implementation method for determining a driving strategy of a trackless rubber-tyred vehicle within a drivable range based on indication information:

[0106] The driving requirement information of the rubber-tyred trackless vehicle is obtained, and a driving environment model is established based on obstacle information and indication information. Multiple groups of feasible driving strategies corresponding to the driving environment model are set based on the driving requirement information, and the optimal driving strategy is selected from the feasible driving strategies based on the reinforcement learning algorithm.

[0107] After completing the obstacle and traffic sign perception tasks in the driving environment, a reasonable driving strategy is made for the rubber-tyred trackless vehicle based on the obstacle information and traffic signs in the driving environment of the rubber-tyred trackless vehicle. Since there are many special driving requirements for the driving of rubber-tyred trackless vehicles in coal mines, the driving requirements may include the "Technical Measures for Safety in Rubber-tyred Vehicle Transportation", such as the safety technical measures that the driving speed of rubber-tyred trackless vehicles is usually low; since the driving environment of rubber-tyred vehicles is relatively closed and complex, there are many traffic signs unique to the coal industry on the driving roads, and accurate and applicable designs need to be made according to the special traffic sign driving specifications; see e.g. Figure 3aThe schematic diagram of two-way single-lane running of rubber-tyred trackless vehicles shown in the figure shows that in the tunnels where rubber-tyred vehicles run, two-way single-lane is usually used. The direction of vehicle travel is subject to certain restrictions. There are strict front and rear distance requirements between vehicles running normally in the same lane and in the same direction, so there is no situation of multiple lanes running in parallel. Figure 3b The schematic diagram of the trackless rubber-tyred vehicle driving on a curve is shown in Figure 3c The schematic diagram of rubber-tyred trackless vehicles meeting each other shows that the safety regulations have clear provisions on the turning speed, turning radius, meeting method, etc. Therefore, in autonomous driving, it is necessary to give a more accurate design for the steering angle and speed, which also plays a great guiding role in the control and prediction of the vehicle's operating status.

[0108] After extracting the perceptual features of the rubber-tyred trackless vehicle's driving environment, a driving environment model is established, the states and actions of driving behavior decisions are defined, and an algorithm model is designed. When completing the tasks at this stage, it is necessary to use a deep reinforcement learning algorithm to make decisions based on prior rules and output decision-making strategies, including vehicle speed control, driving path planning, obstacle avoidance, etc. under different states.

[0109] The DQN (Deep Q-network, a Q-Learing algorithm based on deep learning) and DDPG (Deep Deterministic Policy Gradient, a hybrid algorithm based on DQN and PG (Policy gradient)) reinforcement learning algorithms are used to realize the automatic driving decision of the rubber-wheeled vehicle. The network output includes continuous action control parameters (i.e., driving strategies), such as steering angle and speed. Both the DQN and DDPG algorithms are decision-making algorithms based on reinforcement learning. The DQN algorithm is aimed at decision-making problems in discrete action spaces, while the DDPG algorithm is aimed at decision-making problems in continuous action spaces. In the case of mixed state space and action space, the discrete algorithm based on DQN is difficult to fully exert its performance advantages, and the number of selected actions is relatively limited. The continuous action algorithm based on DDPG can have more advantages in action strategy and control accuracy. By integrating the DQN and DDPG algorithms, the advantages of the two algorithms are effectively utilized, and the performance and generalization ability of the model are improved through the training and optimization of deep neural networks. In terms of algorithm implementation, a hybrid network structure is adopted to combine the CNN-based DQN algorithm and the LSTM-based DDPG algorithm to obtain better generalization ability and robustness.

[0110] Obtain obstacle information and traffic sign indication information identified within the drivable range of the rubber-tyred trackless vehicle; establish a driving environment model based on the obstacle information and traffic sign indication information, which may include information such as simulated road shape, obstacle location and size, and traffic sign indication meaning.

[0111] High-dimensional state space modeling: In order to define the state and action of driving decisions, a high-dimensional state space model needs to be established to describe the vehicle's position, speed, acceleration, direction, angular velocity and other information.

[0112] Perform k random position settings on the driving environment model (i.e., set multiple sets of feasible driving strategies, such as setting the vehicle's position, speed, acceleration, direction, angular velocity and other parameter values), and store the results of the random operations in the experience replay buffer pool;

[0113] Based on the reinforcement learning algorithm, the optimal driving strategy is selected from the feasible driving strategies set, and then the actual training phase is entered. The actions are selected according to the epsilon-greedy strategy, in which some actions are obtained by the DQN network and the other part by the DDPG network; by executing the action, the next state (st+1) and the corresponding reward (reward) are obtained; (st, at, rt, st+1) are stored in the experience replay buffer pool; some samples are randomly selected from the experience replay buffer pool for training with the DQN network and the DDPG network, and the action strategy is updated using policy gradient; during training, Huberloss can be used instead of the mean square error (MSE) loss function to make the model more stable. At regular intervals or when certain conditions are met, the target network of the DDPG network is updated to make the network more stable and prevent overfitting; the Q value of the DQN network is updated and the epsilon value is reduced (epsilon-greedy technology), so that the model is more inclined to choose the best decision strategy in the later stage of training. After the training cycle, the model parameters, such as learning rate, optimizer, etc., are adjusted according to the model performance and goal completion. After training and optimization, the driving strategy is verified by testing in a simulated driving environment model or verifying in an actual environment. The final output is a set of driving strategies for trackless rubber-tyred vehicles, which include speed control, driving path planning, and obstacle avoidance in different states. In addition, since DDPG is an algorithm in a continuous action space, the output of the network includes a continuous action control parameter, such as steering angle, speed, etc.

[0114] Through the environmental perception system, more accurate information data on obstacles and traffic signs is obtained and used as input for driving decisions. The reinforcement learning algorithm is then used to help the rubber-tyred vehicle make autonomous driving decisions, further speeding up underground transportation speed and improving safety levels. This plays a very important role in the application of rubber-tyred vehicles in coal mine environments to realize autonomous driving.

[0115] The automatic driving control method of the above-mentioned trackless rubber-wheeled vehicle provided in this embodiment is to identify environmental obstacles by integrating the multimodal data of visible light, infrared camera and laser radar collected by the rubber-wheeled vehicle when driving in the coal mine tunnel, using visible light data to identify traffic signs in the driving environment, and using visible light and infrared cameras for registration and fusion to enhance the clarity of visible light images, and to obtain clearer images by registration and fusion of images that are unclear due to dust or excessive headlights. Then, the processed images are fused with the point cloud data of the laser radar to extract features, and then target detection and point cloud segmentation of multimodal data are performed to identify the objects to be identified, and corresponding driving decisions are given for the various types of objects identified by the above methods, such as: obstacle avoidance, slow driving, flashing light reminders, etc., and early warnings can be given through the on-board display screen or voice prompts, and the vehicle status can also be controlled in combination with the vehicle power device, so that the trackless rubber-wheeled vehicle can achieve reliable automatic driving management in the coal mine.

[0116] Corresponding to the automatic driving control method of the trackless rubber-tyred vehicle provided in the above embodiment, the embodiment of the present invention provides an automatic driving control device for a trackless rubber-tyred vehicle, see Figure 4 The structure diagram of an automatic driving control device for a trackless rubber-tyred vehicle is shown in FIG. 1 , and the device includes the following modules:

[0117] The first acquisition module 41 is used to acquire an infrared image and a visible light image of the travel direction of the rubber-tyred trackless vehicle, and to perform image registration on the infrared image and the visible light image to obtain a fused image;

[0118] The second acquisition module 42 is used to collect point cloud data within the field of view of the rubber-tyred trackless vehicle in the driving direction, and identify obstacle information within the field of view based on the fused image and the point cloud data;

[0119] A first determination module 43, used to determine the drivable range of the rubber-tyred trackless vehicle based on obstacle information within the field of view;

[0120] The second determination module 44 is used to identify the indication information of the traffic signs within the field of view, and determine the driving strategy of the rubber-tyred trackless vehicle within the drivable range based on the indication information, so that the rubber-tyred trackless vehicle can travel according to the driving strategy; wherein the driving strategy includes speed control, driving path planning and steering angle.

[0121] The automatic driving control device for the above-mentioned trackless rubber-tyred vehicle provided in the present embodiment can identify objects that affect the driving of the trackless rubber-tyred vehicle by identifying obstacle information within the field of view of the trackless rubber-tyred vehicle based on the fused image of the infrared image and the visible light image and the point cloud data, and can determine the corresponding driving strategy according to the drivable range within the field of view of the trackless rubber-tyred vehicle and the traffic sign indication information, so that the trackless rubber-tyred vehicle can automatically avoid obstacles while meeting the driving requirements of traffic signs, thereby improving the reliability of the automatic driving control of the trackless rubber-tyred vehicle driving underground.

[0122] In one embodiment, the first acquisition module is used to perform feature matching and image fusion on the infrared image and the visible light image based on a preset feature matching algorithm to obtain a fused image.

[0123] In one embodiment, the point cloud data within the field of view includes point cloud data of the road surface of the coal mine underground tunnel, point cloud data of the tunnel roof and its attachments, point cloud data of rock walls on both sides of the tunnel and its attachments, and point cloud data of obstacles in the tunnel; the second acquisition module 42 is used to input the fused image and point cloud data into a pre-trained obstacle recognition network to obtain obstacle information within the field of view; wherein the obstacle recognition network is trained based on pre-collected fused image samples and point cloud data samples.

[0124] In one embodiment, the trackless rubber-tyred vehicle is provided with a laser radar and an image sensor, the laser radar is used to collect point cloud data, the image sensor is used to collect infrared images and visible light images, and the second acquisition module 42 is used to input the fused image samples and the point cloud data samples into the obstacle recognition network, and perform data enhancement on the fused image and the point cloud data in the fused image samples and the point cloud data samples;

[0125] Extract image feature information from the enhanced fused image, and generate visual features corresponding to the point cloud data based on the setting position of the laser radar, the internal and external parameters of the image sensor, and the image feature information;

[0126] A four-dimensional spatial feature matrix is ​​generated based on visual features, three-dimensional target detection and point cloud semantic segmentation are performed based on the four-dimensional spatial feature matrix, and the training loss is calculated and back-propagated until the obstacle recognition network training is completed.

[0127] In one embodiment, the second acquisition module 42 is used to voxelize the visual features to obtain dense voxel features and point cloud data with enhanced data dimension;

[0128] Extract light voxel features from the point cloud data after data dimension enhancement to obtain semantic voxel features;

[0129] The dense voxel features and semantic voxel features are input into the three-dimensional bird's-eye view for scattering, generating a four-dimensional spatial feature matrix.

[0130] In one embodiment, the second determination module 44 is used to segment and extract a signboard image from a visible light image, extract a feature vector of the signboard image based on a preset feature extraction algorithm and a pre-built bag-of-words model, and input the feature vector into a trained traffic sign recognition model to obtain the indication information of the traffic sign; wherein the traffic sign recognition model is trained based on a sample image marked with the indication information of the traffic sign.

[0131] In one embodiment, the second determination module 44 is used to obtain a sample image including a traffic sign and perform filtering on the sample image;

[0132] Perform image segmentation on the sample image after filtering to extract the signboard image;

[0133] Extract image features from the signboard image based on a preset feature extraction algorithm, and build a bag-of-words model based on a preset clustering algorithm and image features;

[0134] The feature vector of the sample image is determined based on the bag-of-words model, and the feature vector is input into the neural network model for model training to obtain a trained traffic sign recognition model.

[0135] In one embodiment, the second determination module 44 is used to obtain driving requirement information of the rubber-tyred trackless vehicle and establish a driving environment model based on obstacle information and indication information;

[0136] Based on the driving requirement information, multiple groups of feasible driving strategies corresponding to the driving environment model are set, and the optimal driving strategy is selected from the feasible driving strategies based on the reinforcement learning algorithm.

[0137] The implementation principle and technical effects of the device provided in this embodiment are the same as those of the aforementioned embodiments. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0138] Corresponding to the methods and devices provided in the aforementioned embodiments, an embodiment of the present invention further provides a trackless rubber-tyred vehicle, comprising: a laser radar, an image sensor and a controller, the controller comprising a processor and a storage device; a computer program is stored on the storage device, and when the computer program is run by the processor, the automatic driving control method of the trackless rubber-tyred vehicle provided in the aforementioned embodiments is executed.

[0139] An embodiment of the present invention provides an electronic device, which includes a processor and a memory. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, the steps of the method provided in the above embodiment are implemented.

[0140] An embodiment of the present invention provides a computer-readable medium, wherein the computer-readable medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method described in the above embodiment.

[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned embodiment, and will not be repeated here.

[0142] The computer program product of the automatic driving control method and device for a trackless rubber-tyred vehicle provided in the embodiments of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the method described in the previous method embodiments. The specific implementation can be referred to the method embodiments, which will not be repeated here.

[0143] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0144] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0145] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0146] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for controlling an automatic driving of a trackless rubber-tyred vehicle, characterized in that: include: Collecting an infrared image and a visible light image of the travel direction of the rubber-tyred trackless vehicle, performing image registration on the infrared image and the visible light image, and obtaining a fused image; Collecting point cloud data within the field of view of the rubber-tyred trackless vehicle in the driving direction, and identifying obstacle information within the field of view based on the fused image and the point cloud data; Determining a drivable range of the rubber-tyred trackless vehicle based on obstacle information within the field of view; Identify the indication information of the traffic signs within the field of vision, and determine the driving strategy of the rubber-tyred trackless vehicle within the drivable range based on the indication information, so that the rubber-tyred trackless vehicle drives according to the driving strategy; wherein the driving strategy includes speed control, driving path planning and steering angle.

2. The method according to claim 1, characterized in that The step of performing image registration on the infrared image and the visible light image to obtain a fused image comprises: The infrared image and the visible light image are subjected to feature matching and image fusion based on a preset feature matching algorithm to obtain the fused image.

3. The method according to claim 1, characterized in that The point cloud data within the field of view includes point cloud data of the road surface of the coal mine underground, point cloud data of the road top and its attachments, point cloud data of the rock walls on both sides of the road and its attachments, and point cloud data of obstacles in the road; The step of identifying obstacle information within the field of view based on the fused image and the point cloud data includes: The fused image and the point cloud data are input into a pre-trained obstacle recognition network to obtain obstacle information within the field of view; wherein the obstacle recognition network is trained based on pre-collected fused image samples and point cloud data samples.

4. The method according to claim 3, characterized in that The rubber-tyred trackless vehicle is provided with a laser radar and an image sensor, the laser radar is used to collect the point cloud data, and the image sensor is used to collect the infrared image and the visible light image. The training steps of the obstacle recognition network include: Inputting the fused image samples and the point cloud data samples into the obstacle recognition network, and performing data enhancement on the fused images and the point cloud data in the fused image samples and the point cloud data samples; Extracting image feature information from the enhanced fused image, and generating visual features corresponding to the point cloud data based on the setting position of the laser radar, the internal and external parameters of the image sensor, and the image feature information; A four-dimensional spatial feature matrix is ​​generated based on the visual features, three-dimensional target detection and point cloud semantic segmentation are performed based on the four-dimensional spatial feature matrix, and training loss is calculated and back-propagated until the obstacle recognition network training is completed.

5. The method according to claim 4, characterized in that The step of generating a four-dimensional spatial feature matrix based on the visual features comprises: voxelize the visual features to obtain dense voxel features and point cloud data with enhanced data dimension; Extracting light voxel features from the point cloud data after the data dimension enhancement to obtain semantic voxel features; The dense voxel features and the semantic voxel features are input into the three-dimensional bird's-eye view for scattering to generate a four-dimensional spatial feature matrix.

6. The method according to claim 1, characterized in that The step of identifying the indication information of the traffic signs within the field of view includes: A signboard image is segmented and extracted from the visible light image, a feature vector of the signboard image is extracted based on a preset feature extraction algorithm and a pre-constructed bag-of-words model, and the feature vector is input into a trained traffic sign recognition model to obtain the indication information of the traffic sign; wherein the traffic sign recognition model is trained based on a sample image marked with the indication information of the traffic sign.

7. The method according to claim 6, characterized in that The training steps of the traffic sign recognition model include: Acquire a sample image including a traffic sign, and perform filtering processing on the sample image; Performing image segmentation on the sample image after filtering to extract a signboard image; Extracting image features from the signboard image based on a preset feature extraction algorithm, and constructing a bag-of-words model based on a preset clustering algorithm and the image features; Determine the feature vector of the sample image based on the bag-of-words model, input the feature vector into a neural network model for model training, and obtain a trained traffic sign recognition model.

8. The method according to claim 1, characterized in that The step of determining the driving strategy based on the indication information and the drivable range includes: Acquiring driving requirement information of the rubber-tyred trackless vehicle, and establishing a driving environment model based on the obstacle information and the instruction information; A plurality of groups of feasible driving strategies corresponding to the driving environment model are set based on the driving requirement information, and an optimal driving strategy is selected from the feasible driving strategies based on a reinforcement learning algorithm.

9. An automatic driving control device for a trackless rubber-tyred vehicle, characterized in that: include: A first acquisition module is used to acquire an infrared image and a visible light image of the travel direction of the rubber-tyred trackless vehicle, and to perform image registration on the infrared image and the visible light image to obtain a fused image; A second acquisition module is used to acquire point cloud data within the field of view of the rubber-tyred trackless vehicle in the driving direction, and identify obstacle information within the field of view based on the fused image and the point cloud data; A first determination module, used to determine the drivable range of the rubber-tyred trackless vehicle based on obstacle information within the field of view; The second determination module is used to identify the indication information of the traffic signs within the field of view, and determine the driving strategy of the rubber-tyred trackless vehicle within the drivable range based on the indication information, so that the rubber-tyred trackless vehicle can travel according to the driving strategy; wherein the driving strategy includes speed control, driving path planning and steering angle.

10. A trackless rubber-tyred vehicle, characterized in that: include: A laser radar, an image sensor, and a controller, wherein the controller includes a processor and a storage device; The storage device stores a computer program, which, when executed by the processor, performs the method according to any one of claims 1 to 8.

Images