Unmanned truck driving method and device, electronic equipment and storage medium

Abstract

The application provides an unmanned truck driving method and device, electronic equipment and storage medium. The method comprises the following steps: obtaining a high-precision point cloud map, determining a path point sequence of the unmanned truck and positioning data of the unmanned truck on the high-precision point cloud map; inputting the path point sequence and the positioning data of the current unmanned truck on the high-precision point cloud map into a control instruction neural network to obtain a control instruction output by the control instruction neural network; performing image processing and convolution operation on an image collected by a camera on the unmanned truck to obtain a feature vector corresponding to the image; and inputting the feature vector corresponding to the image into a driving condition neural network corresponding to the control instruction to obtain a steering angle and an acceleration output by the driving condition neural network. The scheme of the application can still realize automatic driving of the unmanned truck when the global positioning system signal coverage is poor and the driving environment changes, and the reliability of the unmanned truck driving is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the communication technology field, and particularly relates to a driverless truck driving method and device, electronic equipment and storage medium. BACKGROUND

[0002] At present, the automatic driving method of the driverless truck mainly includes a driving method depending on a global positioning system and a driving method of driving the vehicle through artificial intelligence.

[0003] The driving method depending on the global positioning system has high requirements for the continuity and stability of the global positioning system signal, and cannot meet the automatic driving conditions of the driverless truck in areas where the global positioning system signal cannot cover. The driving method of driving the vehicle through artificial intelligence has the problem of deeply nested driving strategy and driving environment, and when the driving environment changes, the driverless truck cannot adapt and transplant well.

[0004] In the above prior art, when the global positioning system signal coverage is poor and the driving environment changes, the automatic driving of the driverless truck cannot be realized. SUMMARY

[0005] The present application provides a driverless truck driving method and device, electronic equipment and storage medium, to solve the problem that the automatic driving of the driverless truck cannot be realized when there is no global positioning system signal and the driving environment changes.

[0006] In a first aspect, the present application provides a driverless truck driving method, comprising: acquiring a high-precision point cloud map, determining a path point sequence of the driverless truck and positioning data of the driverless truck on the high-precision point cloud map; wherein the path point sequence of the driverless truck is the positioning data of all points on the driving path of the driverless truck on the high-precision point cloud map; inputting the path point sequence and the positioning data of the current driverless truck on the high-precision point cloud map into a control instruction neural network to obtain a control instruction output by the control instruction neural network; wherein the control instruction includes: continue driving, left turn, right turn and straight driving; the control instruction neural network is a trained neural network; performing image processing and convolution operation on an image collected by a camera on the driverless truck to obtain a feature vector corresponding to the image; inputting the feature vector corresponding to the image into a driving condition neural network corresponding to the control instruction to obtain a steering angle and an acceleration output by the driving condition neural network; wherein the driving condition neural network corresponds one-to-one to the control instruction and is a trained neural network.

[0007] Optionally, the obtaining the high-precision point cloud map comprises: obtaining point cloud data and positioning data collected by a collection vehicle; wherein the collection vehicle is installed with a laser radar and a global positioning navigation system; the point cloud data is collected by the laser radar, and the positioning data is collected by the global positioning navigation system; the position information corresponding to the point cloud data is determined by associating the point cloud data and the positioning data based on a collection time; the position information of the point cloud data at all collection times is spliced to form a point cloud map by applying normal distribution transformation; road condition information is labeled on each position of the point cloud map, and a high-precision point cloud map is obtained; the road condition information comprises lane and intersection information.

[0008] Optionally, the method further comprises: importing the high-precision point cloud map into a pre-constructed simulation environment; the simulation environment comprises an unmanned truck simulation model loaded with a camera, a pose sensor and a laser radar; image collected by the camera, steering angle and acceleration collected by the pose sensor and point cloud data collected by the laser radar are obtained.

[0009] Optionally, the method further comprises: establishing an initial neural network model corresponding to each driving condition respectively; the driving conditions comprise continuous driving, left turn, right turn and straight driving; the images collected by the camera of the unmanned truck simulation model are divided into a continuous driving image set, a left turn image set, a right turn image set and a straight driving image set according to driving conditions; for each image in the image set, the steering angle and acceleration collected by the pose sensor at the same collection time are taken as the reference values of the steering angle and acceleration corresponding to the image; for the image set corresponding to each driving condition, each image in the image set is subjected to image processing and convolution operation to obtain a feature vector corresponding to each image, and the feature vector corresponding to each image is combined with the reference values of the steering angle and acceleration corresponding to each image to obtain an image state feature vector; the image state feature vectors corresponding to all images in the image set corresponding to each driving condition are taken as a training set corresponding to the driving condition; the initial neural network model corresponding to each driving condition is trained based on the training set corresponding to each driving condition, until a trained neural network corresponding to each driving condition is obtained, which is taken as a driving condition neural network corresponding to a corresponding control instruction.

[0010] Optionally, the method further comprises: establishing an initial control instruction neural network; the training data comprises a path point sequence of the unmanned truck simulation model, current positioning data of the unmanned truck simulation model in the high-precision point cloud map, a control instruction reference value corresponding to the path point sequence of the unmanned truck simulation model and the positioning data of the unmanned truck simulation model in the high-precision point cloud map; the initial control instruction neural network is trained based on the training data, until the control instruction neural network is obtained.

[0011] Optionally, the determining the path point sequence of the unmanned vehicle and the positioning data of the unmanned vehicle on the high-precision point cloud map comprises: determining a starting point and an ending point of the unmanned vehicle, and obtaining a driving path of the unmanned vehicle based on a path planning algorithm; determining the positioning data of each point on the driving path on the high-precision point cloud map to obtain the path point sequence; and matching the point cloud data collected by the laser radar on the unmanned vehicle with the high-precision point cloud map to determine the positioning data of the current unmanned vehicle on the high-precision point cloud map.

[0012] In a first aspect, the present application provides an unmanned vehicle driving device, comprising: a first acquisition module configured to acquire a high-precision point cloud map; a navigation module configured to determine a path point sequence of an unmanned vehicle and positioning data of the unmanned vehicle on the high-precision point cloud map; wherein the path point sequence of the unmanned vehicle is the positioning data of all points on the driving path of the unmanned vehicle on the high-precision point cloud map; a first processing module configured to input the path point sequence and the positioning data of the current unmanned vehicle on the high-precision point cloud map into a control instruction neural network to obtain a control instruction output by the control instruction neural network; wherein the control instruction comprises: continue driving, turn left, turn right, and straight driving; the control instruction neural network is a trained neural network; a second processing module configured to perform image processing and convolution operation on an image collected by a camera on the unmanned vehicle to obtain a feature vector corresponding to the image; and an automatic driving module configured to input the feature vector corresponding to the image into a driving condition neural network corresponding to the control instruction to obtain a steering angle and an acceleration output by the driving condition neural network; wherein the driving condition neural network corresponds one-to-one to the control instruction, and is a trained neural network.

[0013] Optionally, the first acquisition module comprises: an acquisition unit configured to acquire point cloud data and positioning data collected by a collection vehicle; wherein the collection vehicle is installed with a laser radar and a global positioning navigation system; the point cloud data is collected by the laser radar, and the positioning data is collected by the global positioning navigation system; an association unit configured to associate the point cloud data and the positioning data based on a collection time to determine position information corresponding to the point cloud data; a splicing unit configured to apply normal distribution transformation to splice the position information of the point cloud data at all collection times to form a point cloud map; and a labeling unit configured to label road condition information on each position on the point cloud map to obtain a high-precision point cloud map; wherein the road condition information comprises lane and intersection information.

[0014] Optionally, the device further comprises: an importing module, configured to import the high-precision point cloud map into a pre-constructed simulation environment; the simulation environment comprises an unmanned truck simulation model loaded with a camera, a pose sensor, and a laser radar; and a collecting module, configured to acquire images collected by the camera, a steering angle and an acceleration collected by the pose sensor, and point cloud data collected by the laser radar.

[0015] Optionally, the device further comprises: a first establishing module, configured to establish an initial neural network model corresponding to each driving condition respectively; the driving conditions comprise continuous driving, left turning, right turning, and straight driving; a classifying module, configured to classify images collected by the camera of the unmanned truck simulation model into a continuous driving image set, a left turning image set, a right turning image set, and a straight driving image set according to driving conditions; for each image in the image set, a steering angle and an acceleration collected by the pose sensor at the same collection time are taken as reference values of the steering angle and the acceleration corresponding to the image; a third processing module, configured to, for the image set corresponding to each driving condition, perform image processing and convolution operation on each image in the image set to obtain a feature vector corresponding to each image, and combine the feature vector corresponding to each image with the reference values of the steering angle and the acceleration corresponding to each image to obtain an image state feature vector; and take the image state feature vectors corresponding to all images in the image set corresponding to each driving condition as a training set corresponding to the driving condition; and a first training module, configured to train the initial neural network model corresponding to each driving condition based on the training set corresponding to each driving condition, until a trained neural network corresponding to each driving condition is obtained, as a driving condition neural network corresponding to a corresponding control instruction.

[0016] Optionally, the device further comprises: a second establishing module, configured to establish an initial control instruction neural network; a second acquiring module, configured to, the training data comprising a path point sequence of the unmanned truck simulation model, positioning data of the unmanned truck simulation model in the high-precision point cloud map, and a control instruction reference value corresponding to the path point sequence of the unmanned truck simulation model and the positioning data of the unmanned truck simulation model in the high-precision point cloud map; and a second training module, configured to train the initial control instruction neural network based on the training data, until the control instruction neural network is obtained.

[0017] Optionally, the navigation module is specifically configured to: determine a starting point and an ending point of the unmanned vehicle, and obtain a driving path of the unmanned vehicle based on a path planning algorithm; determine positioning data corresponding to each point on the driving path on a high-precision point cloud map to obtain the sequence of path points; and match point cloud data collected by a laser radar on the unmanned vehicle with the high-precision point cloud map to determine positioning data of the current unmanned vehicle on the high-precision point cloud map.

[0018] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory connected with the processor; the memory stores computer execution instructions; and the processor executes the computer execution instructions stored in the memory to implement the method as described above.

[0019] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the method as described above.

[0020] In the unmanned vehicle driving method, device, electronic device and storage medium provided by the present application, a high-precision point cloud map is obtained, and a sequence of path points of the unmanned vehicle and positioning data of the unmanned vehicle on the high-precision point cloud map are determined; the sequence of path points and the positioning data of the current unmanned vehicle on the high-precision point cloud map are input into a control instruction neural network to obtain a control instruction output by the control instruction neural network; an image collected by a camera on the unmanned vehicle is subjected to image processing and convolution operation to obtain a feature vector corresponding to the image; and the feature vector corresponding to the image is input into a driving condition neural network corresponding to the control instruction to obtain a steering angle and an acceleration output by the driving condition neural network. The scheme of the present application combines the control instruction network with the driving condition neural network, which can adapt to different driving environments. Based on the control instruction output by the control instruction network, the image collected by the camera of the current unmanned vehicle is input into the driving condition network corresponding to the control instruction to obtain the steering angle and the acceleration of the unmanned vehicle output by the driving condition neural network. The control instruction is output based on the sequence of path points of the unmanned vehicle and the positioning data of the current unmanned vehicle on the high-precision point cloud map input into the control instruction neural network. The sequence of path points of the unmanned vehicle and the positioning data of the current unmanned vehicle on the high-precision point cloud map are determined based on the high-precision point cloud map and are independent of global positioning system signals. Therefore, when the coverage of the global positioning system signals is poor and the driving environment changes, the automatic driving of the unmanned vehicle can still be realized, and the reliability of the unmanned vehicle driving is improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0022] Figure 1 A flowchart of the unmanned truck driving method provided in Embodiment One of the present application is shown in FIG. 1.

[0023] Figure 2 A flowchart of the unmanned truck driving method provided in Embodiment Two of the present application is shown in FIG. 2.

[0024] Figure 3 A flowchart of the unmanned truck driving method provided in Embodiment Three of the present application is shown in FIG. 3.

[0025] Figure 4 A structural diagram of the unmanned truck driving device provided in Embodiment Four of the present application is shown in FIG. 4.

[0026] Figure 5 A structural diagram of the unmanned truck driving device provided in Embodiment Five of the present application is shown in FIG. 5.

[0027] Figure 6 A structural diagram of the unmanned truck driving device provided in Embodiment Six of the present application is shown in FIG. 6.

[0028] Figure 7 A structural diagram of the unmanned truck driving system provided in Embodiment Seven of the present application is shown in FIG. 7.

[0029] Figure 8 A structural diagram of the electronic device provided in Embodiment Eight of the present application is shown in FIG. 8.

[0030] The specific embodiments have been shown and described in the foregoing drawings and specification. It will be apparent to those skilled in the art that numerous modifications, substitutions, and changes can be made in the methods and apparatuses disclosed without departing from the scope of the present application as defined in the appended claims. Accordingly, it is intended that all such modifications, substitutions, and changes be included as falling within the scope of the present application. DETAILED DESCRIPTION

[0031] The illustrative examples set forth herein will vary from to one another in different ways. Some of these examples are described in the following numbered clauses:

[0032] It should be noted that the brief description of terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0033] The terms "first", "second", and the like in the description and in the claims of the present application and above-described drawings are used for distinguishing between similar or identical objects or entities and do not necessarily mean a specific order or sequence, unless otherwise indicated (Unless otherwise indicated). It will be understood that the terms so used are interchangeable under appropriate circumstances and embodiments of the present application are meant to encompass any and all possibilities, such as those permitted under the given order or sequence given in the embodiments illustration or description of the present application.

[0034] Furthermore, the terms "comprise" and "have", and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a product or device that comprises a list of components does not necessarily comprise only those components in the list and can include additional components not expressly listed or inherent to such product or device. The term "module" used in the present application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or a combination of hardware or / and software code capable of performing a function associated with the element.

[0035] In practical applications, the application scenario of the unmanned truck in the present application can be an open-pit mine transportation scenario. In the mining process of the open-pit mine, as the mining, transportation and dumping of the ore body, the driving environment of the unmanned truck changes constantly, and the global positioning system signal coverage of part of the open-pit mine area is poor.

[0036] At present, the automatic driving technology of the unmanned truck is mainly divided into a method relying on global positioning system (Global Navigation Satellite System, GNSS) driving and a method of applying artificial intelligence to drive the vehicle automatic driving. Among them, the method relying on global positioning system driving has high requirements for the continuity and stability of the global positioning system signal, and in the area where the global positioning system signal cannot cover, the unmanned truck cannot meet the conditions of automatic driving. The method of artificial intelligence driving vehicle automatic driving has the problem of deeply embedding the driving strategy and the driving environment, and when the driving environment changes, the unmanned truck cannot adapt and transplant well.

[0037] In the embodiment of the present application, the control instruction network is combined with the driving condition neural network, which can adapt to different driving environments. Based on the control instruction output by the control instruction network, the image collected by the camera of the current unmanned truck is input into the driving condition network corresponding to the control command to obtain the steering angle and acceleration of the unmanned truck output by the driving condition neural network. The control instruction is output based on the path point sequence of the unmanned truck and the positioning data of the current unmanned truck on the high-precision point cloud map input into the control instruction neural network. The path point sequence of the unmanned truck and the positioning data of the current unmanned truck on the high-precision point cloud map are determined based on the high-precision point cloud map and are independent of the global positioning system signal. Therefore, when the global positioning system signal coverage is poor and the driving environment changes, the autonomous driving of the unmanned truck can still be realized, and the reliability of the unmanned truck driving is improved.

[0038] The technical solutions of the present application and the technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. In the description of the present application, unless otherwise explicitly specified and limited, each term should be understood in a broad sense within the art. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0039] Embodiment one

[0040] Figure 1 The flowchart of the unmanned truck driving method provided in the first embodiment of the present application, the implementation subject of the present embodiment can be an unmanned truck driving device, as shown in the figure, the method comprises: Figure 1

[0041] S101, obtaining a high-precision point cloud map, determining a path point sequence of an unmanned truck and positioning data of the unmanned truck on the high-precision point cloud map;

[0042] S102, inputting the path point sequence and the current positioning data of the unmanned truck on the high-precision point cloud map into a control instruction neural network to obtain a control instruction output by the control instruction neural network;

[0043] S103, performing image processing and convolution operation on the image collected by the camera on the unmanned truck to obtain a feature vector corresponding to the image;

[0044] S104, inputting the feature vector corresponding to the image into the driving condition neural network corresponding to the control instruction to obtain a steering angle and acceleration output by the driving condition neural network.

[0045] ​In this embodiment, the path point sequence of the unmanned vehicle is the positioning data of all points on the driving path of the unmanned vehicle on the high-precision point cloud map. It should be noted that the path point sequence of the unmanned vehicle is the positioning data of all points on the driving path of the unmanned vehicle on the high-precision point cloud map, which is planned in advance before the unmanned vehicle drives, and is used as a reference route for the unmanned vehicle to drive.

[0046] In this embodiment, the positioning data of the unmanned vehicle on the high-precision point cloud map refers to the accurate positioning of the unmanned vehicle based on the high-precision point cloud map. It should be noted that each point of the high-precision point cloud map includes position information and road condition information of the point. It can be understood that the positioning data of the unmanned vehicle on the high-precision point cloud map includes the position information and road condition information of the unmanned vehicle determined based on the high-precision point cloud data.

[0047] In actual application, the execution subject of the unmanned vehicle driving method can be an unmanned vehicle driving device. The unmanned vehicle driving device can be implemented in various ways, such as being implemented by a computer program, for example, an application software, or being implemented as a medium storing a related computer program, for example, a U disk, a cloud disk, or being implemented by an entity device integrated or installed with a related computer program, for example, a chip.

[0048] In this embodiment, the control instruction includes continue driving, left turn, right turn, and straight driving. For example, [1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], and [0, 0, 0, 1] respectively represent the four control instructions of continue driving, left turn, right turn, and straight driving. The control instruction neural network is a trained neural network. It can be understood that the control instruction neural network is used to determine the control instruction based on the path point sequence and the current positioning data of the unmanned vehicle on the high-precision point cloud map.

[0049] In this embodiment, the feature vector corresponds to the image collected by the camera on the unmanned vehicle and can uniquely represent the image collected by the camera on the unmanned vehicle. It can be understood that the feature vector represents important features of the image. In actual application, image processing refers to pre-processing the image collected by the camera on the unmanned vehicle, so as to perform convolution on the image subsequently. The image convolution can obtain the significant features of the image. Therefore, the feature vector representing the important features of the image can be obtained by image processing and convolution operation on the image collected by the camera on the unmanned vehicle.

[0050] In actual application, the driving condition neural network corresponds to the control instruction, and is a trained neural network. In an example, the driving condition neural network includes a continue driving condition neural network, a left turn driving condition neural network, a right turn driving condition neural network, and a straight driving condition neural network. Therefore, according to the control instruction, the driving condition neural network to which the feature vector corresponding to the image is input can be determined.

[0051] Optionally, a branch judgment switch is arranged, and the branch judgment switch receives the control instruction and the feature vector corresponding to the image. If the continue driving control instruction is received, the feature vector corresponding to the image is input into the continue driving condition neural network. If the left turn control instruction is received, the feature vector corresponding to the image is input into the left turn driving condition neural network. If the right turn control instruction is received, the feature vector corresponding to the image is input into the right turn driving condition neural network. If the execution control instruction is received, the feature vector corresponding to the image is input into the execution driving condition neural network.

[0052] In actual application, after the steering angle and the acceleration output by the driving condition neural network, the control unit of the unmanned truck is determined, the values of the steering wheel, the accelerator, and the brake are determined, and then the automatic driving of the unmanned truck is realized. Optionally, the steering angle and the acceleration can be input into the vehicle control module, and the control instruction output by the vehicle control module is obtained. The control instruction includes the control instruction for the steering wheel, the accelerator, and the brake.

[0053] In the embodiment, the control instruction network is combined with the driving condition neural network, which can adapt to different driving environments. Based on the control instruction output by the control instruction network, the image collected by the camera of the current unmanned truck is input into the driving condition network corresponding to the control command, and the steering angle and the acceleration of the unmanned truck output by the driving condition neural network are obtained. The control instruction is output based on the path point sequence of the unmanned truck input into the control instruction neural network and the positioning data of the current unmanned truck on the high-precision point cloud map. The path point sequence of the unmanned truck and the positioning data of the current unmanned truck on the high-precision point cloud map are determined based on the high-precision point cloud map, and do not depend on the global positioning system signal. Therefore, when the global positioning system signal coverage is poor and the driving environment changes, the automatic driving of the unmanned truck can still be realized, and the reliability of the unmanned truck driving is improved.

[0054] Optionally, regarding the acquisition process of the high-precision point cloud map, in a possible implementation manner, the high-precision point cloud map is acquired in S101, including:

[0055] The point cloud data and the positioning data collected by the vehicle are acquired.

[0056] The point cloud data and the positioning data corresponding to the same collection time are associated to determine position information corresponding to the point cloud data.

[0057] Each position on the point cloud map is labeled with road condition information to obtain a high-precision point cloud map.

[0058] The point cloud data is a data set of spatial points collected by a laser radar, and each point cloud data includes three-dimensional coordinate information. The collection vehicle is equipped with a laser radar and a global positioning navigation system. The point cloud data is collected by the laser radar, and the positioning data is collected by the global positioning navigation system. In actual application, the collection vehicle equipped with the laser radar and the global positioning navigation system is driven to travel, the point cloud data of each point of the collection vehicle is collected by the laser radar, and the positioning data of each point of the collection vehicle is collected by the global positioning navigation system.

[0059] Specifically, the point cloud data and the positioning data corresponding to the same collection time are associated to determine the positioning data corresponding to the point cloud data, i.e. the position information corresponding to the point cloud data. In actual application, the collected point cloud data is discrete data. For the point cloud data corresponding to each collection time, the positioning data corresponding to the time closest to the collection time is determined, the positioning data corresponding to the time closest to the collection time is calculated by difference, and the position information corresponding to the points near the point cloud data corresponding to the collection time is determined. For example, the difference calculation means that each point cloud data is subjected to coordinate and rotation interpolation transformation, and the extrinsic parameter of the laser radar is supplemented to obtain the basic affine transformation matrix of each point cloud data at the collection time. Through the pose information of each point cloud data and the affine transformation matrix, the point cloud coordinate system is converted to the global coordinate system of the collection vehicle.

[0060] According to the above description, the position information corresponding to the point cloud data is determined through the collection time, and the position information of the point cloud data at all collection times is spliced into a point cloud map based on normal distribution transformation.

[0061] In actual application, a map labeling software can be used to label the road condition information of each position on the point cloud map to obtain a high-precision point cloud map. The road condition information includes lane and intersection information. For example, the lane information can be the height and width of the lane, and the intersection information can be the type of intersection, such as a crossroad.

[0062] In this embodiment, the point cloud data and the positioning data of the vehicle are collected, the point cloud data and the positioning data are associated based on the collection time, the position information corresponding to the point cloud data is determined, the position information of the point cloud data at all collection times is spliced into a point cloud map by applying normal distribution transformation, the road condition information is labeled on each position on the point cloud map, and a high-precision point cloud map is obtained. The high-precision point cloud map is applied to automatic driving of the unmanned vehicle, and the reliability of unmanned vehicle driving is improved.

[0063] In a possible implementation, the determination of the sequence of path points of the unmanned vehicle and the positioning data of the unmanned vehicle on the high-precision point cloud map in S102 includes:

[0064] The starting point and the ending point of the unmanned vehicle are determined, the driving path of the unmanned vehicle is obtained based on a path planning algorithm, the positioning data of each point on the driving path on the high-precision point cloud map is determined, and the sequence of path points is obtained.

[0065] The point cloud data collected by the laser radar on the current unmanned vehicle is matched with the high-precision point cloud map, and the positioning data of the current unmanned vehicle on the high-precision point cloud map is determined.

[0066] It can be understood that there are multiple paths between the starting point and the ending point, and the optimal driving path can be obtained by the path planning algorithm. For example, the optimal driving path refers to the driving path with the shortest driving time. Optionally, the high-precision point cloud map is converted into a topological map composed of a series of nodes and edges, and the driving path of the unmanned vehicle is obtained by a search algorithm.

[0067] In actual application, after the driving path is determined, the positioning data of each point on the driving path on the high-precision point cloud map is determined, and the point cloud data corresponding to all points on the driving path is combined into the sequence of path points.

[0068] The matching of the point cloud data collected by the laser radar on the current unmanned vehicle with the high-precision point cloud map refers to the determination of the positioning data of the point cloud data collected by the laser radar on the unmanned vehicle on the high-precision point cloud map. Optionally, the matching of the point cloud data collected by the laser radar on the unmanned vehicle with the high-precision point cloud map is realized by applying an iterative closest point (ICP) algorithm, the positioning data of the current unmanned vehicle on the high-precision point cloud map is determined, and for example, the expression of the iterative closest point algorithm is as follows:

[0069] J = å ‖pi-(R*pi'+T)‖ 2

[0070] Wherein, pi' is the original point cloud, pi is the target point cloud, R is the rotation transformation, and T is the translation transformation.

[0071] In this embodiment, based on the high-precision point cloud map, the positioning data of the current unmanned vehicle on the high-precision point cloud map is obtained.

[0072] In the unmanned vehicle driving method provided in this embodiment, a high-precision point cloud map is obtained, the path point sequence of the unmanned vehicle and the positioning data of the unmanned vehicle on the high-precision point cloud map are determined, the path point sequence and the positioning data of the current unmanned vehicle on the high-precision point cloud map are input into the control instruction neural network to obtain the control instruction output by the control instruction neural network, the image collected by the camera on the unmanned vehicle is subjected to image processing and convolution operation to obtain the feature vector corresponding to the image, and the feature vector corresponding to the image is input into the driving condition neural network corresponding to the control instruction to obtain the steering angle and acceleration output by the driving condition neural network. In this embodiment, based on the combination of the control instruction network and the driving condition neural network, different driving environments can be adapted, the image collected by the camera of the current unmanned vehicle is input into the driving condition network corresponding to the control command based on the control instruction output by the control instruction network, the steering angle and acceleration of the unmanned vehicle output by the driving condition neural network are obtained, the control instruction is output based on the path point sequence of the unmanned vehicle and the positioning data of the current unmanned vehicle on the high-precision point cloud map input into the control instruction neural network, and the path point sequence of the unmanned vehicle and the positioning data of the current unmanned vehicle on the high-precision point cloud map are determined based on the high-precision point cloud map and are independent of the global positioning system signal. Therefore, when the global positioning system signal coverage is poor and the driving environment changes, the automatic driving of the unmanned vehicle can still be realized, and the reliability of the unmanned vehicle driving is improved.

[0073] Embodiment Two

[0074] Figure 2 The flowchart of the unmanned vehicle driving method provided in Embodiment Two of the present application is shown in Figure 2 The method further includes:

[0075] S201, importing the high-precision point cloud map into a pre-constructed simulation environment;

[0076] S202, obtaining the image collected by the camera, the steering angle and acceleration collected by the pose sensor, and the point cloud data collected by the laser radar;

[0077] S203, establishing a corresponding initial neural network model for each driving condition;

[0078] S204, images collected by the camera of the unmanned vehicle simulation model are divided into a continue driving image set, a left turn image set, a right turn image set and a straight driving image set according to driving conditions; for each image in the image set, the steering angle and acceleration collected by the pose sensor at the same collection time are taken as reference values of the steering angle and acceleration corresponding to the image;

[0079] S205, for each image set corresponding to a driving condition, each image in the image set is subjected to image processing and convolution operation to obtain a feature vector corresponding to each image, the feature vector corresponding to each image is combined with the reference values of the steering angle and acceleration corresponding to each image to obtain an image state feature vector; the image state feature vectors corresponding to all images in the image set corresponding to each driving condition are taken as a training set corresponding to the driving condition;

[0080] S206, based on the training set corresponding to each driving condition, an initial neural network model corresponding to each driving condition is trained until a trained neural network corresponding to each driving condition is obtained as a driving condition neural network corresponding to a corresponding control instruction.

[0081] In the embodiment, the simulation environment includes an unmanned vehicle simulation model equipped with a camera, a pose sensor and a laser radar. The camera is used to collect the road condition image of the current unmanned vehicle, the pose sensor is used to collect the steering angle and acceleration of the current unmanned vehicle, and the laser radar is used to collect the point cloud data. The simulation environment can be established based on an automatic driving platform, for example, by driving the unmanned vehicle simulation model to travel in the simulation environment to obtain the images collected by the camera, the steering angle and acceleration collected by the pose sensor and the point cloud data collected by the laser radar.

[0082] It should be noted that when the unmanned vehicle simulation model is constructed, the range of the steering angle of the unmanned vehicle simulation model, the throttle output range and the brake range need to be set. For example, the range of the steering angle is set to [-1.0, 1.0], less than 0 indicating left turn and greater than 0 indicating right turn.

[0083] In the embodiment, the driving conditions include continue driving, left turn, right turn and straight driving. The driving conditions correspond one-to-one to the control instructions, and the initial neural network model established based on each driving condition corresponds one-to-one to the control instructions. Therefore, based on the training set corresponding to each driving condition, the initial neural network model corresponding to each driving condition is trained to obtain the driving condition neural network corresponding to the control instruction. The initial neural network model corresponding to the driving condition can be a neural network structure with an input layer, a hidden layer and an output layer, for example. For example, two fully connected layers are set in the hidden layer, and the number of nodes in the hidden layer is set to 256.

[0084] In actual application, the driving condition represents the driving state of the unmanned truck. Different driving conditions correspond to different ranges of steering angle and acceleration. For example, the steering angle of straight driving is 0, and the range of steering angle of right turning is [0, 1.0].

[0085] In actual application, according to the image collected by the camera of the unmanned truck simulation model, the driving condition corresponding to the image can be determined, the image is classified based on the driving condition corresponding to the image, and the images corresponding to the same driving condition are grouped into an image set corresponding to the driving condition. Optionally, the significant features of the image are extracted by an image feature extraction method, and the significant features are input into a trained image classification neural network to realize the classification of the image.

[0086] The reference value of the steering angle and the acceleration corresponding to the image refers to the reference value of the steering angle and the acceleration obtained by inputting the feature vector corresponding to the image into the neural network model corresponding to the driving condition, and the accuracy of the steering angle and the acceleration output by the neural network model corresponding to the driving condition can be determined based on the reference value of the steering angle and the acceleration corresponding to the image.

[0087] In this embodiment, image processing refers to preprocessing of the image collected by the camera. For example, the size of the image is adjusted to ensure that the size of each image is consistent, for example, the image is down-sampled to reduce the size of the image; the part of the image irrelevant to the road condition is cropped, for example, the image part of the car engine hood and the sky is cropped; image enhancement is performed to avoid overfitting, for example, random flipping, random brightness adjustment, random shadow addition and random movement are used for image enhancement.

[0088] In actual application, the image processed by image processing is subjected to convolution operation to obtain the feature vector corresponding to the image. For example, the convolution layer is set to 4 layers, the first layer of convolution kernel is set to 8*8, the step is 4, the convolution operation is performed, and the result is processed by a linear rectification function (Linear rectification function, abbreviated as ReLU); the second layer of convolution kernel is set to 4*4, the step is 2, the convolution operation is performed, and the result is processed by an activation function; the third layer of convolution kernel is set to 3*3, the step is 1, the convolution operation is performed, and the result is processed by an activation function; the fourth layer converts the output of the third layer into a feature vector.

[0089] It should be noted that the image state feature vector corresponding to each image is a combination of the feature vector corresponding to the image and the reference value of the steering angle and the acceleration corresponding to the image. The image state feature vectors corresponding to all images in the image set corresponding to each driving condition are grouped into a training set corresponding to the image set.

[0090] In practical applications, the initial neural network model corresponding to each driving condition is trained based on the training set corresponding to each driving condition. It can be understood that the image state feature vector is input into the initial neural network model corresponding to the image state feature vector, and the real values of the steering angle and the acceleration are output.

[0091] Specifically, the mean square error loss function corresponding to the reference values of the steering angle and the acceleration in the image state feature vector and the real values of the steering angle and the acceleration output by the initial neural network model is calculated. When the neural network parameters in the error loss function meet the preset range or the training times reach the maximum training times, the training of the initial neural network model corresponding to the driving condition is ended, and the trained neural network model is used as the driving condition neural network corresponding to the driving condition. Since the control instruction is one-to-one corresponding to the driving condition, the trained neural network can be used as the driving condition neural network corresponding to the control instruction.

[0092] Optionally, the mean square error loss function corresponding to the reference values of the steering angle and the acceleration in the image state feature vector and the real values of the steering angle and the acceleration output by the initial neural network model can be expressed as follows:

[0093] L(θ)=‖s-s gt ‖ 2 +λa‖a-a gt ‖ 2

[0094] Wherein, s is the steering angle output by the initial neural network model, s gt is the reference value of the steering angle in the image state feature vector, a is the acceleration output by the initial neural network model, a gt is the reference value of the acceleration in the image state feature vector, and θ is the neural network parameter.

[0095] In each training of the initial neural network model, the neural network parameter is updated based on the gradient descent method, and the expression is as follows:

[0096]

[0097] Wherein, α is the learning step.

[0098] The unmanned truck driving method provided in the embodiment comprises the following steps: importing the high-precision point cloud map into a simulation environment constructed in advance; acquiring images collected by a camera, a steering angle and acceleration collected by a pose sensor, and point cloud data collected by a laser radar; establishing an initial neural network model corresponding to each driving condition; dividing the images collected by the camera of the unmanned truck simulation model into a continue driving image set, a left turn image set, a right turn image set, and a straight driving image set according to the driving conditions; taking the steering angle and acceleration collected by the pose sensor at the same collection time as reference values of the steering angle and acceleration corresponding to each image in the image set; performing image processing and convolution operation on each image in the image set corresponding to each driving condition to obtain a feature vector corresponding to each image, and combining the feature vector corresponding to each image with the reference values of the steering angle and acceleration corresponding to each image to obtain an image state feature vector; taking the image state feature vectors corresponding to all images in the image set corresponding to each driving condition as a training set corresponding to the driving condition; and training the initial neural network model corresponding to each driving condition based on the training set corresponding to each driving condition until a trained neural network corresponding to each driving condition is obtained as a driving condition neural network corresponding to a corresponding control instruction. In the embodiment, the initial neural network model corresponding to each driving condition is established, the initial neural network model corresponding to each driving condition is trained based on the training set corresponding to each driving condition, the driving condition neural network corresponding to the driving condition neural network corresponding to the corresponding control instruction is obtained, and the feature vector corresponding to the image is input into the driving condition neural network corresponding to the corresponding control instruction, so that the steering angle and acceleration output by the driving condition neural network can be obtained, thereby realizing automatic driving of the unmanned truck.

[0099] Embodiment three

[0100] Figure 3 The flowchart of the unmanned truck driving method provided in the third embodiment of the application is shown in Figure 3 The method further comprises the following steps:

[0101] S301, the high-precision point cloud map is imported into a simulation environment constructed in advance; the simulation environment comprises an unmanned truck simulation model loaded with a camera, a pose sensor, and a laser radar;

[0102] S302, the images collected by the camera, the steering angle and acceleration collected by the pose sensor, and the point cloud data collected by the laser radar are acquired;

[0103] S303, an initial control instruction neural network is established;

[0104] S304, training data is acquired;

[0105] S305, training the initial control instruction neural network based on the training data until the control instruction neural network is obtained.

[0106] S301 and S302 of the embodiment are the same as S201 and S202 of the above-mentioned embodiment, please refer to the above-mentioned embodiment for details, which will not be repeated here.

[0107] For example, the initial control neural network includes a neural network structure of an input layer, a hidden layer and an output layer, wherein the hidden layer includes four fully connected layers, the number of hidden layer nodes is set to 128, and the activation function adopts a linear rectifier function.

[0108] In the embodiment, the training data includes the path point sequence of the unmanned truck simulation model, the positioning data of the current unmanned truck simulation model in the high-precision point cloud map, and the control instruction reference value corresponding to the path point sequence of the unmanned truck simulation model and the positioning data of the unmanned truck simulation model in the high-precision point cloud map.

[0109] Specifically, the point cloud data collected by the laser radar of the current unmanned truck simulation model is matched with the high-precision point cloud map in the simulation model to determine the positioning data of the current unmanned truck simulation model in the high-precision point cloud map. The starting point and the ending point of the unmanned truck simulation model are determined, and the driving path of the unmanned truck simulation model is obtained based on the path planning algorithm; the positioning data of each point on the driving path on the high-precision point cloud map is determined to obtain the path point sequence. The corresponding control instruction reference value is set based on the path point sequence of the unmanned truck simulation model and the positioning data of the unmanned truck simulation model in the high-precision point cloud map.

[0110] In actual application, the path point sequence of the unmanned truck simulation model and the positioning data of the current unmanned truck simulation model in the high-precision point cloud map in the training data are input into the initial control instruction neural network to obtain the true value of the control instruction output by the control instruction neural network. The mean square loss function of the true value and the reference value of the control instruction is calculated, the neural network parameters are updated based on the policy gradient descent method, and the training is ended when the neural network parameters reach the preset range or the training times reach the set maximum training times, and the control instruction neural network is obtained.

[0111] Optionally, the mean square loss function of the true value and the reference value of the control instruction can be represented by the following formula:

[0112]

[0113] wherein y t is the neural network output, c t is the supervised learning data, t is the data dimension, and θ is the neural network parameter. The neural network parameter is updated based on the gradient descent method for each training of the initial neural network model, and the expression is as follows:

[0114]

[0115] wherein, a is a learning step size.

[0116] The unmanned truck driving method provided in the embodiment imports the high-precision point cloud map into a simulation environment constructed in advance; acquires images collected by a camera, a steering angle and acceleration collected by a pose sensor, and point cloud data collected by a laser radar; establishes an initial control instruction neural network; acquires training data; and trains the initial control instruction neural network based on the training data until a control instruction neural network is obtained. In the embodiment, the initial control instruction neural network is trained based on the training data to obtain the control instruction neural network. The path point sequence and the positioning data of the current unmanned truck on the high-precision point cloud map are input into the control instruction neural network to obtain the control instruction output by the control instruction neural network. The feature vector corresponding to the image is input into the driving condition neural network corresponding to the control instruction, thereby improving the accuracy of unmanned truck driving.

[0117] Embodiment Four

[0118] Figure 4 The structure schematic diagram of the unmanned truck driving device provided in Embodiment Four of the present application is shown in FIG. 4, which comprises: Figure 4

[0119] The first acquisition module 41 is configured to acquire a high-precision point cloud map.

[0120] The navigation module 42 is configured to determine a path point sequence of the unmanned truck and positioning data of the unmanned truck on the high-precision point cloud map.

[0121] The first processing module 43 is configured to input the path point sequence and the positioning data of the current unmanned truck on the high-precision point cloud map into a control instruction neural network to obtain a control instruction output by the control instruction neural network.

[0122] The second processing module 44 is configured to perform image processing and convolution operation on images collected by a camera on the unmanned truck to obtain a feature vector corresponding to the images.

[0123] The autonomous driving module 45 is configured to input the feature vector corresponding to the images into a driving condition neural network corresponding to the control instruction to obtain a steering angle and acceleration output by the driving condition neural network.

[0124] ​In this embodiment, the path point sequence of the unmanned vehicle is the positioning data of all points on the driving path of the unmanned vehicle on the high-precision point cloud map. It should be noted that the path point sequence of the unmanned vehicle is the positioning data of all points on the driving path of the unmanned vehicle on the high-precision point cloud map, which is planned in advance before the unmanned vehicle drives, and is used as a reference route for the unmanned vehicle to drive.

[0125] In this embodiment, the path point sequence of the unmanned vehicle is the positioning data of all points on the driving path of the unmanned vehicle on the high-precision point cloud map. It should be noted that the path point sequence of the unmanned vehicle is the positioning data of all points on the driving path of the unmanned vehicle on the high-precision point cloud map, which is planned in advance before the unmanned vehicle drives, and is used as a reference route for the unmanned vehicle to drive.

[0126] In practical applications, the execution subject of the unmanned vehicle driving method can be an unmanned vehicle driving device. The unmanned vehicle driving device can be implemented in various ways, such as through a computer program, for example, application software, or a medium storing a related computer program, for example, a U disk, a cloud disk, or an entity device integrated or installed with a related computer program, for example, a chip.

[0127] In this embodiment, the path point sequence of the unmanned vehicle is the positioning data of all points on the driving path of the unmanned vehicle on the high-precision point cloud map. It should be noted that the path point sequence of the unmanned vehicle is the positioning data of all points on the driving path of the unmanned vehicle on the high-precision point cloud map, which is planned in advance before the unmanned vehicle drives, and is used as a reference route for the unmanned vehicle to drive.

[0128] In this embodiment, the path point sequence of the unmanned vehicle is the positioning data of all points on the driving path of the unmanned vehicle on the high-precision point cloud map. It should be noted that the path point sequence of the unmanned vehicle is the positioning data of all points on the driving path of the unmanned vehicle on the high-precision point cloud map, which is planned in advance before the unmanned vehicle drives, and is used as a reference route for the unmanned vehicle to drive.

[0129] In practical applications, the driving condition neural network corresponds to the control instruction one by one, and is a trained neural network. In this embodiment, the driving condition neural network includes a continue driving condition neural network, a left turn driving condition neural network, a right turn driving condition neural network, and a straight driving condition neural network. Therefore, according to the control instruction, the driving condition neural network to which the feature vector corresponding to the image is to be input can be determined.

[0130] Optionally, a branch judgment switch is arranged, the branch judgment switch receives a control instruction and a feature vector corresponding to an image, if a continue driving control instruction is received, the feature vector corresponding to the image is input into the continue driving working condition neural network; if a left turn control instruction is received, the feature vector corresponding to the image is input into the left turn driving working condition neural network; if a right turn control instruction is received, the feature vector corresponding to the image is input into the right turn driving working condition neural network; if an execution control instruction is received, the feature vector corresponding to the image is input into the execution driving working condition neural network.

[0131] In actual application, after the steering angle and the acceleration output by the driving working condition neural network, the values of the steering wheel, the throttle and the brake of the unmanned truck are determined in cooperation with a control unit of the unmanned truck, and then the automatic driving of the unmanned truck is realized. Optionally, the steering angle and the acceleration can be input into a vehicle control module, and a control instruction output by the vehicle control module is obtained, the control instruction including control instructions for the steering wheel, the throttle and the brake.

[0132] In the embodiment, the control instruction network is combined with the driving working condition neural network, which can adapt to different driving environments. Based on the control instruction output by the control instruction network, an image collected by a camera of the current unmanned truck is input into the driving working condition network corresponding to the control command, and the steering angle and the acceleration of the unmanned truck output by the driving working condition neural network are obtained; the control instruction is output based on a path point sequence of the unmanned truck and positioning data of the current unmanned truck on the high-precision point cloud map input into the control instruction neural network; wherein the path point sequence of the unmanned truck and the positioning data of the current unmanned truck on the high-precision point cloud map are determined based on the high-precision point cloud map and are independent of global positioning system signals; therefore, when the global positioning system signal coverage is poor and the driving environment changes, the automatic driving of the unmanned truck can still be realized, and the reliability of the unmanned truck driving is improved.

[0133] Optionally, in a possible implementation, the first acquisition module 41 includes:

[0134] The acquisition unit is configured to acquire point cloud data and positioning data collected by a vehicle;

[0135] The association unit is configured to associate the point cloud data and the positioning data based on a collection time, and determine position information corresponding to the point cloud data;

[0136] The splicing unit is configured to apply normal distribution transformation to splice the position information of the point cloud data at all collection times to form a point cloud map;

[0137] The labeling unit is configured to label road condition information on each position on the point cloud map, and obtain a high-precision point cloud map.

[0138] The point cloud data is a data set of space points collected by a laser radar, and each point cloud data contains three-dimensional coordinate information. The collection vehicle is provided with a laser radar and a global positioning navigation system; the point cloud data is collected by the laser radar, and the positioning data is collected by the global positioning navigation system. In practical application, the driving of the collection vehicle provided with the laser radar and the global positioning navigation system collects the point cloud data of each point of the collection vehicle by the laser radar, and collects the positioning data of each point of the collection vehicle by the global positioning navigation system.

[0139] Specifically, the point cloud data and the positioning data collected at the same time are associated to determine the positioning data corresponding to the point cloud data, i.e. the position information corresponding to the point cloud data. In practical application, the collected point cloud data is discrete data, for the point cloud data corresponding to each collection time, the positioning data corresponding to the time closest to the collection time is determined, and the position information corresponding to the points near the point cloud data corresponding to the collection time is determined by difference calculation on the positioning data corresponding to the time closest to the collection time. For example, the difference calculation refers to coordinate and rotation interpolation transformation on each point cloud data, and the extrinsic parameters of the laser radar are supplemented to obtain the basic affine transformation matrix of each point cloud data at the collection time, and the point cloud coordinate system is converted to the global coordinate system of the collection vehicle through the pose information and the affine transformation matrix of each point cloud data.

[0140] According to the above description, the position information corresponding to the point cloud data at the collection time is determined, and the position information of the point cloud data at all collection times is spliced into a point cloud map based on normal distribution transformation.

[0141] In practical application, a map annotation software can be applied to label the road condition information of each position on the point cloud map to obtain a high-precision point cloud map. The road condition information includes lane and intersection information.

[0142] In this embodiment, the point cloud data and the positioning data are collected by the collection vehicle, the point cloud data and the positioning data are associated based on the collection time to determine the position information corresponding to the point cloud data, the position information of the point cloud data at all collection times is spliced into a point cloud map based on normal distribution transformation, and the road condition information of each position on the point cloud map is labeled to obtain a high-precision point cloud map. The high-precision point cloud map is applied to automatic driving of the unmanned truck to improve the reliability of driving of the unmanned truck.

[0143] Optionally, in a possible implementation, the navigation module 42 is specifically configured to: determine the starting point and the ending point of the unmanned truck, obtain the driving path of the unmanned truck based on a path planning algorithm; determine the positioning data on the high-precision point cloud map corresponding to each point on the driving path to obtain the sequence of path points.

[0144] The point cloud data collected by the laser radar on the current unmanned truck is matched with the high-precision point cloud map to determine the positioning data of the current unmanned truck on the high-precision point cloud map.

[0145] It can be understood that there are multiple paths between the start point and the end point, and the optimal driving path can be obtained through a path planning algorithm. For example, the optimal driving path refers to the driving path with the shortest driving time. Alternatively, the high-precision point cloud map is converted into a topological map composed of a series of nodes and edges, and the driving path of the unmanned truck is obtained through a search algorithm.

[0146] In actual application, after the driving path is determined, the positioning data of each point on the driving path on the high-precision point cloud map is determined, and the point cloud data corresponding to all points on the driving path is composed into a path point sequence.

[0147] The point cloud data collected by the laser radar on the current unmanned truck is matched with the high-precision point cloud map to determine the positioning data of the current unmanned truck on the high-precision point cloud map. Alternatively, the closest point iteration algorithm is applied to realize the matching of the point cloud data collected by the laser radar on the unmanned truck with the high-precision point cloud map, and the positioning data of the current unmanned truck on the high-precision point cloud map is determined.

[0148] In this embodiment, based on the high-precision point cloud map, the positioning data of the current unmanned truck on the high-precision point cloud map is obtained.

[0149] The unmanned truck driving device provided in the embodiment comprises a first acquisition module, a navigation module, a first processing module, a second processing module and an automatic driving module.

[0150] Embodiment five

[0151] Figure 5 The structure schematic diagram of the unmanned truck driving device provided in the embodiment five of the present application is shown in the figure, and the device further comprises: Figure 5

[0152] The import module 51 is used for importing the high-precision point cloud map into the simulation environment constructed in advance;

[0153] The acquisition module 52 is used for acquiring the image collected by the camera, the steering angle and acceleration collected by the pose sensor and the point cloud data collected by the laser radar;

[0154] The first establishment module 53 is used for establishing the initial neural network model corresponding to each driving condition respectively;

[0155] The classification module 54 is used for classifying the image collected by the camera of the unmanned truck simulation model into the continue driving image set, the left turn image set, the right turn image set and the straight driving image set according to the driving condition; for each image in the image set, the steering angle and acceleration collected by the pose sensor at the same collection time are taken as the reference value of the steering angle and acceleration corresponding to the image;

[0156] ​The third processing module 55 is configured to, for each image set corresponding to each driving condition, perform image processing and convolution operation on each image in the image set to obtain a feature vector corresponding to each image, combine the feature vector corresponding to each image with the reference values of the steering angle and the acceleration corresponding to each image to obtain an image state feature vector, and take the image state feature vectors corresponding to all images in the image set corresponding to each driving condition as a training set corresponding to the driving condition.

[0157] The first training module 56 is configured to train the initial neural network model corresponding to each driving condition based on the training set corresponding to each driving condition until a trained neural network corresponding to each driving condition is obtained as a driving condition neural network corresponding to the corresponding control instruction.

[0158] In this embodiment, the simulation environment includes an unmanned truck simulation model equipped with a camera, a pose sensor, and a laser radar. The camera is used to collect the road condition images of the current unmanned truck, the pose sensor is used to collect the steering angle and the acceleration of the current unmanned truck, and the laser radar is used to collect the point cloud data. For example, the simulation environment can be established based on an automatic driving platform, and the image collected by the camera, the steering angle and the acceleration collected by the pose sensor, and the point cloud data collected by the laser radar are obtained by driving the unmanned truck simulation model to travel in the simulation environment.

[0159] In this embodiment, the driving conditions include continuing driving, left turning, right turning, and straight driving. The driving conditions correspond one-to-one to the control instructions, and the initial neural network model established based on each driving condition corresponds one-to-one to the control instructions. Therefore, training the initial neural network model corresponding to each driving condition based on the training set corresponding to each driving condition can obtain the driving condition neural network corresponding to the control instruction. For example, the initial neural network model corresponding to the driving condition can be a neural network structure with an input layer, a hidden layer, and an output layer.

[0160] In actual applications, the driving conditions represent the driving state of the unmanned truck. The ranges of the steering angle and the acceleration corresponding to different driving conditions are different.

[0161] In actual applications, according to the image collected by the camera of the unmanned truck simulation model, the driving condition corresponding to the image can be determined, the image is classified based on the driving condition corresponding to the image, and the images corresponding to the same driving condition are grouped to form an image set corresponding to the driving condition. Optionally, the significant features of the image are extracted by an image feature extraction method, the significant features are input into a trained image classification neural network, and the classification of the image is realized.

[0162] The reference value of the steering angle and the acceleration corresponding to the image is a reference value of the steering angle and the acceleration obtained by inputting the feature vector corresponding to the image into the neural network model corresponding to the driving condition.

[0163] In this embodiment, image processing refers to pre-processing of images collected by the camera. For example, the size of the image is adjusted to ensure that the size of each image is consistent. For example, the image is down-sampled to reduce the size of the image; the part of the image irrelevant to the road condition is cropped; and the overfitting is avoided by image enhancement.

[0164] It should be noted that the image state feature vector corresponding to each image is a combination of the feature vector corresponding to the image and the reference value of the steering angle and the acceleration corresponding to the image. The image state feature vectors corresponding to all images in the image set corresponding to each driving condition are combined to form a training set corresponding to the image set.

[0165] In practical applications, the initial neural network model corresponding to each driving condition is trained based on the training set corresponding to each driving condition. It can be understood that the image state feature vector is input into the corresponding initial neural network model to output the true value of the steering angle and the acceleration.

[0166] Specifically, the mean square error loss function corresponding to the reference value of the steering angle and the acceleration in the image state feature vector and the true value of the steering angle and the acceleration output by the initial neural network model is calculated. When the neural network parameters in the error loss function meet the preset range or the training times reach the maximum training times, the training of the initial neural network model corresponding to the driving condition is ended, and the trained neural network model is used as the driving condition neural network corresponding to the driving condition. Since the control instruction is one-to-one corresponding to the driving condition, the trained neural network can be used as the driving condition neural network corresponding to the control instruction.

[0167] The unmanned truck driving device provided by the embodiment imports the high-precision point cloud map into a simulation environment constructed in advance through an importing module; an image collected by a camera, a steering angle and acceleration collected by a pose sensor, and point cloud data collected by a laser radar are acquired through a collecting module; an initial neural network model corresponding to each driving condition is established through a first establishing module; the image collected by the camera of the unmanned truck simulation model is classified into a continue driving image set, a left turn image set, a right turn image set, and a straight driving image set according to the driving conditions through a classifying module; for each image in the image set, the steering angle and the acceleration collected by the pose sensor at the same collection time are taken as reference values of the steering angle and the acceleration corresponding to the image through a third processing module; for the image set corresponding to each driving condition, each image in the image set is subjected to image processing and convolution operation to obtain a feature vector corresponding to each image, and the feature vector corresponding to each image is combined with the reference values of the steering angle and the acceleration corresponding to each image to obtain an image state feature vector; the image state feature vectors corresponding to all images in the image set corresponding to each driving condition are taken as a training set corresponding to the driving condition; the initial neural network model corresponding to each driving condition is trained based on the training set corresponding to each driving condition through a first training module until a trained neural network corresponding to each driving condition is obtained, which is taken as a driving condition neural network corresponding to a corresponding control instruction. In the embodiment, the initial neural network model corresponding to each driving condition is established, the initial neural network model corresponding to each driving condition is trained based on the training set corresponding to each driving condition, the driving condition neural network corresponding to the driving condition neural network corresponding to the corresponding control instruction is obtained, and the feature vector corresponding to the image is input into the driving condition neural network corresponding to the corresponding control instruction, so that the steering angle and the acceleration output by the driving condition neural network can be obtained, thereby realizing the automatic driving of the unmanned truck.

[0168] Embodiment six

[0169] Figure 6 The structure schematic diagram of the unmanned truck driving device provided by the embodiment six is shown in the figure, and the device further includes: Figure 6

[0170] The importing module 61 is used for importing the high-precision point cloud map into a simulation environment constructed in advance;

[0171] The collecting module 62 is used for acquiring an image collected by a camera, a steering angle and acceleration collected by a pose sensor, and point cloud data collected by a laser radar;

[0172] The second establishing module 63 is used for establishing an initial control instruction neural network;

[0173] The second acquiring module 64 is used for acquiring training data;​

[0174] The second training module 65 is configured to train the initial control instruction neural network based on the training data until the control instruction neural network is obtained.

[0175] In this embodiment, the training data includes the path point sequence of the unmanned truck simulation model, the positioning data of the current unmanned truck simulation model in the high-precision point cloud map, and the control instruction reference value corresponding to the path point sequence of the unmanned truck simulation model and the positioning data of the unmanned truck simulation model in the high-precision point cloud map.

[0176] Specifically, the positioning data of the current unmanned truck simulation model in the high-precision point cloud map is determined by matching the point cloud data collected by the laser radar of the current unmanned truck simulation model with the high-precision point cloud map in the simulation model. The starting point and the ending point of the unmanned truck simulation model are determined, and the driving path of the unmanned truck simulation model is obtained based on the path planning algorithm. The positioning data of each point on the driving path on the high-precision point cloud map is determined to obtain the path point sequence. The corresponding control instruction reference value is set based on the path point sequence of the unmanned truck simulation model and the positioning data of the unmanned truck simulation model in the high-precision point cloud map.

[0177] In actual application, the path point sequence of the unmanned truck simulation model and the positioning data of the current unmanned truck simulation model in the high-precision point cloud map in the training data are input into the initial control instruction neural network to obtain the true value of the control instruction output by the control instruction neural network. The mean square loss function of the true value and the reference value of the control instruction is calculated, and the neural network parameters are updated based on the policy gradient descent method. When the neural network parameters reach the preset range or the training times reach the set maximum training times, the training is ended, and the control instruction neural network is obtained.

[0178] In the unmanned truck driving device provided in this embodiment, the import module imports the high-precision point cloud map into the simulation environment constructed in advance. The acquisition module acquires the image collected by the camera, the steering angle and acceleration collected by the pose sensor, and the point cloud data collected by the laser radar. The second establishment module establishes the initial control instruction neural network. The second acquisition module acquires the training data. The second training module trains the initial control instruction neural network based on the training data until the control instruction neural network is obtained. In this embodiment, the initial control instruction neural network is trained based on the training data to obtain the control instruction neural network. The path point sequence and the positioning data of the current unmanned truck in the high-precision point cloud map are input into the control instruction neural network to obtain the control instruction output by the control instruction neural network. The feature vector corresponding to the image is input into the driving condition neural network corresponding to the control instruction to improve the accuracy of the unmanned truck driving.

[0179] Embodiment Seven

[0180] Figure 7 An architecture schematic diagram of the unmanned truck driving system provided in Embodiment Seven is shown in FIG. 7. As shown in FIG. 7, the device further includes: Figure 7

[0181] The first acquisition module 71 is configured to acquire a high-precision point cloud map.

[0182] The navigation module 72 is configured to determine a path point sequence of the unmanned truck and positioning data of the unmanned truck on the high-precision point cloud map.

[0183] The first processing module 73 is configured to input the path point sequence and the positioning data of the current unmanned truck on the high-precision point cloud map into a control command neural network to obtain a control command output by the control command neural network.

[0184] The second processing module 74 is configured to perform image processing and convolution operation on an image collected by a camera on the unmanned truck to obtain a feature vector corresponding to the image.

[0185] The classification switch module 75 is configured to determine a driving condition neural network to which the feature vector corresponding to the image is to be input according to the control command.

[0186] The automatic driving module 75 is configured to input the feature vector corresponding to the image into the driving condition neural network corresponding to the control instruction to obtain a steering angle and an acceleration output by the driving condition neural network.

[0187] The vehicle control module 76 is configured to control the unmanned truck to drive based on the steering angle and the acceleration output by the driving condition neural network.

[0188] In the unmanned truck driving system provided in the embodiment, the first acquisition module acquires a high-precision point cloud map, the navigation module determines a path point sequence of the unmanned truck and positioning data of the unmanned truck on the high-precision point cloud map, the first processing module inputs the path point sequence and the positioning data of the current unmanned truck on the high-precision point cloud map into a control command neural network to obtain a control command output by the control command neural network, the second processing module performs image processing and convolution operation on an image collected by a camera on the unmanned truck to obtain a feature vector corresponding to the image, the classification switch module determines a driving condition neural network to which the feature vector corresponding to the image is to be input according to the control command, the automatic driving module inputs the feature vector corresponding to the image into the driving condition neural network corresponding to the control instruction to obtain a steering angle and an acceleration output by the driving condition neural network, and the vehicle control module controls the unmanned truck to drive based on the steering angle and the acceleration output by the driving condition neural network. Therefore, the automatic driving of the unmanned truck can be realized when the global positioning system signal coverage is poor and the driving environment changes, and the reliability of the unmanned truck driving is improved.

[0189] Embodiment Eight ​

[0190] Figure 8 Figure 8 shows a schematic diagram of an electronic device according to an embodiment of the present application. Figure 8

[0191] The electronic device further comprises a processor 81 and a memory 82. The electronic device can further comprise a communication interface 83 and a bus 84. The processor 81, the memory 82 and the communication interface 83 can communicate with each other through the bus 84. The communication interface 83 can be used for information transmission. The processor 81 can invoke the logic instructions in the memory 82 to execute the method of the above embodiments.

[0192] In addition, the logic instructions in the memory 82 can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0193] The memory 82 can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the method of the embodiments of the present application. The processor 71 executes the functions and data processing by running the software programs, instructions and modules stored in the memory 82, that is, implements the method in the above method embodiments.

[0194] The memory 82 can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required by a function. The data storage area can store data created during use of the terminal device, etc. In addition, the memory 82 can include a high-speed random access memory, and can further include a non-volatile memory.

[0195] The embodiments of the present application provide a non-transitory computer readable storage medium, and the computer readable storage medium stores computer execution instructions. When the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the method of the above embodiments.

[0196] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The specification and examples given are considered exemplary only, and the true scope and spirit of the application is indicated by the following claims. The true scope and spirit of the application are indicated by the following claims.

[0197] ​It is to be understood that the application is not limited to the precise construction already described above and shown in the drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application should only be limited by the claims appended hereto.

Claims

1. A method for driving an unmanned truck, characterized by, The method comprises: acquiring a high-precision point cloud map, determining a path point sequence of the unmanned vehicle and positioning data of the unmanned vehicle on the high-precision point cloud map; wherein the path point sequence of the unmanned vehicle is the positioning data of all points on the driving path of the unmanned vehicle on the high-precision point cloud map; inputting the path point sequence and the current positioning data of the unmanned vehicle on the high-precision point cloud map into a control instruction neural network to obtain a control instruction output by the control instruction neural network; wherein the control instruction comprises: continue driving, left turn, right turn and straight driving; the control instruction neural network is a trained neural network; performing image processing and convolution operation on the image collected by the camera on the unmanned vehicle to obtain a feature vector corresponding to the image; inputting the feature vector corresponding to the image into a driving condition neural network corresponding to the control instruction to obtain a steering angle and an acceleration output by the driving condition neural network; wherein the driving condition neural network corresponds to the control instruction one by one and is a trained neural network; The method comprises: acquiring point cloud data and positioning data collected by a collection vehicle; wherein the collection vehicle is installed with a laser radar and a global positioning navigation system; the point cloud data is collected by the laser radar, and the positioning data is collected by the global positioning navigation system; associating the point cloud data and the positioning data based on the collection time to determine the position information corresponding to the point cloud data; applying normal distribution transformation to splice the position information of the point cloud data at all collection times to form a point cloud map; labeling road condition information on each position of the point cloud map to obtain a high-precision point cloud map; the road condition information comprises lane and intersection information; The method comprises: determining the starting point and the ending point of the unmanned vehicle, obtaining the driving path of the unmanned vehicle based on a path planning algorithm; determining the positioning data of each point on the driving path on the high-precision point cloud map to obtain the path point sequence; matching the point cloud data collected by the laser radar on the current unmanned vehicle with the high-precision point cloud map to determine the positioning data of the current unmanned vehicle on the high-precision point cloud map; the positioning data comprises position information and road condition information.

2. The method of claim 1, wherein, The method further comprises: importing the high-precision point cloud map into a pre-constructed simulation environment; the simulation environment comprises an unmanned vehicle simulation model loaded with a camera, a pose sensor and a laser radar; acquiring the image collected by the camera, the steering angle and the acceleration collected by the pose sensor, and the point cloud data collected by the laser radar.

3. The method of claim 2, wherein, The method further comprises: establishing a corresponding initial neural network model for each driving condition respectively; the driving conditions comprise continue driving, left turn, right turn and straight driving; The images collected by the camera of the unmanned vehicle simulation model are divided into a continue driving image set, a left turn image set, a right turn image set and a straight driving image set according to driving conditions; for each image in the image set, the steering angle and acceleration collected by the pose sensor at the same collection time are taken as reference values of the steering angle and acceleration corresponding to the image; For each image set corresponding to each driving condition, each image in the image set is subjected to image processing and convolution operation to obtain a feature vector corresponding to each image, and the feature vector corresponding to each image is combined with the reference values of the steering angle and acceleration corresponding to each image to obtain an image state feature vector; the image state feature vectors corresponding to all images in the image set corresponding to each driving condition are taken as a training set corresponding to the driving condition; Based on the training set corresponding to each driving condition, an initial neural network model corresponding to each driving condition is trained until a trained neural network corresponding to each driving condition is obtained as a driving condition neural network corresponding to a corresponding control instruction.

4. The method of claim 2, wherein, The method further comprises: establishing an initial control instruction neural network; obtaining training data, the training data including a path point sequence of the unmanned vehicle simulation model and current positioning data of the unmanned vehicle simulation model in the high-precision point cloud map, and a control instruction reference value corresponding to the path point sequence of the unmanned vehicle simulation model and the positioning data of the unmanned vehicle simulation model in the high-precision point cloud map; training the initial control instruction neural network based on the training data until the control instruction neural network is obtained.

5. An unmanned truck driving device characterized by comprising: comprises: a first acquisition module for acquiring a high-precision point cloud map; a navigation module for determining a path point sequence of an unmanned vehicle and positioning data of the unmanned vehicle on the high-precision point cloud map; wherein the path point sequence of the unmanned vehicle is the positioning data of all points on the driving path of the unmanned vehicle on the high-precision point cloud map; a first processing module for inputting the path point sequence and the current positioning data of the unmanned vehicle on the high-precision point cloud map into a control instruction neural network to obtain a control instruction output by the control instruction neural network; wherein the control instruction includes continue driving, left turn, right turn and straight driving; the control instruction neural network is a trained neural network; a second processing module for performing image processing and convolution operation on an image collected by a camera on the unmanned vehicle to obtain a feature vector corresponding to the image; an automatic driving module for inputting the feature vector corresponding to the image into a driving condition neural network corresponding to the control instruction to obtain a steering angle and acceleration output by the driving condition neural network; wherein the driving condition neural network corresponds one-to-one to the control instruction and is a trained neural network; the first acquisition module comprises: The collection unit is configured to acquire point cloud data and positioning data collected by a collection vehicle; the collection vehicle is provided with a laser radar and a global positioning navigation system; the point cloud data is collected by the laser radar, and the positioning data is collected by the global positioning navigation system; The association unit is configured to associate the point cloud data and the positioning data based on a collection time, and determine position information corresponding to the point cloud data; The splicing unit is configured to apply normal distribution transformation to splice position information of the point cloud data at all collection times to form a point cloud map; The labeling unit is configured to label road condition information on each position on the point cloud map, and obtain a high-precision point cloud map; the road condition information includes lane and intersection information. The navigation module is specifically configured to: determine a starting point and an ending point of the unmanned delivery vehicle, and obtain a driving path of the unmanned delivery vehicle based on a path planning algorithm; determine positioning data corresponding to each point on the driving path on the high-precision point cloud map, and obtain the sequence of path points; match point cloud data collected by a laser radar on the unmanned delivery vehicle with the high-precision point cloud map, and determine positioning data of the unmanned delivery vehicle on the high-precision point cloud map; the positioning data includes position information and road condition information.

6. The apparatus of claim 5, wherein, The device further includes: The import module is configured to import the high-precision point cloud map into a pre-constructed simulation environment; the simulation environment includes an unmanned delivery vehicle simulation model provided with a camera, a pose sensor, and a laser radar; The collection module is configured to acquire images collected by the camera, a steering angle and acceleration collected by the pose sensor, and point cloud data collected by the laser radar.

7. The apparatus of claim 6, wherein, The device further includes: The first establishment module is configured to establish an initial neural network model corresponding to each driving condition; the driving conditions include continuous driving, left turn, right turn, and straight driving; The classification module is configured to divide images collected by the camera of the unmanned delivery vehicle simulation model into a continuous driving image set, a left turn image set, a right turn image set, and a straight driving image set according to driving conditions; for each image in the image set, a steering angle and acceleration collected by the pose sensor at the same collection time are taken as reference values of the steering angle and acceleration corresponding to the image; The third processing module is configured to, for each image set corresponding to a driving condition, perform image processing and convolution operation on each image in the image set to obtain a feature vector corresponding to each image, combine the feature vector corresponding to each image with the reference values of the steering angle and acceleration corresponding to each image to obtain an image state feature vector; and take the image state feature vectors corresponding to all images in each image set corresponding to a driving condition as a training set corresponding to the driving condition. The first training module is configured to train the initial neural network model corresponding to each driving condition based on the training set corresponding to each driving condition, until a trained neural network corresponding to each driving condition is obtained, as a driving condition neural network corresponding to a corresponding control instruction.

8. The apparatus of claim 6, wherein, The device further includes: The second establishment module is configured to establish an initial control instruction neural network. A second obtaining module is configured to obtain training data, wherein the training data comprises a path point sequence of the unmanned vehicle simulation model, current positioning data of the unmanned vehicle simulation model in the high-precision point cloud map, and a control instruction reference value corresponding to the path point sequence of the unmanned vehicle simulation model and the positioning data of the unmanned vehicle simulation model in the high-precision point cloud map. A second training module is configured to train the initial control instruction neural network based on the training data until the control instruction neural network is obtained.

9. An electronic device, comprising: Comprise: A processor, and a memory connected with the processor in communication; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the method in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method in any one of claims 1-4.

Images