Low-speed automatic driving vehicle control method, device, equipment and medium

Abstract

The present application relates to a control method, device, equipment and medium for a low-speed autonomous driving car, the method comprising: updating the occupation grid map data according to the dynamic target position information, the drivable road area and the vehicle position information, determining the path points for navigating the low-speed autonomous driving car according to the updated occupation grid map data based on the A* algorithm, determining the driving path of the low-speed autonomous driving car according to the path points of the low-speed autonomous driving car based on the TEB algorithm; responding to the control instruction of the low-speed autonomous driving car, driving the low-speed autonomous driving car to drive according to the driving path, so as to complete the control of the low-speed autonomous driving car. The present application can form an environmental map by acquiring road condition image frames and occupation grid map data, taking into full account various obstacles in the environment, avoiding obstacles on the driving path, and causing collisions and accidents of vehicles during transportation.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to a method for controlling a low-speed autonomous driving vehicle, a corresponding device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In existing technical solutions, the development and application of autonomous driving systems are usually carried out on some high-performance development boards. However, such development boards need to be embedded in vehicles, and different types of designs are required for different vehicles, which are not portable and universal. At the same time, in order to maintain high-performance computing capabilities, such development boards require high power consumption, which will also have a certain impact on the vehicle's endurance.

[0003] In existing sensor information fusion solutions, various sensors are usually installed at different locations on the vehicle to obtain environmental information for autonomous driving algorithm processing. This also requires different types of designs for different vehicles, which is not portable and universal. At the same time, due to the scattered arrangement of the sensors, the acquired information needs to be additionally time-aligned, which increases the time for information processing.

[0004] In summary, the development and application of autonomous driving systems in the prior art are usually carried on some high-performance development boards, which require high power consumption. In addition, since the sensors are arranged in a scattered manner, the acquired information needs to be additionally time-aligned, which increases the time for information processing. The applicant has made corresponding explorations in order to solve this problem. Summary of the invention

[0005] The purpose of this application is to solve the above-mentioned problems and to provide a low-speed automatic driving vehicle control method, corresponding device, electronic device and computer-readable storage medium.

[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0007] A low-speed automatic driving vehicle control method proposed to meet one of the purposes of this application includes:

[0008] Obtain road condition image frames and occupancy grid map data from various angles of the low-speed autonomous driving car, and use an intersection-over-union matching algorithm based on the position of moving objects and a re-identification matching algorithm based on appearance features to calculate and determine the confidence of each dynamic target detection frame between the current road condition image frame and the previous road condition image frame, so as to match and track the dynamic target position information of the current road condition image frame and the previous road condition image frame;

[0009] Inferring surface normal information from the road condition image frame, using an RGB encoder and a surface normal encoder to extract feature maps from the RGB image and the inferred surface normal information respectively, and using a pre-trained semantic road segmentation model to determine a drivable road surface area based on the fused RGB and surface normal information;

[0010] Determine wheel odometer data and GPS data, and fuse the wheel odometer data and GPS data based on a preset extended Kalman filter algorithm to determine vehicle position information of the low-speed autonomous driving vehicle;

[0011] The occupancy grid map data is updated according to the dynamic target position information, the drivable road area and the vehicle position information, and the path points for navigating the low-speed autonomous driving vehicle are determined according to the updated occupancy grid map data based on the A* algorithm, and the driving path of the low-speed autonomous driving vehicle is determined according to the path points of the low-speed autonomous driving vehicle based on the TEB algorithm;

[0012] Respond to the low-speed automatic driving vehicle control instruction, drive the low-speed automatic driving vehicle to travel along the driving path to complete the control of the low-speed automatic driving vehicle.

[0013] Optionally, the steps of using an intersection-over-union matching algorithm based on the position of a moving object and a re-identification matching algorithm based on appearance features to calculate and determine the confidence of each dynamic target detection frame between the current road condition image frame and the previous road condition image frame to match and track the dynamic target position information of the current road condition image frame and the previous road condition image frame include:

[0014] Constructing a three-dimensional model corresponding to each dynamic target to obtain the motion state information of each dynamic target relative to the low-speed automatic driving vehicle;

[0015] Determine the observation value corresponding to each dynamic target acquired by the monocular camera, use the Kalman filter algorithm to estimate the state information prediction value corresponding to each dynamic target in the current road condition image frame according to the motion state information of the low-speed automatic driving vehicle in a preset number of road condition image frames, and update the optimal estimation value of the current road condition image frame based on the observation value corresponding to each dynamic target and the state information prediction value corresponding to each dynamic target, and use it as the latest state information of each dynamic target;

[0016] Based on the latest state information of each dynamic target, an intersection-over-union matching algorithm based on the position of the moving object and a re-identification matching algorithm based on the appearance feature are used to match the weighted confidence to perform target matching and tracking between the current road condition image frame and the previous road condition image frame.

[0017] Optionally, the step of inferring surface normal information from the road condition image frame, using an RGB encoder and a surface normal encoder to extract feature maps from the RGB image and the inferred surface normal information respectively, and using a pre-trained semantic road segmentation model to determine the drivable road surface area according to the fused RGB and surface normal information includes:

[0018] Inferring surface normal information from the depth map corresponding to the road condition image frame, obtaining three-dimensional information of each pixel according to the CCS coordinates, calculating the horizontal gradient and the vertical gradient based on the Sobel operator to determine the x-direction component and the y-direction component of the surface normal information, and using the K-dimensional spatial data structure to select 8 neighboring points to estimate the normal vector component in the z direction;

[0019] The RGB encoder and the surface normal encoder are used to extract feature maps from the RGB image and the surface normal information respectively. The extracted feature maps are hierarchically fused by element-wise summation and fused again in the decoder through densely connected skip connections to determine the fused RGB and surface normal information.

[0020] A pre-trained semantic road segmentation model is used to determine the drivable road area based on the fused RGB and surface normal information.

[0021] Optionally, the step of fusing the wheel odometer data and GPS data based on a preset extended Kalman filter algorithm to determine the vehicle position information of the low-speed autonomous driving vehicle includes:

[0022] The steps of fusing the wheel odometer data and GPS data based on the preset extended Kalman filter algorithm are as follows:

[0023] According to the formula X k - =AX k-1 +Bu k-1 , through the state transfer matrix A and the prior estimate X in the previous cycle k-1 To predict the prior estimate X of the next cycle k - , the initial prior state value X k - The value provided for the sensor;

[0024] According to the formula P k - =AP k-1 A T +Q, update the prior covariance matrix, Q is the covariance matrix of the system process error, P k-1 is the posterior covariance matrix in the previous cycle;

[0025] According to the formula Get K for each cycle k Kalman gain value, where H is the observation matrix, which is responsible for converting the estimated value into data of the same type as the observed value, and R is the covariance of the measurement noise;

[0026] According to the formula X k =X k - +K k (Z k -HX k - ), the posterior estimate X k The prior estimate of this cycle X k - , Kalman gain K k , observation value Z k It is calculated by the observation matrix H, where the observation value is the sensor information obtained in each cycle;

[0027] According to the formula P k =(IK k H)P k - , update the new round of posterior covariance matrix P k ;

[0028] Iterate steps (1) to (5) to obtain the fused wheel odometer data and GPS data.

[0029] Optionally, the step of determining a path point for navigating the low-speed autonomous driving vehicle according to the updated occupancy grid map data based on an A* algorithm includes:

[0030] The updated occupancy grid map data is used as the map data for global path planning of the A* algorithm, the current location of the low-speed autonomous driving car is set as the starting point of the path, and the location specified by the GPS coordinates is used as the end point, and the occupancy grid map data includes the current location of the low-speed autonomous driving car, the end point location, the road network, and obstacle information;

[0031] Manhattan distance is used to define the heuristic function, which measures the sum of the distances from the current node to the target node in the horizontal and vertical directions;

[0032] Initialize the data structure and add the path start point to an open list, create an open list to store nodes to be explored, create a closed list to store nodes that have been explored, add the path start point to the open list, and set its estimated cost value to 0;

[0033] In each iteration, a node with a minimum cost value is selected from the open list, the selected node is moved to the closed list to indicate that it has been explored, the cost values ​​of the neighboring nodes of the selected node are calculated, and they are added to the open list. If they are not in the closed list, the parent node information is updated to record the best path to reach these nodes;

[0034] Each time a node is added to the open list, it is checked whether the target node has been found. If the target node is found, it means that the path has been found and the search can be terminated;

[0035] After finding the target node, trace back the parent node information, trace back from the target node to the starting point, and obtain the complete path;

[0036] The searched path is optimized, Bezier curves are used to smooth the path to reduce sharp turns and acute angles in the path, redundant nodes are removed by merging nodes to reduce the path length, and a string of path points for navigating the low-speed autonomous driving car is output.

[0037] Optionally, the step of determining the driving path of the low-speed autonomous driving vehicle according to the path points of the low-speed autonomous driving vehicle based on the TEB algorithm includes:

[0038] Discretizing the path points used to navigate the low-speed autonomous driving vehicle into a series of points, and then connecting the points to form a preliminary elastic band;

[0039] Adjusting the preliminary elastic band according to a path optimization target based on a preset path optimization algorithm to determine an optimized elastic band, wherein the path optimization target includes one or more of minimizing the length of the elastic band, minimizing the steering angle of the low-speed autonomous driving vehicle, and maximizing the distance between the elastic band and the obstacle;

[0040] The optimized elastic band is used as a local path of the low-speed autonomous driving vehicle, and the low-speed autonomous driving vehicle is driven to travel along the local path.

[0041] Optionally, the low-speed autonomous driving vehicle includes one or any multiple of an unmanned express delivery vehicle and an intelligent transport vehicle.

[0042] A low-speed automatic driving vehicle control device provided for another purpose of the present application includes:

[0043] The dynamic target tracking module is configured to obtain the road condition image frames and the occupied grid map data of the low-speed autonomous driving car at various angles, and adopt the intersection-over-union matching algorithm based on the position of the moving object and the re-identification matching algorithm based on the appearance features to calculate and determine the confidence of each dynamic target detection frame between the current road condition image frame and the previous road condition image frame, so as to match and track the dynamic target position information of the current road condition image frame and the previous road condition image frame;

[0044] a drivable road surface determination module, configured to infer surface normal information from the road condition image frame, extract feature maps from the RGB image and the inferred surface normal information using an RGB encoder and a surface normal encoder, respectively, and determine a drivable road surface area based on the fused RGB and surface normal information using a pre-trained semantic road segmentation model;

[0045] A vehicle position determination module, configured to determine wheel odometer data and GPS data, and fuse the wheel odometer data and GPS data based on a preset extended Kalman filter algorithm to determine vehicle position information of the low-speed autonomous driving vehicle;

[0046] a driving path determination module, configured to update the occupancy grid map data according to the dynamic target position information, the drivable road area, and the vehicle position information, determine the path points for navigating the low-speed autonomous driving vehicle according to the updated occupancy grid map data based on the A* algorithm, and determine the driving path of the low-speed autonomous driving vehicle according to the path points of the low-speed autonomous driving vehicle based on the TEB algorithm;

[0047] The automatic driving control module is configured to respond to the control instructions of the low-speed automatic driving vehicle and drive the low-speed automatic driving vehicle to travel along the driving path to complete the control of the low-speed automatic driving vehicle.

[0048] An electronic device provided to meet another purpose of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the low-speed automatic driving vehicle control method described in the present application.

[0049] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the low-speed automatic driving vehicle control method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

[0050] Compared with the prior art, the present application aims at the problems that the development and application of the autonomous driving system in the prior art is usually carried on some high-performance development boards, which require high power consumption, and because the sensors are arranged in a dispersed manner, the acquired information needs to be additionally time-aligned, which increases the time for information processing. The present application includes but is not limited to the following beneficial effects:

[0051] First, this application greatly improves portability and versatility. It is developed and used based on the hardware architecture of mainstream smartphones. The vehicle body does not need additional modification. Users only need to connect the smartphone to the vehicle to enable the vehicle to obtain the function of autonomous driving.

[0052] Second, this application does not affect the vehicle's endurance experience, and there is no need to modify the vehicle or install additional sensors. Only some of the sensor devices that come with the smartphone are used, and the collected information is processed by the smartphone's low-power processor. The system can maintain normal operation for a long time by relying on the phone's own battery, and will not cause additional impact on the vehicle's power supply.

[0053] Third, the present application greatly reduces the user's usage cost. The user does not need to bear the cost of vehicle modification, such as the current mainstream lidar, millimeter wave radar, GPS and other sensors. Only daily-used smartphones and ordinary vehicles are needed to experience the full functions of the present application.

[0054] Furthermore, the present application can form an environmental map by acquiring road condition image frames and occupied grid map data, taking into full account various obstacles in the environment, avoiding obstacles on the driving path, causing collisions and accidents during transportation, and thus affecting transportation efficiency. It ensures that there are no obstacles on the driving path, avoids accidental damage caused by collisions with people and objects, and avoids time waste caused by accidents. Thus, the transportation efficiency of vehicles is improved while ensuring safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0056] Figure 1 This is an exemplary network architecture used in the low-speed autonomous driving vehicle control method of this application;

[0057] Figure 2 This is a flow chart of a method for controlling a low-speed automatic driving vehicle in an embodiment of the present application;

[0058] Figure 3 This is a general structural diagram of the Bluetooth pairing of a smart phone to a designated vehicle in an embodiment of the present application;

[0059] Figure 4This is a principle block diagram of a low-speed automatic driving vehicle control device in an embodiment of the present application;

[0060] Figure 5 It is a schematic diagram of the structure of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0061] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0062] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0063] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0064] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0065] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0066] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0067] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0068] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0069] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0070] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality with each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0071] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0072] The computing resources of smartphones have been significantly improved in the past few years, allowing them to host more and more complex applications, including deep learning applications. Deep learning is a machine learning technology that requires a lot of computing resources for model training and inference. As the computing resources of smartphones continue to improve and optimize, they have the ability to host deep learning applications.

[0073] With the continuous development of science and technology, the functions and performance of smartphones have been greatly improved. Among them, the application of multiple sensors is one of the keys for smartphones to achieve more functions and provide a better user experience. Smartphones' multi-sensors include many different types of sensors, such as visual sensors, lidar, GPS, etc. These sensors can sense the environment and status around the phone, collect various data, and analyze and process them through algorithms and data processing technologies to provide more accurate and reliable information for the phone.

[0074] Visual perception plays a key role in autonomous driving. By acquiring rich color and texture information, autonomous vehicles can quickly and accurately identify and locate targets on the road, such as vehicles, pedestrians, road markings, etc. At the same time, visual sensors can build an environmental model around the vehicle to help the vehicle plan its path and avoid obstacles. In addition, visual perception allows vehicles to understand dynamic changes in traffic scenes and perform tasks such as obstacle avoidance and road recognition. In order to improve the accuracy and reliability of visual perception, new technologies such as deep learning algorithms are used to process complex image information and extract useful features.

[0075] High-precision fusion positioning is a solution that improves positioning accuracy by combining different sensors and positioning technologies. It can fuse data from different sources, including GPS, inertial measurement unit (IMU), wheel speed sensor, laser radar (LiDAR), image sensor, etc., to provide more accurate, reliable and comprehensive location information, helping vehicles achieve safe and efficient autonomous driving in various scenarios.

[0076] The core systems of low-speed unmanned vehicles include the autonomous driving system, sensor system, navigation system, etc. The autonomous driving system can control the vehicle's speed, direction, and braking, etc., based on the vehicle's location and surrounding environment information. The sensor system includes a variety of sensors, such as lidar, cameras, and ultrasound, which are used to sense the surrounding environment and provide vehicle positioning, navigation, and obstacle avoidance functions. The navigation system can provide route planning and navigation for the vehicle to ensure that the vehicle can drive autonomously along the preset route.

[0077] See also Figure 1The low-speed autonomous driving vehicle control method of the present application can be implemented based on the low-speed autonomous driving vehicle control system in the mobile terminal. The front camera, rear camera, GPS and wheel odometer of the smartphone can collect data at the same time, and the obtained data can be sent to the central processor of the smartphone for separate processing.

[0078] The road condition image frames from various angles, such as the front image and the rear image, are put into two neural networks to obtain: three-dimensional tracking information of obstacles around the vehicle (such as people, cars, etc.) and road information on which the vehicle can safely pass (normal lanes, park roads, indoor floors, etc.).

[0079] The positioning information of GPS and the vehicle's own motion information provided by the wheel odometer are integrated through Kalman filtering to obtain the positioning information of the vehicle's real-time motion. The various information obtained above are synchronously used for trajectory planning to provide route planning and navigation for the vehicle, ensuring that the vehicle can drive autonomously along the preset route and safely avoid various obstacles encountered.

[0080] The entire low-speed unmanned vehicle system is well packaged and can be well adapted and applied on mainstream smartphones. Ultimately, users only need to connect their mobile phones to the unmanned vehicle to enable the vehicle to have the function of autonomous driving.

[0081] Based on the above exemplary scenarios, please refer to Figure 2 In one embodiment, the low-speed automatic driving vehicle control method of the present application includes:

[0082] Step S10, obtaining road condition image frames and occupancy grid map data of the low-speed autonomous driving car at various angles, using an intersection-over-union matching algorithm based on the position of a moving object and a re-identification matching algorithm based on appearance features, calculating and determining the confidence of each dynamic target detection frame between the current road condition image frame and the previous road condition image frame, so as to match and track the dynamic target position information of the current road condition image frame and the previous road condition image frame;

[0083] The mobile terminal can obtain road condition image frames and occupancy grid map data of the low-speed autonomous driving vehicle at various angles. The mobile terminal can be a smart phone, a computer, etc. The road condition image frame is an image frame containing static obstacles, dynamic targets, and road surfaces, etc., and adopts an intersection-over-union matching algorithm based on the position of moving objects and a re-identification matching algorithm based on appearance features to calculate and determine the confidence of each dynamic target detection frame between the current road condition image frame and the previous road condition image frame, so as to match and track the dynamic target position information of the current road condition image frame and the previous road condition image frame; the low-speed autonomous driving vehicle includes one or any multiple of an unmanned express delivery vehicle and an intelligent transport vehicle.

[0084] Specifically, the steps of calculating and determining the confidence of each dynamic target detection frame between the current road condition image frame and the previous road condition image frame by using an intersection-over-union matching algorithm based on the position of the moving object and a re-identification matching algorithm based on the appearance feature, so as to match and track the dynamic target position information of the current road condition image frame and the previous road condition image frame, include:

[0085] Step S101: construct a three-dimensional model corresponding to each dynamic target to obtain the motion state information of each dynamic target relative to the low-speed automatic driving vehicle;

[0086] Step S103, determining the observation values ​​corresponding to each dynamic target acquired by the monocular camera, using the Kalman filter algorithm to estimate the state information prediction value corresponding to each dynamic target in the current road condition image frame according to the motion state information of the low-speed automatic driving car in a preset number of road condition image frames, and updating the optimal estimation value of the current road condition image frame based on the observation values ​​corresponding to each dynamic target and the state information prediction value corresponding to each dynamic target, and using it as the latest state information of each dynamic target;

[0087] Step S105: Based on the latest state information of each dynamic target, an intersection-over-union matching algorithm based on the position of the moving object and a re-identification matching algorithm based on the appearance feature are used to match the weighted confidence to perform target matching and tracking between the current road condition image frame and the previous road condition image frame.

[0088] More specifically, based on the pre-trained target detection model, target detection is performed on common dynamic targets on real roads, such as pedestrians and cars. The robustness of trajectory detection is enhanced by using Kalman filtering, and the moving targets in the two-dimensional image are projected into the three-dimensional space. The confidence obtained by weighting the intersection-over-union (IOU) matching algorithm based on the position of the moving object and the re-identification matching algorithm based on the appearance features is used to match the target between the current road condition image frame and the previous road condition image frame. This method can reduce the frequency of false detection and missed detection, and effectively realize the tracking of objects with nonlinear motion in three dimensions. The specific steps are as follows:

[0089] (1) Through the use of five conventional three-dimensional frame detection heads and five auxiliary training heads, the monocular camera is also capable of restoring the three-dimensional model by using auxiliary learning. The three-dimensional frame detection head is mainly used to predict the size of the three-dimensional frame, the rough depth value of the center point of the two-dimensional frame, and the coordinate offset of the center. The auxiliary head is mainly used to quantify the errors of the center point of the two-dimensional frame and the eight vertices.

[0090] (2) Establishing a three-dimensional model of each dynamic target, using the return information of the above step (1), reconstructing the dynamic target in three dimensions, and obtaining the motion state of each dynamic target relative to the vehicle. Establishing the three-dimensional model includes the following steps:

[0091] The camera is mounted on the vehicle and moves with the vehicle. To obtain the motion trajectory of the moving target, we must first establish a camera coordinate system with the optical center of the camera as the origin. According to the right-hand rule, the X-axis of the camera coordinate system is defined to be parallel to the horizontal plane, perpendicular to the forward direction of the car and positive to the right, the Y-axis is parallel to the horizontal plane, positive to the forward direction of the car, and the Z-axis is perpendicular to the horizontal direction. At this time, the three-dimensional coordinates of each target relative to the camera can be obtained. The motion trajectory of the target is also the relative trajectory of the target relative to the camera. Secondly, define the upper left corner of the image acquired by the camera as the origin of the pixel uv coordinate system, the horizontal right is the u axis, and the vertical downward is the v axis; and the principal point of the image (u0, v0), is also the x-axis and y-axis in the same direction as the u-axis and v-axis, respectively, and the camera internal parameters are known. The above three are used together to convert the pixel coordinates of the target to the camera coordinates. Our conversion of the uv coordinate system and the xy coordinate system satisfies:

[0092]

[0093] According to the central projection feature, the coordinate relationship of the pixel points satisfies the proportional relationship, so the camera coordinate system and the xy coordinate system satisfy:

[0094]

[0095] Combining the above formulas (1) and (2), we can get the conversion relationship between pixel coordinates and camera coordinates:

[0096]

[0097] (3) Kalman filter prediction and update: Based on the state information of the target in the previous 1 to t-1 frames, the Kalman filter is used to estimate the state information prediction value of the target in the current road condition image frame, and the observation value of the target obtained by the monocular camera is added. Based on the observation value corresponding to each dynamic target and the state information prediction value corresponding to each dynamic target, the optimal estimation value of the current road condition image frame is updated and used as the latest state information of each dynamic target;

[0098] (4) Multi-target matching: The latest status information of multiple targets in the current road condition image frame is obtained by the above step (3). Through the joint data association of IOU similarity and Re-ID re-identification, the targets of the current road condition image frame and the previous road condition image frame are matched to achieve the purpose of matching and tracking.

[0099] The IOU similarity is calculated by projecting the cuboid of the previous road condition image frame and the current road condition image frame target onto the xy plane and calculating the overlapping area of ​​the two detection frames in the plane:

[0100]

[0101] At the same time, the overlapping height is multiplied by the overlapping area to get the overlapping volume. Similarly, the three-dimensional IOU similarity can be calculated, denoted as d (1) (i,j).

[0102] A deep appearance feature descriptor is designed for each target, which is a 128-dimensional unit feature vector of the Re-ID network trained offline on the pedestrian re-identification dataset. For each tracker, retain its last 100 appearance feature descriptors R that are successfully associated with the detection box and calculate their minimum cosine distance with the detection box:

[0103]

[0104] To establish a matching association, we use a weighted sum to combine the two indicators. We can control the impact of each indicator on the matching through the hyperparameter λ:

[0105] c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j), (6)

[0106] Based on the latest state information of each dynamic target, an intersection-over-union matching algorithm based on the position of the moving object and a re-identification matching algorithm based on the appearance feature are used to match the weighted confidence to perform target matching and tracking between the current road condition image frame and the previous road condition image frame.

[0107] Step S20, inferring surface normal information from the road condition image frame, using an RGB encoder and a surface normal encoder to extract feature maps from the RGB image and the inferred surface normal information respectively, and using a pre-trained semantic road segmentation model to determine a drivable road area according to the fused RGB and surface normal information;

[0108] After matching and tracking the dynamic target position information of the current road condition image frame and the previous road condition image frame, inferring the surface normal information from the road condition image frame, using an RGB encoder and a surface normal encoder to extract feature maps from the RGB image and the inferred surface normal information respectively, and using a pre-trained semantic road segmentation model to determine the drivable road area according to the fused RGB and surface normal information;

[0109] Specifically, the steps of inferring surface normal information from the road condition image frame, extracting feature maps from the RGB image and the inferred surface normal information using an RGB encoder and a surface normal encoder respectively, and determining a drivable road surface area according to the fused RGB and surface normal information using a pre-trained semantic road segmentation model include:

[0110] Step S201, inferring surface normal information from the depth map corresponding to the road condition image frame, obtaining three-dimensional information of each pixel according to the CCS coordinates, calculating the horizontal gradient and the vertical gradient based on the Sobel operator to determine the x-direction component and the y-direction component of the surface normal information, and using the K-dimensional spatial data structure to select 8 neighboring points to estimate the normal vector component in the z direction;

[0111] Step S203, using an RGB encoder and a surface normal encoder to extract feature maps from the RGB image and the surface normal information respectively, the extracted feature maps are hierarchically fused by element summation, and are fused again in the decoder through densely connected jump connections to determine the fused RGB and surface normal information;

[0112] Step S205: using a pre-trained semantic road segmentation model to determine the drivable road area according to the fused RGB and surface normal information.

[0113] The detection of traversable areas of roads by fusing surface normal information in CNN neural network, inferring surface normal information from dense disparity / depth images, and merging RGB and surface normal information into semantic road segmentation using deep learning models are as follows:

[0114] The surface normal information is inferred from the depth / disparity image. For the depth map corresponding to the road image frame, the three-dimensional information of each element (pixel) is obtained by linking the CCS coordinates, and then the Sobel operator is used to calculate the horizontal and vertical gradients to obtain the x-direction component and y-direction component of the surface normal information. At the same time, the K-dimensional spatial data structure (KD tree) is used to select 8 neighboring points to estimate the normal vector component in the z direction, and n is obtained. x 、n y and n zi The expression is:

[0115] n x =-df x g x , n y =-df y g y , (7)

[0116]

[0117] The RGB and surface normal information are fused to realize the drivable area detection. The RGB encoder and the surface normal encoder are used to extract feature maps from the RGB image and the surface normal information respectively. The extracted feature maps are hierarchically fused by element-wise summation and fused again in the decoder through densely connected jump connections to determine the fused RGB and surface normal information. The pre-trained semantic road segmentation model is used to determine the drivable road area according to the fused RGB and surface normal information.

[0118] Step S30, determining the wheel odometer data and the GPS data, and fusing the wheel odometer data and the GPS data based on a preset extended Kalman filter algorithm to determine the vehicle position information of the low-speed autonomous driving vehicle;

[0119] After determining the drivable road area based on the fused RGB and surface normal information using a pre-trained semantic road segmentation model, wheel odometer data and GPS data are determined, and the wheel odometer data and GPS data are fused based on a preset extended Kalman filter algorithm to determine the vehicle position information of the low-speed autonomous driving vehicle;

[0120] Specifically, the step of fusing the wheel odometer data and the GPS data based on a preset extended Kalman filter algorithm to determine the vehicle position information of the low-speed autonomous driving vehicle includes:

[0121] GPS signals consist of carrier waves, ranging codes, and navigation data codes. The carrier waves are two carrier frequencies in the L band. The ranging codes include C / A codes and P codes, which are used to determine the pseudo-range error from the satellite to the observation point. The navigation data code contains information such as satellite ephemeris, satellite clock correction parameters, and ionosphere correction parameters, which are used to calculate the position of the satellite in space and correct the pseudo-range error from the satellite to the observation point. These signals work together to enable GPS to accurately locate the position of the observation point.

[0122] According to the front wheel steering angle θ and the linear velocity v of the rear wheel fed back by the Ackerman chassis, the two-dimensional coordinate point (x, y) and heading angle of the wheel odometer are obtained through the simplified model "single vehicle model" of the Ackerman kinematic model. Calculate the turning radius R:

[0123]

[0124] Where R is the turning radius and L is the wheelbase between the front and rear wheels.

[0125] Calculate the angular velocity ω:

[0126]

[0127] where ω is the angular velocity.

[0128] Calculate heading angle

[0129]

[0130] in, is the heading angle obtained in the last heading angle calculation process, and Δt is the interval between two data samplings.

[0131] Calculate the two-dimensional component of velocity v x 、v y :

[0132]

[0133]

[0134] Among them, v x is the velocity component of the linear velocity in the x-axis direction, v y is the velocity component of the linear velocity in the y-direction.

[0135] Calculate the two-dimensional coordinates (x, y):

[0136]

[0137]

[0138] Where x is the coordinate value in the x-axis direction, x′ is the coordinate value in the x-axis direction obtained in the previous two-dimensional coordinate calculation, y is the coordinate value in the y-axis direction, and y′ is the coordinate value in the y-axis direction obtained in the previous two-dimensional coordinate calculation.

[0139] Kalman filter is a highly efficient recursive filter (autoregressive filter) that can estimate the state of a dynamic system from a series of incomplete and noisy measurements. Kalman filter will produce an estimate of the unknown variable based on the values ​​of each measurement at different times, taking into account the joint distribution at each time, so it will be more accurate than the estimation method based on a single measurement. The algorithm of Kalman filter has two steps, namely prediction estimation and update correction:

[0140] In the estimation step, the Kalman filter produces an estimate of the current state, which also includes uncertainty;

[0141] In the update step, once the next measurement is observed (which will certainly contain some degree of error, including random noise), the estimate is corrected by weighted averaging, with the more certain the measurement, the higher the weight.

[0142] The sensors currently used are mainly wheel odometers and GPS positioning modules. Since the frequency of wheel odometers is higher, the attitude solution of wheel odometers is often used as the prediction of trajectory increments. The steps of fusing the wheel odometer data and GPS data based on the preset extended Kalman filter algorithm are as follows:

[0143] According to the formula X k - =AX k-1 +Bu k-1 , through the state transfer matrix A and the prior estimate X in the previous cycle k-1 To predict the prior estimate X of the next cycle k - , the initial prior state value X k - The value provided for the sensor;

[0144] According to the formula P k - =AP k-1 A T +Q, update the prior covariance matrix, Q is the covariance matrix of the system process error, P k-1 is the posterior covariance matrix in the previous cycle;

[0145] According to the formula Get K for each cycle k Kalman gain value, where H is the observation matrix, which is responsible for converting the estimated value into data of the same type as the observed value, and R is the covariance of the measurement noise;

[0146] According to the formula X k =X k - +K k (Z k -HX k - ), the posterior estimate X k The prior estimate of this cycle X k - , Kalman gain K k , observation value Z k And the observation matrix H is calculated. The observation value here is the sensor information obtained in each cycle.

[0147] (5) According to the formula P k =(IK k H)P k - , update the new round of posterior covariance matrix P k ;

[0148] Steps (1) and (2) belong to a priori prediction, and steps (3) to (5) belong to a posteriori correction. Steps (1) to (5) are iterated cyclically to obtain the data information of the fused wheel odometer data and GPS data.

[0149] Step S40, updating the occupancy grid map data according to the dynamic target position information, the drivable road area and the vehicle position information, determining the path points for navigating the low-speed autonomous driving vehicle according to the updated occupancy grid map data based on the A* algorithm, and determining the driving path of the low-speed autonomous driving vehicle according to the path points of the low-speed autonomous driving vehicle based on the TEB algorithm;

[0150] After determining the vehicle position information of the low-speed autonomous driving car, the occupancy grid map data is updated according to the dynamic target position information, the drivable road area and the vehicle position information, and the path points for navigating the low-speed autonomous driving car are determined according to the updated occupancy grid map data based on the A* algorithm, and the driving path of the low-speed autonomous driving car is determined according to the path points of the low-speed autonomous driving car based on the TEB algorithm;

[0151] In some embodiments, a custom gridOS class is used to uniformly manage map-related variables and functions in the system. Its member objects mainly include occupancy grid map variables of the nav_msgs::OccupancyGridPtr type, quaternions and translation vectors representing the camera (body) posture transformation, the current position of the camera (body), and various message variables for storing ROS topic information; member functions mainly include constructors, destructors, map update functions, camera (body) posture update functions, and functions for processing subscribed ROS topic information. After starting to run the main function of the map system, a map system object of the gridOS class is first constructed to handle various tasks of the map system. In the constructor, each pointer variable in the member variable is initialized, and various default values ​​of the map variables are set, such as resolution, width, height, and initial offset rotation. Then, in the main function, start the ZED2 camera, set the camera's hardware parameters, obtain the camera's left and right RGB images and depth map, and publish the image information through the ROS topic for 3D target detection and tracking, trajectory prediction, detection of passable areas and lane lines, and detection of road edges; at the same time, obtain the camera's visual odometer posture transformation information, publish this information through the ROS topic, and use it to fuse the camera (car body) positioning information; in addition, obtain the obstacle point cloud generated by the camera, perform voxel filtering on the point cloud and only retain the point cloud within a given range to update the obstacle information on the map. Set up the ROS subscriber and callback function, and correspondingly obtain the 3D target tracking and prediction information, passable area and lane line information, road edge information, and camera (car body) posture transformation information obtained after information fusion from each topic, and store them in each member variable of the map system object accordingly.

[0152] After receiving the message of each topic, it enters each member function of the map system object and processes it accordingly. The coordinates of the 3D target and its predicted trajectory information, lane line information, road edge information, and obstacle point cloud are uniformly converted to the vehicle coordinate system. With the camera position as the origin, a small window is constructed on the map. The range of all the above points in the vehicle coordinate system is divided according to the map resolution, and projected onto the window area accordingly. The local state of the map is updated in real time according to the window information. Among them, the cost value of the grid corresponding to the 3D target, lane line information, road edge information, and obstacle point cloud on the map is set to 100, indicating that there is an obstacle (occupied) at this location on the map; the cost value of the grid corresponding to the predicted trajectory of the 3D target is set to a number between 0 and 100, indicating that there may be or will be an obstacle at this location on the map; the cost value of the grid at the remaining locations is set to 0, indicating that there is no obstacle (not occupied) at this location on the map. After the projection process is completed, the grid with obstacles is "expanded", that is, the grids near this grid are also set to be occupied, to prevent the vehicle from getting too close to obstacles when driving along the path planned, causing safety accidents. Finally, the map variables after the above processing are published through the ROS topic, and then subscribed to this topic in other nodes to obtain map information and perform path planning.

[0153] Further, the step of determining the path points for navigating the low-speed autonomous driving vehicle according to the updated occupancy grid map data based on the A* algorithm includes:

[0154] Step S401, using the updated occupancy grid map data as map data for global path planning of the A* algorithm, setting the current location of the low-speed autonomous driving vehicle as the starting point of the path, and using the location specified by the GPS coordinates as the end point, the occupancy grid map data includes the current location of the low-speed autonomous driving vehicle, the end point location, the road network, and obstacle information;

[0155] Step S402: define a heuristic function using Manhattan distance to measure the sum of the distances from the current node to the target node in the horizontal and vertical directions;

[0156] Step S403, initialize the data structure and add the path starting point to an open list, create an open list to store the nodes to be explored, create a closed list to store the nodes that have been explored, add the path starting point to the open list, and set its estimated cost value to 0;

[0157] Step S404, in each iteration, select a node with a minimum cost value from the open list, move the selected node to the closed list to indicate that it has been explored, calculate the cost values ​​of the neighboring nodes of the selected node, and add them to the open list. If they are not in the closed list, update the parent node information to record the best path to reach these nodes;

[0158] Step S405: Each time a node is added to the open list, it is checked whether the target node has been found. If the target node is found, it means that the path has been found and the search can be terminated.

[0159] Step S406: After finding the target node, trace back the parent node information, trace back from the target node to the starting point, and obtain the complete path;

[0160] Step S407, optimize the searched path, use Bezier curves to smooth the path to reduce sharp turns and acute angles in the path, merge nodes to remove redundant nodes to reduce the path length, and output a string of path points for navigating the low-speed autonomous driving car.

[0161] Furthermore, the step of determining the driving path of the low-speed autonomous driving vehicle according to the path points of the low-speed autonomous driving vehicle based on the TEB algorithm includes:

[0162] Step S4001: discretize the path points used to navigate the low-speed automatic driving car into a series of points, and then connect these points to form a preliminary elastic band;

[0163] Step S4003: adjusting the preliminary elastic band according to a path optimization target based on a preset path optimization algorithm to determine an optimized elastic band, wherein the path optimization target includes one or more of minimizing the length of the elastic band, minimizing the steering angle of the low-speed autonomous driving vehicle, and maximizing the distance between the elastic band and the obstacle;

[0164] Step S4003: Use the optimized elastic band as a local path for the low-speed autonomous driving vehicle, and drive the low-speed autonomous driving vehicle to travel along the local path.

[0165] Global path planning for unmanned vehicles is a complex process that requires comprehensive consideration of map data, path planning algorithms, and vehicle dynamics. The following factors need to be considered when selecting a path planning algorithm: ensuring that the path does not intersect with obstacles to avoid collisions; optimizing the path length as much as possible to reduce the driving distance; and optimizing the path to minimize the vehicle's energy consumption.

[0166] Taking all the above factors into consideration, the A* algorithm is selected for path search. The following are the steps for using the A* algorithm for global path planning:

[0167] a. Prepare maps and environment data:

[0168] The obtained occupancy grid map is used as the map data for path planning, which contains the current location of the car, the destination location, the road network, obstacle information, etc. The current location of the car is set as the starting point of the path, and the location specified by the GPS coordinates is used as the destination. The user clicks on the map or enters the longitude and latitude coordinates of the target location. Search for the path in the obtained occupancy grid map.

[0169] b. Define the heuristic function:

[0170] Define a heuristic function (valuation function) h(n), which is used to estimate the cost from node n to the target node. Define the heuristic function using Manhattan distance, which measures the sum of the distances from the current node to the target node in the horizontal and vertical directions.

[0171] c. Initialize the data structure and add the starting point to the open list:

[0172] Create an Open List to store the nodes to be explored, and a Closed List to store the nodes that have been explored. Add the starting point to the Open List and set its estimated cost (f value) to 0.

[0173] d. Search in a loop:

[0174] In each iteration, select the node with the minimum f-value from the open list. Move the selected node to the closed list to indicate that it has been explored. Calculate the f-values ​​of the neighboring nodes of the selected node and add them to the open list if they are not in the closed list. Update the parent node information to record the best path to reach these nodes.

[0175] e. Check the target node:

[0176] Each time you add a node to the open list, check whether the target node has been found. If the target node is found, it means the path has been found and the search can be ended.

[0177] f. Path backtracking:

[0178] After finding the target node, trace back the parent node information, trace back from the target node all the way to the starting point to obtain the complete path.

[0179] g. Path optimization:

[0180] Optimize the searched path, use Bezier curves to smooth the path, reduce sharp turns and sharp angles in the path, improve the quality of the path, reduce energy consumption, etc. Remove redundant nodes by merging nodes and reduce the path length.

[0181] h. Output path:

[0182] Ultimately, the output is a string of waypoints that can be used to navigate a self-driving car.

[0183] i. Real-time updates:

[0184] The path is searched cyclically in the real-time updated occupancy grid map. During the movement of the car, it is determined whether the 20 nodes before the current position of the car are passable. If they are passable, there is no need to search the path again and the previous path can be used, thus reducing the path planning time.

[0185] Local path planning is the connection part from the global path to the actual chassis control. It obtains the path point information released by the global path planning, and outputs the speed and angular velocity information to the chassis after processing. That is, it is responsible for following the global path points to generate a specific, feasible, obstacle-avoiding path, and real-time obstacle avoidance when the global path cannot keep up. The local path planning algorithm adopted in this application is the TEB (Timed Elastic Band) algorithm. TEB (Timed Elastic Band) is a method for local path planning of low-speed autonomous driving vehicles. Its main idea is to regard the path of the low-speed autonomous driving vehicle as an elastic band (Elastic Band), and adjust this elastic band through an optimization algorithm so that it can avoid obstacles and be as close to the global path as possible.

[0186] The algorithm steps are as follows:

[0187] a. Initialization: First, the global path is discretized into a series of points, and then these points are connected to form a preliminary elastic band.

[0188] b. Optimization: Then, the elastic band is adjusted through an optimization algorithm. The optimization objectives include minimizing the length of the elastic band, minimizing the steering angle of the low-speed autonomous driving car, and maximizing the distance between the elastic band and the obstacle.

[0189] c. Update: Finally, the optimized elastic band is used as the local path of the low-speed autonomous driving car, and the movement of the low-speed autonomous driving car is controlled according to this path.

[0190] In some embodiments, some important functions of the TEB algorithm are:

[0191] 1. Time elastic band:

[0192]

[0193] Where t represents time, x(t) represents the state of the low-speed autonomous driving car at time t, represents n-dimensional Euclidean space, l(t) and u(t) respectively represent the lower and upper bounds of the time elastic band at time t. The time elastic band at time t is used to describe the dynamic constraints of the low-speed autonomous driving car during its motion, ensuring that the motion state of the low-speed autonomous driving car is within a certain range.

[0194] 2. Obstacle cost function:

[0195]

[0196] Among them, x represents the state of the low-speed autonomous driving car, n obs represents the number of obstacles, d i (x) represents the distance between the low-speed autonomous driving car and the $i$th obstacle, and ρ(·) represents the obstacle cost function. The obstacle cost function is used to describe the obstacle avoidance constraint between the low-speed autonomous driving car and the obstacle, ensuring that the motion state of the low-speed autonomous driving car will not collide with the obstacle.

[0197] 3. Path cost function:

[0198]

[0199] Where x represents the state of the low-speed autonomous driving car, α and β represent the weights of the obstacle cost function and the control cost function, respectively, u(t) represents the control input of the low-speed autonomous driving car at time t, and t0 and t f They represent the start time and end time of the path respectively.

[0200] 4. Steering angle:

[0201] The steering of the low-speed autonomous driving car is as smooth as possible, which can be achieved by minimizing the steering angle between adjacent path segments, that is: Where angle(a, b) is the angle between vectors a and b. Where angle(a, b) is the angle between vectors a and b.

[0202] Step S50, responding to the low-speed automatic driving vehicle control instruction, driving the low-speed automatic driving vehicle to travel along the driving path to complete the control of the low-speed automatic driving vehicle.

[0203] After determining the driving path of the low-speed autonomous driving vehicle according to the path points of the low-speed autonomous driving vehicle based on the TEB algorithm, the mobile terminal can respond to the control instruction of the low-speed autonomous driving vehicle to drive the low-speed autonomous driving vehicle along the driving path to complete the control of the low-speed autonomous driving vehicle.

[0204] In some embodiments, see Figure 3, you can use a smartphone to download custom software, control the phone to pair with a designated vehicle via Bluetooth, and at the same time, Bluetooth transmits a 2.4G frequency band radio wave signal. The communication protocol, modulation method, transmission rate and security mechanism of the signal are all compatible with the car hardware; when the vehicle radio wave receiver receives the signal, the receiver demodulates the special radio wave signal. After demodulation, the original digital signal is obtained; then verification and digital signature operations are performed; after verification, the digital data will be formatted as a CAN message. Finally, the CAN message, i.e., the message format, is transmitted through the vehicle's CAN bus, and various electronic control units (ECUs) in the car can read the message and perform the requested operation. Among them, the vehicle system has a radio wave transmission module, so this process is reversible, and the vehicle status information is passed to the visualization window of the mobile phone custom software.

[0205] (1) Mobile phone Bluetooth transmitter

[0206] Select the Bluetooth wireless communication protocol. Bluetooth is a wireless technology standard for exchanging data over short distances. It works by establishing a secure, low-power, short-range connection in the 2.4GHz ISM band. Select the Gaussian Frequency Shift Keying (GFSK) modulation method. Bluetooth's Gaussian Frequency Shift Keying (GFSK) modulation is a frequency modulation (FM) scheme in which binary data is represented by small frequency deviations.

[0207] The basic formula of GFSK modulation can be expressed as:

[0208] f(t)=f_c+Δf·g(t),

[0209] Where f(t) is the instantaneous frequency, f_c is the carrier frequency, and Δf is the frequency deviation, which determines the difference between the frequencies of logic 1 and 0. g(t) is the baseband data signal processed by the Gaussian filter. The baseband signal refers to the unmodulated original data signal.

[0210] (2) Vehicle system receiving end

[0211] When the vehicle's radio wave signal receiver receives a GFSK modulated signal from a specific mobile phone, the receiver will perform the following steps to demodulate it:

[0212] a. Received signal: The received RF signal is first amplified.

[0213] b. Frequency down conversion: The frequency of the signal is reduced to a lower intermediate frequency (IF) through a mixer.

[0214] c. Filtering: Use a bandpass filter to remove noise and unwanted signals.

[0215] d. Frequency Demodulation: The signal passes through a frequency demodulator, usually a frequency discriminator, which is able to convert the changes in frequency back into voltage changes, corresponding to the original digital data.

[0216] e.Digitalization: The analog signal is converted into a digital signal through an analog-to-digital converter (ADC).

[0217] The resulting digital signal often contains some authentication data, such as a digital signature, which the receiver checks to verify that the signal has not been tampered with and comes from the legitimate source.

[0218] The process of converting the demodulated digital signal into CAN information usually involves the following steps:

[0219] a. Data stream analysis: The original data stream obtained after demodulation is first parsed into a message frame structure.

[0220] b. Message frame identification: According to the CAN protocol specification, identify the different parts of the message frame, such as frame ID, control field, data field, CRC, etc.

[0221] c. Data extraction: Extract the data field from the message frame, which is the part containing the actual information.

[0222] d. Error checking and handling: Use CRC or other error detection codes in the frame to verify data defects.

[0223] e. Signal mapping: Map the extracted data into the corresponding CAN message format so that it can be transmitted to the CAN bus. The message format corresponds to a series of control commands of the vehicle, etc.

[0224] It can be seen from the above embodiments that, compared with the prior art, the present application aims at the problems that the development and application of the autonomous driving system in the prior art is usually carried on some high-performance development boards, which require high power consumption, and because the sensors are arranged in a dispersed manner, the acquired information needs to be additionally time-aligned, which increases the time for information processing. The present application includes but is not limited to the following beneficial effects:

[0225] First, this application greatly improves portability and versatility. It is developed and used based on the hardware architecture of mainstream smartphones. The vehicle body does not need additional modification. Users only need to connect the smartphone to the vehicle to enable the vehicle to obtain the function of autonomous driving.

[0226] Second, this application does not affect the vehicle's endurance experience, and there is no need to modify the vehicle or install additional sensors. Only some of the sensor devices that come with the smartphone are used, and the collected information is processed by the smartphone's low-power processor. The system can maintain normal operation for a long time by relying on the phone's own battery, and will not cause additional impact on the vehicle's power supply.

[0227] Third, the present application greatly reduces the user's usage cost. The user does not need to bear the cost of vehicle modification, such as the current mainstream lidar, millimeter wave radar, GPS and other sensors. Only daily-used smartphones and ordinary vehicles are needed to experience the full functions of the present application.

[0228] Furthermore, the present application can form an environmental map by acquiring road condition image frames and occupied grid map data, taking into full account various obstacles in the environment, avoiding obstacles on the driving path, causing collisions and accidents during transportation, and thus affecting transportation efficiency. It ensures that there are no obstacles on the driving path, avoids accidental damage caused by collisions with people and objects, and avoids time waste caused by accidents. Thus, the transportation efficiency of vehicles is improved while ensuring safety.

[0229] See also Figure 4 , a low-speed automatic driving car control device provided to meet one of the purposes of the present application, includes a dynamic target tracking module 1100, a drivable road surface determination module 1200, a vehicle position determination module 1300, a driving path determination module 1400 and an automatic driving control module 1500. Among them, the dynamic target tracking module 1100 is configured to obtain road condition image frames and occupied grid map data of the low-speed automatic driving car at various angles, and adopts an intersection-over-union matching algorithm based on the position of moving objects and a re-identification matching algorithm based on appearance features to calculate and determine the confidence of each dynamic target detection frame between the current road condition image frame and the previous road condition image frame, so as to match and track the dynamic target position information of the current road condition image frame and the previous road condition image frame; the drivable road surface determination module 1200 is configured to infer surface normal information from the road condition image frame, and adopts an RGB encoder and a surface normal encoder to extract feature maps from the RGB image and the inferred surface normal information respectively, and adopts a pre-trained semantic road segmentation model to determine the drivable road surface area according to the fused RGB and surface normal information; the vehicle position determination module 1300 , configured to determine wheel odometer data and GPS data, and fuse the wheel odometer data and GPS data based on a preset extended Kalman filter algorithm to determine the vehicle position information of the low-speed autonomous driving car; a driving path determination module 1400, configured to update the occupancy grid map data according to the dynamic target position information, the drivable road area and the vehicle position information, and determine the path points for navigating the low-speed autonomous driving car based on the updated occupancy grid map data based on the A* algorithm, and determine the driving path of the low-speed autonomous driving car based on the path points of the low-speed autonomous driving car based on the TEB algorithm; an autonomous driving control module 1500, configured to respond to a low-speed autonomous driving car control instruction, and drive the low-speed autonomous driving car to travel along the driving path to complete the control of the low-speed autonomous driving car.

[0230] Based on any embodiment of this application, please refer to Figure 5 Another embodiment of the present application further provides an electronic device, which can be implemented by a computer device, such as Figure 5 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a low-speed automatic driving vehicle control method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the low-speed automatic driving vehicle control method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0231] In this embodiment, the processor is used to execute Figure 4 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the low-speed automatic driving car control device of this application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0232] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the low-speed automatic driving vehicle control method described in any embodiment of the present application.

[0233] The present application also provides a computer program product, including a computer program / instruction, which, when executed by one or more processors, implements the steps of the low-speed automatic driving vehicle control method described in any embodiment of the present application.

[0234] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0235] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

[0236] In summary, the present application can form an environmental map by acquiring road condition image frames and occupying grid map data, which fully considers various obstacles in the environment and avoids obstacles on the driving path that may cause collisions and accidents during transportation.

Claims

1. A low-speed automatic driving vehicle control method, characterized in that: include: Obtain road condition image frames and occupancy grid map data of low-speed autonomous driving vehicles at various angles, use the intersection-over-union matching algorithm based on the position of moving objects and the re-identification matching algorithm based on appearance features, calculate and determine the confidence of each dynamic target detection frame between the current road condition image frame and the previous road condition image frame, so as to match and track the dynamic target position information of the current road condition image frame and the previous road condition image frame, which includes: Constructing a three-dimensional model corresponding to each dynamic target to obtain the motion state information of each dynamic target relative to the low-speed automatic driving vehicle; Determine the observation value corresponding to each dynamic target acquired by the monocular camera, use the Kalman filter algorithm to estimate the state information prediction value corresponding to each dynamic target in the current road condition image frame according to the motion state information of the low-speed automatic driving vehicle in a preset number of road condition image frames, and update the optimal estimation value of the current road condition image frame based on the observation value corresponding to each dynamic target and the state information prediction value corresponding to each dynamic target, and use it as the latest state information of each dynamic target; Based on the latest state information of each dynamic target, an intersection-over-union matching algorithm based on the position of the moving object and a re-identification matching algorithm based on the appearance feature are used to match the confidence obtained by weighting, so as to perform target matching tracking between the current road condition image frame and the previous road condition image frame; Inferring surface normal information from the road condition image frame, using an RGB encoder and a surface normal encoder to extract feature maps from the RGB image and the inferred surface normal information respectively, and using a pre-trained semantic road segmentation model to determine a drivable road surface area based on the fused RGB and surface normal information; Determine wheel odometer data and GPS data, and fuse the wheel odometer data and GPS data based on a preset extended Kalman filter algorithm to determine vehicle position information of the low-speed autonomous driving vehicle; The occupancy grid map data is updated according to the dynamic target position information, the drivable road area and the vehicle position information, and the path points for navigating the low-speed autonomous driving vehicle are determined according to the updated occupancy grid map data based on the A* algorithm, and the driving path of the low-speed autonomous driving vehicle is determined according to the path points of the low-speed autonomous driving vehicle based on the TEB algorithm; Respond to the low-speed automatic driving vehicle control instruction, drive the low-speed automatic driving vehicle to travel along the driving path to complete the control of the low-speed automatic driving vehicle.

2. The low-speed automatic driving vehicle control method according to claim 1 is characterized in that: The steps of inferring surface normal information from the road condition image frame, extracting feature maps from the RGB image and the inferred surface normal information using an RGB encoder and a surface normal encoder respectively, and determining a drivable road surface area according to the fused RGB and surface normal information using a pre-trained semantic road segmentation model include: Inferring surface normal information from the depth map corresponding to the road condition image frame, obtaining three-dimensional information of each pixel according to the CCS coordinates, calculating the horizontal gradient and the vertical gradient based on the Sobel operator to determine the x-direction component and the y-direction component of the surface normal information, and using the K-dimensional spatial data structure to select 8 neighboring points to estimate the normal vector component in the z direction; The RGB encoder and the surface normal encoder are used to extract feature maps from the RGB image and the surface normal information respectively. The extracted feature maps are hierarchically fused by element-wise summation and fused again in the decoder through densely connected skip connections to determine the fused RGB and surface normal information. A pre-trained semantic road segmentation model is used to determine the drivable road area based on the fused RGB and surface normal information.

3. The low-speed automatic driving vehicle control method according to claim 1 is characterized in that: The step of fusing the wheel odometer data and the GPS data based on a preset extended Kalman filter algorithm to determine the vehicle position information of the low-speed autonomous driving vehicle includes: The steps of fusing the wheel odometer data and GPS data based on the preset extended Kalman filter algorithm are as follows: According to the formula X k - =AX k-1 +Bu k-1 , through the state transfer matrix A and the prior estimate X in the previous cycle k-1 To predict the prior estimate of the next cycle X k - , the initial prior state value X k - The value provided for the sensor; According to the formula P k - =AP k-1 A T +Q, update the prior covariance matrix, Q is the covariance matrix of the system process error, P k-1 is the posterior covariance matrix in the previous cycle; According to the formula Get K for each cycle k Kalman gain value, where H is the observation matrix, which is responsible for converting the estimated value into data of the same type as the observed value, and R is the covariance of the measurement noise; According to the formula X k =X k - +K k (Z k -HX k - ), the posterior estimate X k The prior estimate of this cycle X k - , Kalman gain K k , observation value Z k It is calculated by the observation matrix H, where the observation value is the sensor information obtained in each cycle; According to the formula P k =(IK k H)P k - , update the new round of posterior covariance matrix P k ; The above steps are iterated cyclically to obtain the data information of the fused wheel odometer data and GPS data.

4. The low-speed automatic driving vehicle control method according to claim 1 is characterized in that: The step of determining a path point for navigating the low-speed autonomous driving vehicle according to the updated occupancy grid map data based on the A* algorithm comprises: The updated occupancy grid map data is used as the map data for global path planning of the A* algorithm, the current location of the low-speed autonomous driving car is set as the starting point of the path, and the location specified by the GPS coordinates is used as the end point, and the occupancy grid map data includes the current location of the low-speed autonomous driving car, the end point location, the road network, and obstacle information; Manhattan distance is used to define the heuristic function, which measures the sum of the distances from the current node to the target node in the horizontal and vertical directions; Initialize the data structure and add the path start point to an open list, create an open list to store nodes to be explored, create a closed list to store nodes that have been explored, add the path start point to the open list, and set its estimated cost value to 0; In each iteration, a node with a minimum cost value is selected from the open list, the selected node is moved to the closed list to indicate that it has been explored, the cost values ​​of the neighboring nodes of the selected node are calculated, and they are added to the open list. If they are not in the closed list, the parent node information is updated to record the best path to reach these nodes; Each time a node is added to the open list, it is checked whether the target node has been found. If the target node is found, it means that the path has been found and the search can be terminated; After finding the target node, trace back the parent node information, trace back from the target node to the starting point, and obtain the complete path; The searched path is optimized, Bezier curves are used to smooth the path to reduce sharp turns and acute angles in the path, redundant nodes are removed by merging nodes to reduce the path length, and a string of path points for navigating the low-speed autonomous driving car is output.

5. The low-speed automatic driving vehicle control method according to claim 1 is characterized in that: The step of determining the driving path of the low-speed automatic driving vehicle according to the path points of the low-speed automatic driving vehicle based on the TEB algorithm includes: Discretizing the path points used to navigate the low-speed autonomous driving vehicle into a series of points, and then connecting the points to form a preliminary elastic band; Adjusting the preliminary elastic band according to a path optimization target based on a preset path optimization algorithm to determine an optimized elastic band, wherein the path optimization target includes one or more of minimizing the length of the elastic band, minimizing the steering angle of the low-speed autonomous driving vehicle, and maximizing the distance between the elastic band and the obstacle; The optimized elastic band is used as a local path of the low-speed autonomous driving vehicle, and the low-speed autonomous driving vehicle is driven to travel along the local path.

6. The low-speed automatic driving vehicle control method according to any one of claims 1 to 5, characterized in that: The low-speed self-driving vehicle includes one or more of an unmanned express delivery vehicle and an intelligent transport vehicle.

7. A low-speed automatic driving vehicle control device, characterized in that: include: The dynamic target tracking module is configured to obtain the road condition image frames and the occupied grid map data of the low-speed autonomous driving car at various angles, and adopt the intersection-over-union matching algorithm based on the position of the moving object and the re-identification matching algorithm based on the appearance features to calculate and determine the confidence of each dynamic target detection frame between the current road condition image frame and the previous road condition image frame, so as to match and track the dynamic target position information of the current road condition image frame and the previous road condition image frame; a drivable road surface determination module, configured to infer surface normal information from the road condition image frame, extract feature maps from the RGB image and the inferred surface normal information using an RGB encoder and a surface normal encoder, respectively, and determine a drivable road surface area based on the fused RGB and surface normal information using a pre-trained semantic road segmentation model; A vehicle position determination module, configured to determine wheel odometer data and GPS data, and fuse the wheel odometer data and GPS data based on a preset extended Kalman filter algorithm to determine vehicle position information of the low-speed autonomous driving vehicle; a driving path determination module, configured to update the occupancy grid map data according to the dynamic target position information, the drivable road area, and the vehicle position information, determine the path points for navigating the low-speed autonomous driving vehicle according to the updated occupancy grid map data based on the A* algorithm, and determine the driving path of the low-speed autonomous driving vehicle according to the path points of the low-speed autonomous driving vehicle based on the TEB algorithm; The automatic driving control module is configured to respond to the control instructions of the low-speed automatic driving vehicle and drive the low-speed automatic driving vehicle to travel along the driving path to complete the control of the low-speed automatic driving vehicle.

8. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 6 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images