Vehicle control method, device, equipment and medium

Abstract

The invention discloses a vehicle control method. The method comprises the steps that map data, point cloud data and view angle images of different view angles of a road in the driving direction of a vehicle are acquired; detecting the road abnormal area based on the map data to obtain a first abnormal area detection result; detecting the road abnormal region based on the point cloud data and the view angle images of different view angles to obtain a second abnormal region detection result; judging whether the road has an abnormal area according to the first abnormal area detection result and the second abnormal area detection result to obtain a judgment result; the vehicle is controlled according to the judgment result; according to the method, the abnormity in front of the road is detected by combining the map data, the point cloud data of the road and the multi-view image, the information provided by different sensors is fully utilized to detect the abnormal event of the road, and the vehicle is braked or a driver is reminded to check the road condition when the abnormal area is detected; and the risk of traffic accidents is greatly reduced.

Description

Technical Field

[0001] The present application relates to the field of vehicle control technology, and in particular to a vehicle control method, device, equipment and medium. Background Art

[0002] In recent years, road safety has become an extremely important part of road management, and road collapse has also become an important factor affecting road safety. Road collapse may cause vehicle damage, driving difficulties, and even traffic accidents. Road collapse can cause vehicles to bump during driving, affecting driving stability and increasing driving risks. Summary of the invention

[0003] In view of the above-mentioned shortcomings of the prior art, the present application provides a vehicle control method, device, equipment and medium to solve at least one defect in the prior art.

[0004] To achieve the above objectives and other objectives, the present application provides a vehicle control method, the vehicle control method comprising:

[0005] Obtaining map data, point cloud data and perspective images of the road in the direction of vehicle travel;

[0006] Detecting the abnormal area of ​​the road based on the map data to obtain a first abnormal area detection result;

[0007] Detecting the abnormal area of ​​the road based on the point cloud data and the perspective images of different perspectives to obtain a second abnormal area detection result;

[0008] Determine whether there is an abnormal area on the road according to the first abnormal area detection result and the second abnormal area detection result, and obtain a determination result;

[0009] The vehicle is controlled according to the determination result.

[0010] In one embodiment of the present application, when the judgment result includes the existence of an abnormal area, the vehicle control method further includes:

[0011] Determining whether the abnormal area can be avoided according to the position information of the abnormal area on the road surface;

[0012] If the abnormal area can be avoided, an avoidance prompt is issued to the vehicle;

[0013] If the abnormal area cannot be avoided, a danger warning is issued to the vehicle.

[0014] In one embodiment of the present application, judging whether the abnormal area can be avoided according to the position information of the abnormal area on the road surface includes:

[0015] Determine the target lane where the abnormal area is located according to the position information;

[0016] determining an adjacent lane adjacent to the target lane;

[0017] According to the shortest distance between the lane line of the adjacent lane and the edge of the abnormal area;

[0018] When the shortest distance is greater than the safe passing distance, the abnormal area can be avoided;

[0019] When the shortest distance is less than the safe passing distance, the abnormal area cannot be avoided.

[0020] In one embodiment of the present application, the detecting of the abnormal road area based on the point cloud data and the perspective images of different perspectives to obtain the second abnormal area detection result includes:

[0021] Detecting the abnormal area of ​​the road according to the point cloud data to obtain a first detection result; wherein, when the confidence of the first detection result is greater than a first preset confidence, the first detection result is that an abnormal area exists;

[0022] Detecting the abnormal area of ​​the road according to the perspective images of different perspectives to obtain a second detection result; wherein, when the confidence of the second detection result is greater than a second preset confidence, the second detection result is that an abnormal area exists;

[0023] When any one of the first detection result and the second detection result indicates that an abnormal area exists, the second abnormal area detection result indicates that an abnormal area exists.

[0024] In one embodiment of the present application, judging whether there is an abnormal area on the road according to the first abnormal area detection result and the second abnormal area detection result includes:

[0025] When the confidence level of the first abnormal area detection result is greater than a third preset confidence level, the first abnormal area detection result is that an abnormal area exists;

[0026] When both the first abnormal area detection result and the second abnormal area detection result indicate that an abnormal area exists, the judgment result is that an abnormal area exists on the road.

[0027] In one embodiment of the present application, the abnormal road area is detected according to the perspective images of different perspectives, and the second detection result obtained includes:

[0028] Extracting features from the perspective images of different perspectives to obtain multiple perspective image features;

[0029] converting the plurality of perspective image features into a bird's-eye view space;

[0030] fusing multiple perspective image features in the bird's-eye view space to obtain a bird's-eye view fusion feature;

[0031] The abnormal area of ​​the road is detected based on the bird's-eye view fusion feature to obtain a third abnormal area detection result.

[0032] In one embodiment of the present application, after the step of acquiring the map data, point cloud data and perspective images of different perspectives of the road, the method further includes: performing time synchronization and spatial calibration on the map data, point cloud data and perspective images of different perspectives of the road.

[0033] To achieve the above objectives and other related objectives, the present application provides a vehicle control device, the vehicle control device comprising:

[0034] A data acquisition module is used to acquire map data, point cloud data and perspective images of different perspectives of the road in the direction of vehicle travel;

[0035] A first detection module, configured to detect an abnormal road area based on the map data to obtain a first abnormal area detection result;

[0036] A second detection module is used to detect the abnormal area of ​​the road based on the point cloud data and the perspective images of different perspectives to obtain a second abnormal area detection result;

[0037] an abnormality judgment module, used for judging whether there is an abnormal area on the road according to the first abnormal area detection result and the second abnormal area detection result, and obtaining a judgment result;

[0038] A vehicle control module is used to control the vehicle according to the judgment result.

[0039] To achieve the above objectives and other related objectives, the present application provides a vehicle control device, the vehicle control device comprising:

[0040] A data acquisition module, used to acquire road map data, road point cloud data, and multi-view images of the road;

[0041] A first detection module, used to detect abnormal road areas based on road map data to obtain a first abnormal area detection result;

[0042] A second detection module is used to input the point cloud data of the road and the multi-view images of the road into a pre-trained road anomaly detection model, detect the road abnormal area through the road anomaly detection model, and obtain a second abnormal area detection result;

[0043] An abnormality judgment module, used for judging whether there is an abnormal area on the road according to the first abnormal area detection result and / or the second abnormal area detection result;

[0044] The vehicle control module is used to control the vehicle based on the judgment result of whether there is an abnormal area on the road.

[0045] To achieve the above objectives and other related objectives, the present application provides a device, including:

[0046] one or more processors; and

[0047] The memory is used to store one or more programs. When the one or more programs are executed by the one or more processors, the memory implements the method described.

[0048] To achieve the above objectives and other related objectives, the present application provides one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to execute the described methods.

[0049] Beneficial effects of this application:

[0050] A vehicle control method of the present application includes: acquiring map data, point cloud data and perspective images of different perspectives of a road in the direction of vehicle travel; detecting an abnormal area of ​​the road based on the map data to obtain a first abnormal area detection result; detecting an abnormal area of ​​the road based on the point cloud data and the perspective images of different perspectives to obtain a second abnormal area detection result; judging whether there is an abnormal area on the road according to the first abnormal area detection result and the second abnormal area detection result to obtain a judgment result; and controlling the vehicle according to the judgment result. The present application detects anomalies in front of the road in combination with map data, point cloud data of the road and multi-perspective images of the road, makes full use of information provided by different sensors to detect abnormal road events, and brakes the vehicle or reminds the driver to detect the road conditions when an abnormal area is detected, thereby greatly reducing the risk of traffic accidents, improving the driver's survival probability and emergency handling capabilities when encountering sudden abnormal road events such as road collapse and road cracking, and providing strong safety protection for the driver in a complex and changeable road environment.

[0051] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0053] Figure 1 A flow chart of a vehicle control method according to an embodiment of the present application;

[0054] Figure 2 This is a flow chart of obtaining a second abnormal area detection result according to an embodiment of the present application;

[0055] Figure 3 This is a schematic diagram of determining whether there is an abnormal area based on point cloud data and view images according to an embodiment of the present application;

[0056] Figure 4 This is a schematic diagram of determining whether there is an abnormal area on a road based on map data according to an embodiment of the present application;

[0057] Figure 5 This is a schematic diagram of determining whether there is an abnormal area based on the first abnormal area detection result and the second abnormal area detection result according to an embodiment of the present application;

[0058] Figure 6 A schematic diagram of a vehicle control method according to an embodiment of the present application;

[0059] Figure 7 A schematic diagram of the principle of a vehicle control device according to an embodiment of the present application;

[0060] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing the memory of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0061] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0062] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0063] Although the terms "first", "second", "A", and "B", etc. may be used herein to describe various elements, these elements should not be limited by these terms and are only used to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the following technology. The term "and / or" includes a combination of multiple related items or any of the multiple related items.

[0064] As used herein, unless the context indicates otherwise, the singular form is intended to include the plural form, and it will be understood that the term "comprising" means the presence of stated features, quantities, steps, operations, elements, or combinations thereof, but does not exclude the presence or addition of one or more other features, quantities, steps, operations, elements, components, or combinations thereof.

[0065] Before the detailed description, it is intended to clarify that the division of components in this specification is divided only by the main function of each component. That is, two or more components to be described below can be combined into one component, or can be divided into two or more components according to more detailed functions. In addition to the main function of the component, each component to be described below can also perform some or all of the functions of other components, and some of the main functions of each component can be exclusively performed by other components.

[0066] See also Figure 1 , Figure 1 This is a flow chart of a vehicle control method according to an embodiment of the present application. Figure 1 As shown, a vehicle control method at least includes steps S110 to S150:

[0067] Step S110, obtaining map data, point cloud data and perspective images of different perspectives of the road in the direction of vehicle travel;

[0068] As an important part of the in-vehicle navigation system, high-precision maps collect and integrate a large amount of structured map data to output road conditions. These map data include detailed geometric information and attribute information of the road. Based on the current capabilities of high-precision maps, they can directly output the road ahead information, information confidence, and judge the abnormality of the road ahead.

[0069] The point cloud data of the road can be collected by the on-board laser radar, which uses 3D point clouds to accurately "describe" the surrounding environment for the self-driving car. The point cloud data contains information such as the distance, angle, and reflection intensity of the target. It does not rely on external lighting conditions or the radiation characteristics of the target itself. It can better perceive the road environment around the car and provide a basis for decision-making for the perception system. The laser radar generates detailed point cloud data by continuously emitting and receiving laser beams. These data describe the road conditions of the road ahead in detail, including but not limited to terrain slope, terrain changes, abnormal ground features, etc. In addition, the laser radar can also evaluate the confidence of the information, that is, the reliability of the data, based on the density and distribution of the point cloud, to provide a basis for subsequent decision-making. This ability is crucial for detecting potential dangers in advance in complex and changing road environments.

[0070] Among them, the terrain slope refers to the slope of the road ahead. When the road collapses, the downhill slope of the road is ≥75° (this parameter can be calibrated according to the actual road); terrain change refers to the sudden disappearance of the road ahead; abnormal ground features refer to the sinking, uplift or fracture of a large area of ​​ground ahead.

[0071] The perspective images of different perspectives of the road are image data taken by cameras of multiple perspectives on the vehicle. In the present embodiment, the multi-perspective images can be collected by a three-eye camera system, and the multi-perspective images further enhance the visual perception ability of the vehicle. It captures images of different angles by installing three cameras at different positions on the front of the vehicle, and fuses these images into a three-dimensional image through corresponding image processing algorithms. This process not only improves the spatial resolution of the image, but also realizes the accurate measurement of attributes such as distance, size, and shape of the object. More importantly, the three-eye camera can effectively remove errors and improve the accuracy and confidence of information by comparing the images of three perspectives, so as to more accurately identify key information such as pedestrians, vehicles, traffic signs, and road surface changes on the road.

[0072] Step S120, detecting the abnormal area of ​​the road based on the map data to obtain a first abnormal area detection result;

[0073] After the map data of the road is acquired, it is determined based on the map data whether there is an abnormal area on the road ahead of the vehicle to complete the detection of the abnormal area of ​​the road and obtain a first abnormal area detection result.

[0074] Step S130, detecting the abnormal area of ​​the road based on the point cloud data and the perspective images of different perspectives to obtain a second abnormal area detection result;

[0075] Here, the detection of abnormal road areas based on point cloud data and perspective images of different perspectives includes detecting abnormal road areas based on point cloud data to obtain a first detection result and detecting abnormal road areas based on perspective images of different perspectives to obtain a second detection result, and then fusing the first detection result and the second detection result to obtain a second abnormal area detection result. By combining the precise detection of the laser radar with the stereoscopic vision of the multi-camera, a comprehensive and accurate perception of the road environment ahead is achieved.

[0076] Step S140, judging whether there is an abnormal area on the road according to the first abnormal area detection result and the second abnormal area detection result, and obtaining a judgment result;

[0077] As an example, judging whether there is an abnormal area on the road based on the first abnormal area detection result and the second abnormal area detection result includes: when the first abnormal area detection result and the second abnormal area detection result both indicate that there is an abnormal area on the driving road surface, determining that there is an abnormal area.

[0078] Step S150: Control the vehicle according to the judgment result.

[0079] Assume that there is an abnormal area ahead of the road, such as a road collapse, the driver is reminded of the abnormality of the road ahead at a location at a first distance from the abnormal area, and a request is sent to the driver whether to stop and check. When the driver agrees to the inspection request, the vehicle is braked at a location at a second distance from the abnormal area to remind the driver to get off the vehicle and check the road conditions.

[0080] See also Figure 2 , Figure 2 This is a flow chart of obtaining the second abnormal area detection result according to an embodiment of the present application. Figure 2 As shown, detecting the abnormal area of ​​the road based on the point cloud data and the perspective images of different perspectives to obtain the second abnormal area detection result includes steps S210 to S230:

[0081] Step S210, detecting the abnormal area of ​​the road according to the point cloud data to obtain a first detection result; wherein, when the confidence of the first detection result is greater than a first preset confidence, the first detection result is that an abnormal area exists;

[0082] As an example, the point cloud data of the road is input into a pre-trained first road anomaly detection model, and the road anomaly area is detected by the first road anomaly detection model to obtain a first detection result;

[0083] As an example, feature extraction is performed on point cloud data of the road through the backbone network in the feature extraction layer in the first road anomaly detection model to obtain point cloud features, and abnormal road areas are detected based on the point cloud features to obtain a first detection result.

[0084] The first road anomaly detection model is a model pre-trained through machine learning, which can detect abnormal areas (for example, potholes and holes on the road) that appear in the passable area, and then can guide the vehicle to pass based on the detected abnormal areas to improve driving safety. When the first road anomaly detection model detects the abnormal area of ​​the road, it will output a detection result and a corresponding confidence level. When the confidence level exceeds the first preset confidence level, the detection result is considered correct. That is, when the confidence level output by the first road anomaly detection model is greater than the first preset confidence level, the first detection result indicates that there is an abnormal area. The first preset confidence level can be set as needed. As an example, the first preset confidence level is set to 0.95, that is, when the confidence level of the first detection result is greater than 0.95, it can be considered that the first detection result is that there is an abnormal area on the road.

[0085] Step S220, detecting the abnormal area of ​​the road according to the perspective images of different perspectives to obtain a second detection result; wherein, when the confidence of the second detection result is greater than a second preset confidence, the second detection result is that an abnormal area exists;

[0086] As an example, the perspective images of different perspectives are input into a pre-trained second road anomaly detection model, and the road anomaly area is detected by the second road anomaly detection model to obtain a second detection result;

[0087] The second road anomaly detection model is a model pre-trained through machine learning, which can detect abnormal areas (for example, potholes and holes on the road) that appear in the passable area, and then can guide the vehicle to pass based on the detected abnormal areas to improve driving safety. When the second road anomaly detection model detects the abnormal area of ​​the road, it will output a detection result and a corresponding confidence level. When the confidence level exceeds the second preset confidence level, the detection result is considered correct. That is, when the confidence level output by the second road anomaly detection model is greater than the second preset confidence level, the second detection result indicates that there is an abnormal area. The second preset confidence level can be set as needed. As an example, the second preset confidence level is set to 0.95, that is, when the confidence level of the second detection result is greater than 0.95, it can be considered that the second detection result is that there is an abnormal area on the road.

[0088] In one embodiment, the abnormal road area is detected according to the perspective images of different perspectives, and the second detection result obtained includes:

[0089] Feature extraction is performed on perspective images of different perspectives to obtain multiple perspective image features; the multiple perspective image features are converted into a bird's-eye view space; the multiple perspective image features are fused in the bird's-eye view space to obtain a bird's-eye fusion feature; abnormal road areas are detected based on the bird's-eye fusion feature to obtain a third abnormal area detection result.

[0090] As an example, the backbone network in the feature extraction layer of the first road anomaly detection model is used to extract features of perspective images of different perspectives to obtain multiple perspective image features; the multi-perspective image features are converted into a bird's-eye view space - BEV space (Bird's Eye View, BEV) through the cross-attention network Transformer in the feature extraction layer.

[0091] Among them, the backbone network can use CNN (Convolutional Neural Networks, CNN) network to input the 2D image data captured by multiple cameras into the shared backbone network (Backbone). These image data contain visual information of the vehicle's surrounding environment. The image data is feature extracted through the CNN network, and high-level features in the image are extracted to obtain two-dimensional multiple-view image features, including: image edge, texture, color and other information. The multiple-view image features extracted from the 2D image are transformed into the BEV bird's-eye view space through the Transformer network architecture to obtain the BEV image features, realizing the conversion from two-dimensional to three-dimensional. The cross-attention mechanism of Transformer can handle the conversion between multiple sequences and convert the two-dimensional features captured by the camera into features in three-dimensional space. In the conversion process, it may be necessary to use technologies such as inverse perspective transformation (IPM, Inverse Perspective Mapping), and obtain the homography matrix from the camera plane to the ground plane through the internal and external parameter calibration of multiple cameras to realize the conversion from plane to plane.

[0092] After obtaining the features of multiple perspective images, the features of multiple perspective images are fused through splicing, weighted summation, and attention mechanism to obtain bird's-eye view fusion features. Splicing is to splice different features together in a certain order to form a longer feature vector; weighted summation is to multiply different features by different weights and then add them together to obtain the final feature vector. The attention mechanism calculates the similarity between different features and then fuses the features according to the size of the similarity. Among them, feature fusion can be performed through the cross-attention mechanism of network architectures such as Transformer.

[0093] Through feature fusion, features from different sources are generated into a unified feature representation containing multi-source information, which can extract more comprehensive and accurate information.

[0094] As an example, the fused bird's-eye view fusion features can also be optimized to improve the accuracy of subsequent tasks. The optimization process includes feature dimension reduction, feature enhancement and other technologies.

[0095] Step S230: When any one of the first detection result and the second detection result indicates that an abnormal area exists, the second abnormal area detection result indicates that an abnormal area exists.

[0096] See also Figure 3 , Figure 3 Schematic diagram of determining whether there is an abnormal area based on the first detection result and the second detection result. Figure 3 When the first detection result is that an abnormal detection area exists and the second detection result is that an abnormal detection area does not exist, the second abnormal area detection result is that an abnormal area exists; or when the first detection result is that an abnormal detection area does not exist and the second detection result is that an abnormal detection area exists, the second abnormal area detection result is that an abnormal area exists, or when the first detection result is that an abnormal detection area exists and the second detection result is that an abnormal detection area exists, the second abnormal area detection result is that an abnormal area exists.

[0097] In one embodiment, when the judgment result includes the existence of an abnormal area, the vehicle control method further includes: judging whether the abnormal area can be avoided based on the position information of the abnormal area on the road surface; if the abnormal area can be avoided, issuing an avoidance prompt to the vehicle; if the abnormal area cannot be avoided, issuing a danger prompt to the vehicle.

[0098] After detecting the existence of an abnormal area on the road ahead, it can be determined whether the abnormal area can be avoided, and corresponding prompts can be given on the impact of the abnormal area on vehicle traffic, thereby improving the safety of vehicle traffic.

[0099] In one embodiment, the step of judging whether the abnormal area can be avoided based on the position information of the abnormal area on the road surface includes: determining the target lane where the abnormal area is located based on the position information; determining the adjacent lane adjacent to the target lane; based on the closest distance between the lane line of the adjacent lane and the edge of the abnormal area; when the closest distance is greater than the safe passing distance, the abnormal area can be avoided; when the closest distance is less than the safe passing distance, the abnormal area cannot be avoided.

[0100] The safe traveling distance is determined based on the lateral distance between the vehicle wheels. The safe traveling distance represents the road width required for the vehicle to pass, and can usually be greater than or equal to the distance between the two front wheels or the distance between the two rear wheels of the vehicle.

[0101] After identifying the lane where the vehicle is located and taking it as the target lane, the adjacent lanes adjacent to the target lane are determined, and then judging whether the required distance for safe passage is met based on the closest distance between the boundary of the abnormal area and the lane line of the target lane, if the closest distance is greater than the safe passage distance, the abnormal area can be avoided; if the closest distance is less than the safe passage distance, the abnormal area cannot be avoided.

[0102] By using the lane lines of the adjacent lanes of the lane where the abnormal area is located and the position information of the abnormal area, it is possible to judge whether the vehicle can avoid the abnormal area, and then issue an avoidance prompt in time when the vehicle can avoid the abnormal area, and issue a danger prompt in time when the vehicle cannot avoid the abnormal area, thereby improving vehicle driving safety.

[0103] In one embodiment, the step of judging whether there is an abnormal area on the road based on the first abnormal area detection result and the second abnormal area detection result includes: when the confidence level of the first abnormal area detection result is greater than a third preset confidence level, the first abnormal area detection result is that an abnormal area exists; when both the first abnormal area detection result and the second abnormal area detection result are that an abnormal area exists, the judgment result is that there is an abnormal area on the road.

[0104] When abnormality detection is performed on roads based on map data, the detection results and the corresponding third confidence level are output. Figure 4 As shown, when the third confidence level is greater than the third preset confidence level, it is considered that an abnormal area exists. The third preset confidence level can be set as needed. As an example, the third preset confidence level is set to 0.95, that is, when the confidence level of the first abnormal area detection result is greater than 0.95, it can be considered that the first abnormality detection result is that there is an abnormal area on the road.

[0105] The second abnormal area detection result determined based on the first detection result and the second detection result and the first abnormal area detection result determined based on the map data determines whether there is an abnormal area. Figure 5 As shown, when both the first abnormal area detection result and the second abnormal area detection result indicate that an abnormal area exists, it is determined that an abnormal area exists on the road.

[0106] In one embodiment, after the step of acquiring the map data of the road, the point cloud data of the road and the multi-view image of the road, the method further includes: performing time synchronization and spatial calibration on the map data of the road, the point cloud data of the road and the multi-view image of the road.

[0107] Since the road map data, the road point cloud data and the perspective images of the road at different perspectives come from different cameras and sensors respectively, it is necessary to ensure that the data between different cameras or sensors are synchronized and calibrated. Therefore, the road map data, the road point cloud data and the multi-perspective images of the road are time synchronized and spatially calibrated.

[0108] See also Figure 6 , Figure 6 This is a schematic diagram of a vehicle control method according to an embodiment of the present application. The point cloud data collected by the laser radar and the view image collected by the three-eye camera are used to detect abnormal areas respectively, and a first detection result and a second detection result are obtained. The first detection result and the second detection result are sensed and fused to determine the road condition ahead. The road condition ahead is determined by the map data provided by the vehicle navigation, and then a planning decision is made based on the road condition ahead obtained by sense fusion and the road condition ahead determined by the map data to determine whether there is an abnormal area on the road ahead. If there is an abnormal area, a message indicating the existence of the abnormal area is output to the in-vehicle interactive system (HMI). The in-vehicle interactive system sends a request to the driver whether to stop and check. After the driver agrees to the inspection request, the vehicle stops in front of the abnormal area to remind the driver to get off and check the road condition.

[0109] This application realizes comprehensive and accurate perception of the road environment ahead by real-time analysis and processing of map data, point cloud data, and perspective images from different perspectives, and then predicts whether there are abnormal areas on the road ahead based on the map data, point cloud data, and perspective images from different perspectives. When an abnormal area appears, it can automatically warn the driver, or automatically take safety measures such as braking, deceleration, and avoidance when necessary, thereby greatly reducing the risk of traffic accidents, effectively improving the safety and stability of car driving, and bringing drivers a more secure and comfortable travel experience.

[0110] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0111] Figure 7 FIG. 1 is a block diagram of a vehicle control device according to an embodiment of the present application. Figure 7 As shown, a vehicle control device includes:

[0112] The data acquisition module 710 is used to acquire the map data, point cloud data and perspective images of different perspectives of the road in the direction of vehicle travel;

[0113] A first detection module 720, configured to detect an abnormal road area based on map data and obtain a first abnormal area detection result;

[0114] The second detection module 730 is used to detect the abnormal area of ​​the road based on the point cloud data and the perspective images of different perspectives to obtain a second abnormal area detection result;

[0115] An abnormality determination module 740 is used to determine whether there is an abnormal area on the road according to the first abnormal area detection result and the second abnormal area detection result, and obtain a determination result;

[0116] The vehicle control module 750 is used to control the vehicle according to the judgment result.

[0117] It should be noted that the vehicle control device provided in the above embodiment and the vehicle control method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the vehicle control device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0118] An embodiment of the present application also provides a device, including: one or more processors; and a memory for storing one or more programs, when the one or more programs are executed by one or more processors, the memory implements the vehicle control method in the above embodiment.

[0119] Embodiments of the present application also provide one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to execute the vehicle control method in the above-mentioned embodiment.

[0120] Figure 8 The schematic diagram of the structure of a computer system suitable for implementing the memory of the embodiment of the present application is shown. It should be noted that: Figure 8 The computer system of the memory shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0121] like Figure 8As shown, the computer system includes a central processing unit (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) or the program loaded from the storage part to the random access memory (RAM), such as the method in the above embodiment. In the RAM, various programs and data required for system operation are also stored. The CPU, ROM and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus.

[0122] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that a computer program read therefrom is installed into the storage part as needed.

[0123] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the vehicle control method implemented above. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), various functions defined in the system of the present application are executed.

[0124] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0125] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0126] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0127] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes the vehicle control method as described above. The computer-readable storage medium may be included in the memory described in the above embodiment, or may exist independently without being assembled into the memory.

[0128] Another aspect of the present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the vehicle control method provided in each of the above embodiments.

[0129] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A vehicle control method, characterized in that: The vehicle control method comprises: Obtaining map data, point cloud data and perspective images of the road in the direction of vehicle travel; Detecting the abnormal area of ​​the road based on the map data to obtain a first abnormal area detection result; Detecting the abnormal area of ​​the road based on the point cloud data and the perspective images of different perspectives to obtain a second abnormal area detection result; Determine whether there is an abnormal area on the road according to the first abnormal area detection result and the second abnormal area detection result, and obtain a determination result; The vehicle is controlled according to the determination result.

2. The vehicle control method according to claim 1, characterized in that: When the judgment result includes the existence of an abnormal area, the vehicle control method further includes: Determining whether the abnormal area can be avoided according to the position information of the abnormal area on the road surface; If the abnormal area can be avoided, an avoidance prompt is issued to the vehicle; If the abnormal area cannot be avoided, a danger warning is issued to the vehicle.

3. The vehicle control method according to claim 2, characterized in that: The step of judging whether the abnormal area can be avoided according to the position information of the abnormal area on the road surface comprises: Determine the target lane where the abnormal area is located according to the position information; determining an adjacent lane adjacent to the target lane; According to the shortest distance between the lane line of the adjacent lane and the edge of the abnormal area; When the shortest distance is greater than the safe passing distance, the abnormal area can be avoided; When the shortest distance is less than the safe passing distance, the abnormal area cannot be avoided.

4. The vehicle control method according to claim 1, characterized in that: The detecting of the abnormal road area based on the point cloud data and the perspective images of different perspectives to obtain a second abnormal area detection result includes: Detecting the abnormal area of ​​the road according to the point cloud data to obtain a first detection result; wherein, when the confidence of the first detection result is greater than a first preset confidence, the first detection result is that an abnormal area exists; Detecting the abnormal area of ​​the road according to the perspective images of different perspectives to obtain a second detection result; wherein, when the confidence of the second detection result is greater than a second preset confidence, the second detection result is that an abnormal area exists; When any one of the first detection result and the second detection result indicates that an abnormal area exists, the second abnormal area detection result indicates that an abnormal area exists.

5. The vehicle control method according to claim 4, characterized in that: The determining whether there is an abnormal area on the road according to the first abnormal area detection result and the second abnormal area detection result comprises: When the confidence level of the first abnormal area detection result is greater than a third preset confidence level, the first abnormal area detection result is that an abnormal area exists; When both the first abnormal area detection result and the second abnormal area detection result indicate that an abnormal area exists, the judgment result is that an abnormal area exists on the road.

6. The vehicle control method according to claim 4, characterized in that: Detecting the abnormal road area according to the perspective images of different perspectives to obtain the second detection result includes: Extracting features from the perspective images of different perspectives to obtain multiple perspective image features; converting the plurality of perspective image features into a bird's-eye view space; fusing multiple perspective image features in the bird's-eye view space to obtain a bird's-eye view fusion feature; The abnormal area of ​​the road is detected based on the bird's-eye view fusion feature to obtain a third abnormal area detection result.

7. The vehicle control method according to claim 1, characterized in that: After the step of acquiring the map data, point cloud data and perspective images of different perspectives of the road, the method further comprises: performing time synchronization and space calibration on the map data, point cloud data and perspective images of different perspectives of the road.

8. A vehicle control device, characterized in that: The vehicle control device comprises: A data acquisition module is used to acquire map data, point cloud data and perspective images of different perspectives of the road in the direction of vehicle travel; A first detection module, configured to detect an abnormal road area based on the map data to obtain a first abnormal area detection result; A second detection module is used to detect the abnormal area of ​​the road based on the point cloud data and the perspective images of different perspectives to obtain a second abnormal area detection result; an abnormality judgment module, used for judging whether there is an abnormal area on the road according to the first abnormal area detection result and the second abnormal area detection result, and obtaining a judgment result; A vehicle control module is used to control the vehicle according to the judgment result.

9. A vehicle control device, characterized in that: include: one or more processors; and A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the memory implements the vehicle control method as described in any one of claims 1 to 7.

10. A machine-readable medium, characterized in that Instructions are stored thereon, which, when executed by one or more processors, enable the processors to execute the vehicle control method as described in any one of claims 1-7.

Images