Method and System for Constructing Road Model

Abstract

Methods and apparatuses for constructing a real-time road model are provided. The method may include and the apparatus may be used to: at a first moment, obtain a first set of sensor data output by a plurality of different types of sensors loaded on a vehicle; feed the first set of sensor data into a general statistical model to form general statistical model format data for the first moment; extract map data within a threshold range around the position where the vehicle is located at the first moment from a map; correct each sensor data set in the general statistical model format data for the first moment based on the map data; fuse the general statistical model format data for the first moment with historical general statistical model format data to update the general statistical model format data for the first moment; compare the updated general statistical model format data for the first moment with an implicit model to identify an object; and combine the identified object with the map data to form a real-time road model.

Description

Technical Field

[0001] The present invention relates to constructing a road model, and more particularly, to constructing a road model using real-time sensor data and an offline map. Background Art

[0002] An autonomous vehicle (also known as a driverless car, self-driving car, robotic car) is a vehicle that can sense its surrounding environment and navigate without human input. Autonomous vehicles (hereinafter referred to as "ADVs") use various technologies to detect their surrounding environment, such as radar, lidar, GPS, rangefinding, and computer vision. Advanced control systems interpret the sensed information to identify appropriate navigation paths, as well as obstacles and relevant signs.

[0003] More specifically, an ADV collects sensor data from various in-vehicle sensors, such as vision sensors (e.g., cameras), radar-based rangefinding sensors (such as lidar, millimeter-wave radar, ultrasonic radar), etc. Based on the sensor data, the ADV can construct a real-time road model around it. The road model can include various information, including but not limited to lane information (such as the position, type, width, etc. of lane lines), traffic lights, traffic signs, road boundaries, etc. By comparing the constructed road model with a pre-acquired road model (e.g., the road model included in the HD map provided by an HD map provider), the ADV can more accurately determine its position on the road. At the same time, the ADV can also identify objects around it based on the sensor data, such as vehicles, pedestrians, and buildings. The ADV can make appropriate driving decisions, such as lane change, acceleration, braking, etc., according to the determined road model and the identified surrounding objects.

[0004] As is known in the art, different types of sensors produce different forms or formats of data. When processing sensor data from different sensors, each type of sensor data must be processed separately. Therefore, for each type of sensor data, one or more models for storing that type of sensor data must be established for object identification. Currently, there does not exist a model that can support multiple different types of sensor data simultaneously.

[0005] In addition, a single set of sensor data obtained for a single moment is unstable and unreliable. For example, at a certain moment, an object on the road (such as a vehicle) may block the sensor, an object on the road (such as a lane marker) may be blocked by other vehicles on the road or the sensor may shake due to the jitter of the vehicle. In either case, incorrect sensor data obtained by the ADV's sensor may lead to the construction of an incorrect road model. Therefore, it is desirable to compare the sensor data for that moment with prior information (such as a map) to correct the obviously incorrect sensor data, and to fuse the sensor data for that moment with multiple sets of sensor data for multiple previous moments to fit the data, in order to construct a real-time road model.

[0006] Therefore, it is desirable to provide a solution that can support multiple types of sensor data simultaneously in a model and can combine historical data to construct a real-time road model, so as to overcome the above-mentioned defects. Summary of the Invention

[0007] The present invention content is provided to introduce in a simplified form some concepts that will be further described in the following detailed implementation. The present invention content is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter.

[0008] According to an embodiment of the present invention, a method for constructing a real-time road model is provided, including: at a first moment, obtaining a first set of sensor data output by a plurality of different types of sensors loaded on a vehicle; feeding the first set of sensor data into a general statistical model to form general statistical model format data for the first moment, the general statistical model format including a plurality of sensor data sets, wherein each sensor data set in the plurality of sensor data sets is output by one of the different types of sensors; extracting map data within a threshold range around the position where the vehicle is located at the first moment from a map; correcting one or more of the plurality of sensor data sets in the general statistical model format data for the first moment based on the map data; fusing the general statistical model format data for the first moment with historical general statistical model format data to update the general statistical model format data for the first moment; comparing the updated general statistical model format data for the first moment with an implicit model to identify objects, the implicit model including a plurality of sensor data sample sets for describing predefined objects, wherein each sensor data sample set in the plurality of sensor data sample sets includes a pre-acquired sensor data sample set for describing the predefined object by one of the different types of sensors; and combining the identified objects with the map data to form the real-time road model.

[0009] According to an embodiment of the present invention, there is provided a device for constructing a real-time road model, including: a sensor data acquisition module configured to acquire a first set of sensor data output by different types of sensors loaded on a vehicle at a first moment; a data feeding module configured to feed the first set of sensor data into a general statistical model to form general statistical model format data for the first moment, the general statistical model format including a plurality of sensor data sets, wherein each of the plurality of sensor data sets is output by one of the different types of sensors; a map data extraction module configured to extract map data within a threshold range around the position where the vehicle is located at the first moment from a map; a data correction module configured to correct one or more of the plurality of sensor data sets in the general statistical model format data for the first moment based on the map data; a data fusion module configured to fuse the general statistical model format data for the first moment with historical general statistical model format data to update the general statistical model format data for the first moment; an object identification module configured to compare the updated general statistical model format data for the first moment with an implicit model to identify an object, the implicit model including a plurality of sensor data sample sets for describing predefined objects, wherein each of the plurality of sensor data sample sets includes a pre-acquired sensor data sample set for describing the predefined object output by one of the different types of sensors; and a real-time road model formation module configured to combine the identified object with the map data to form the real-time road model.

[0010] According to an embodiment of the present invention, there is provided a vehicle, including: a plurality of different types of sensors; and the above device for constructing a real-time road model. The plurality of different types of sensors include vision sensors and radar ranging sensors, the vision sensors include cameras, and the radar includes one or more of lidar, ultrasonic radar, and millimeter-wave radar.

[0011] By adopting the methods, devices, and vehicles disclosed in the present invention, it is possible to support various different types of sensor data in one model, improving the accuracy of vehicle positioning at specific positions in the road model. In addition, by taking historical data into account, the accuracy of the obtained sensor data can be greatly improved. Moreover, by using map data as prior information to eliminate obvious errors in real-time sensor data, the constructed real-time road model is made more reliable.

[0012] These and other features and advantages will become apparent by reading the following detailed description and referring to the associated drawings. It should be understood that the foregoing general description and the following detailed description are illustrative only and do not limit the various aspects claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to understand in detail the manner in which the above-described features of the present invention are used, the above briefly summarized content may be described more specifically with reference to the embodiments, some aspects of which are shown in the drawings. However, it should be noted that the drawings only show certain typical aspects of the present invention and should not be considered to limit its scope, as the description may allow other equally effective aspects.

[0014] Figure 1 A schematic diagram showing an autonomous vehicle 100 with different types of sensors traveling on a road according to an embodiment of the present invention.

[0015] Figure 2 A flowchart showing a method 200 for constructing a real-time road model according to an embodiment of the present invention.

[0016] Figure 3 Schematic diagrams 301 and 302 showing data fusion for a general statistical model format according to an embodiment of the present invention.

[0017] Figure 4 Shows according to Figure 3 An embodiment of a flowchart of a method 400 for fusing general statistical model format data for a first moment with historical general statistical model format data.

[0018] Figure 5 A block diagram of an apparatus 500 for constructing a real-time road model according to an embodiment of the present invention.

[0019] Figure 6 A block diagram showing an exemplary computing device 600 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be described in detail below in conjunction with the drawings, and the features of the present invention will be further revealed in the following specific description.

[0021] The following detailed description refers to the drawings showing exemplary embodiments of the present invention. However, the scope of the present invention is not limited to these embodiments, but is defined by the appended claims. Therefore, embodiments outside those shown in the drawings, such as modified versions of the illustrated embodiments, are still encompassed by the present invention.

[0022] References to "one embodiment", "an embodiment", "an example embodiment", etc. in this specification mean that the embodiment may include a specific feature, structure, or characteristic, but each embodiment does not necessarily include the specific feature, structure, or characteristic. In addition, these phrases do not necessarily refer to the same embodiment. Further, when a specific feature, structure, or characteristic is described in connection with an embodiment, it should be understood that the specific feature, structure, or characteristic can be implemented in combination with other embodiments within the knowledge of those skilled in the relevant art, whether or not explicitly described.

[0023] For ease of explanation, only embodiments in which the technical solution of the present invention is applied to a "vehicle" or an "autonomous vehicle" (the two terms may be used interchangeably hereinafter) are described in detail herein. However, those skilled in the art can fully understand that the technical solution of the present invention can be applied to any vehicle capable of realizing unmanned autonomous driving, such as an airplane, a helicopter, a train, a subway, a ship, etc. Unless otherwise specified, the term "A or B" used in this specification refers to "A and B" and "A or B", rather than A and B being exclusive.

[0024] General statistical model

[0025] When an autonomous vehicle is driving on the road, it needs to know the real-time road conditions. For this purpose, various types of sensors are installed on the vehicle to act as the "eyes" of the vehicle. Currently, widely used sensors include vision sensors (e.g., cameras) and radar ranging sensors (such as lidar, millimeter-wave radar, ultrasonic radar). Each sensor has its own advantages and disadvantages. For example, a camera has a low cost, can identify different objects, and has advantages in aspects such as the measurement accuracy of object height and width, lane line recognition, and pedestrian recognition accuracy. It is an indispensable sensor for realizing functions such as lane departure warning and traffic sign recognition, but its operating range and ranging accuracy are inferior to those of radar, and it is easily affected by factors such as light and weather. Millimeter-wave radar is suitable for all-weather environments, is not affected by bad weather such as light, haze, and sandstorms, and can realize the recognition of moving and static obstacles. However, in the driving environment, the influence of the multi-band coexistence environment on millimeter-wave radar is relatively large, resulting in a low data accuracy situation. Lidar has a wide detection range and high detection accuracy, but its performance is poor in extreme weather such as rain, snow, and fog, and its cost is relatively high. Therefore, it is ideal to use two or more sensors to cover and cross-check each other during driving. Currently, autonomous vehicles widely adopt the following comprehensive solutions to achieve the purpose of safety redundancy: (1) combining a camera and millimeter-wave radar; (2) combining a camera and lidar; (3) combining a camera, millimeter-wave radar, and lidar.

[0026] Reference Figure 1, which shows a schematic diagram of an autonomous vehicle 100 driving on a road with different types of sensors. Although, for illustration purposes, Figure 1 The vehicle 100 in the figure uses a camera 101, a millimeter wave radar 102, and a laser radar 103 to identify an object, but those skilled in the art will fully understand that the technical solution of the present invention may include one or more other vehicle-mounted sensors 104. In addition, using more or fewer sensors, such as only using a camera 101 and a millimeter wave radar 102 or only using a camera 101 and a laser radar 103, is also within the scope of the present invention.

[0027] In the case of a vehicle with multiple types of sensors, each sensor records its own sensor data and provides it to the vehicle's central processing unit. The formats of sensor data provided by various types or various sensor manufacturers are usually different. Generally speaking, depending on the settings of the vehicle manufacturer or the sensor manufacturer, the sensor outputs raw sensor data or outputs data after preprocessing the raw sensor data (e.g., feature extraction, target extraction, etc.). For example, for a vehicle such as Figure 1 For the same object such as the lane line 105 in the image, the camera 101 outputs camera data representing the lane line 105, such as raw image data or image features extracted from the raw image data. The millimeter wave radar 102 outputs millimeter wave radar data representing the lane line 104, such as raw millimeter wave radar data or polygon sequence data constructed from the raw millimeter wave radar data. The laser radar 103 outputs laser radar data representing the lane line 105, such as raw laser radar data or three-dimensional point cloud data constructed from raw laser radar data. Of course, the data formats of the sensor outputs listed above are merely illustrative, and those skilled in the art can fully understand that any data format of the sensor output is within the scope of the present invention.

[0028] Typically, for each type of sensor data, one or more models are used to record the sensor data of that type. However, this solution will generate multiple sensor data records. For example, at a certain time t, for the same object, multiple data records will be generated that respectively record the camera data output by the camera 101, the millimeter wave radar data output by the millimeter wave radar 102, and the laser radar data output by the laser radar 103. This solution not only consumes too much memory space, but also potentially causes a delay in data processing speed.

[0029] The present invention defines a general statistical model that can support multiple types of sensor data at the same time. The format of the general statistical model is as follows: {t,d s1 ,d s2 …d sn}, which represents a set of sensor data sets output by sensors 1 to n at time t. Among them, s1 represents sensor 1 installed on the vehicle, s2 represents sensor 2 installed on the vehicle, and sn represents sensor n installed on the vehicle (n is any integer greater than 1). These n sensors are different types of sensors for identifying objects, such as cameras, millimeter-wave radars, lidars, etc. Those skilled in the art can fully understand that depending on the specific configuration and requirements of the autonomous vehicle manufacturer, other types of sensors (such as ultrasonic sensors) are also included within the scope of the present invention.

[0030] d s1 represents the data set output by s1 at time t, d s2 represents the data set output by s2 at time t, d sn represents the data set output by sn (n is any integer greater than 1) at time t. Generally speaking, d s1 ……d sn Each data set in... has a different data format, such as camera data, millimeter-wave radar data, lidar data, etc. Those skilled in the art can fully understand that according to the sensors used, the above data sets can completely include sensor data output by other types of sensors.

[0031] By adopting this general statistical model, data sets output by multiple different types of sensors at a single moment can be integrated into one data model, thereby reducing the storage pressure and enabling more efficient data processing.

[0032] Hidden model

[0033] In the present invention, multiple implicit models are predefined, and each implicit model includes multiple different types of sensor data describing an object. Just to give a few examples for illustration, the object can be: road signs, lane lines, buildings, pedestrians, road edges, bridges, utility poles, elevated structures, traffic lights, or traffic signs, etc.

[0034] The format of this implicit model is: {pd s1 , pd s2 …pd sn , object}, which represents a set of different types of sensor data sample sets for describing an object. Among them, s1... sn respectively correspond to sensors s1... sn in the above general statistical model. pd s1 represents the sample set output by s1 obtained in advance for describing this object, pd s2 represents the sample set output by s2 obtained in advance for describing this object, and pd snDenote the sample set describing the object output by the pre-acquired sn. For example, if the object is a building, and s1 is a camera, s2 is a millimeter-wave radar, and s3 is a lidar, the implicit model is instantiated as: {pd 摄像头 , pd 毫米波雷达 , pd 激光雷达 , building}, where pd 摄像头 can include the camera data sample set describing the building, pd 毫米波雷达 can include the millimeter-wave radar data sample set describing the building, and pd 激光雷达 can include the lidar data sample set describing the building.

[0035] The sensor data sample set can be pre-acquired by an autonomous vehicle manufacturer, a sensor manufacturer, or a user. For example, as is known to those skilled in the art, vehicle manufacturers and sensor manufacturers can collect a large number of sensor data samples during the training of a target model or a road model, and label the collected sensor data samples through various algorithms or feature extraction methods. Thus, the sample data output by a certain sensor and labeled as describing the same kind of object can be grouped together to form the sensor data sample set of the sensor for the object.

[0036] For further illustration, only as an example, {pd 摄像头 , pd 毫米波雷达 , pd 激光雷达 , building} can be constructed in the following way. For example, during training, a vehicle manufacturer can drive a vehicle from point A to point B, during which sensors such as cameras, millimeter-wave radars, and lidars acquire a large number of sensor data samples. After processing each of these sensor data samples, each sensor data sample can be labeled to identify a certain kind of object (e.g., road signs, lane lines, buildings, pedestrians, road edges, bridges, utility poles, elevated structures, or traffic signs, etc.). Then, multiple camera data samples identifying the same kind of object (e.g., buildings) are grouped into pd 摄像头 for the object, multiple millimeter-wave radar data samples identifying the same kind of object (e.g., buildings) are grouped into pd 毫米波雷达 for the object, and multiple lidar data samples identifying the same kind of object are grouped into pd 激光雷达 for the object. Finally, the above three sensor data sample sets are fed into the implicit model to be instantiated as the implicit model for buildings, i.e., {pd 摄像头 , pd 毫米波雷达 , pd 激光雷达 , building}.

[0037] In addition, the number of samples included in the sensor data sample set can vary according to specific hardware limitations, usage scenarios, and user requirements. According to an embodiment of the present invention, the sensor data sample set can be pre-stored in the storage device of the autonomous vehicle or obtained in real time through the network from the server of the autonomous vehicle manufacturer, the server of the sensor manufacturer, or various cloud services.

[0038] According to an embodiment of the present invention, after obtaining real-time sensors, objects can be identified by comparing the sensor data with the sensor data sample set. The specific method will be described in detail below.

[0039] Implementation method

[0040] Figure 2 A flowchart of a method 200 for constructing a real-time road model according to an embodiment of the present invention is described. For example, the method 200 can be implemented within at least one processor (e.g., Figure 6 processor 604), which can be located in an in-vehicle computer system, a remote server, or a combination thereof. Of course, in various aspects of the present invention, the method 200 can also be implemented by any suitable device capable of performing related operations.

[0041] The method 200 starts at step 210. At step 210, at a first moment (in the following description, the "first moment" is understood as "in real time"), a first set of sensor data output by a plurality of different types of sensors loaded on the vehicle at this first moment is acquired. The vehicle can employ two or more different types of sensors. According to an embodiment of the present invention, the plurality of different types of sensors can include a camera and one or more of a millimeter-wave sensor and a lidar sensor. For example, the vehicle can use a camera and a millimeter-wave radar, a camera and a lidar, or a camera, a millimeter-wave radar, and a lidar. The first set of sensor data includes a plurality of sensor data sets corresponding respectively to the real-time sensor data output by the camera, the millimeter-wave sensor, and / or the lidar sensor. Of course, as those skilled in the art can understand, other quantities and other types of sensors are also within the scope of the present invention.

[0042] At step 220, the acquired first set of sensor data is fed into a general statistical model to form general statistical model format data for this first moment. That is, by feeding time information (such as a timestamp) and the plurality of sensor data sets included in the first set of sensor data into the general statistical model {t, d s1 , d s2 … d sn} to instantiate the model. For example, in the case where a vehicle uses a camera, a millimeter-wave radar, and a lidar, the general statistical model format data for this first moment is {t 第一时刻 ,d 摄像头 ,d 毫米波雷达 ,d 激光雷达}. As described above, d 摄像头 ,d 毫米波雷达 ,d 激光雷达 The output data format can be set by the vehicle manufacturer or the sensor manufacturer. Generally speaking, d 摄像头 ,d 毫米波雷达 ,d 激光雷达 have different data formats.

[0043] In step 230, map data within a threshold range around the position where the vehicle is located at the first moment is extracted from the map. In step 240, one or more of the sensor data sets in the general statistical model format data for this first moment are corrected based on the extracted map data. As described above, sensors such as cameras or lidars are vulnerable to weather factors, and the accuracy of the sensor data collected in an unsatisfactory detection environment (for example, rain, snow, haze, fog) is relatively low. Also, in practice, it is often found that ground signs or lane lines are severely worn, and the sensor data for such objects cannot well identify the original features of the objects. In addition, when in-vehicle sensors are blocked by other obstacles, the collected sensor data is also inaccurate or even incorrect. For this reason, the map data can be used as a priori information to provide support when the above sensors perform poorly or fail.

[0044] According to an embodiment of the present invention, the map can be pre-installed in the memory of the autonomous vehicle or obtained from a map provider through a network. According to an embodiment of the present invention, the map as a priori information can include an offline map such as an OSM (Open Street Map) offline map. Alternatively, in order to obtain a further more accurate centimeter-level positioning, a high-precision map (such as the high-precision maps provided by map providers such as Google, HERE, etc.) can be used. Those skilled in the art can understand that other types of maps are also within the scope of the present invention.

[0045] Through the GPS module installed on the vehicle, the GPS information of the current vehicle can be obtained. Based on this GPS information, map data within a threshold range around the vehicle is extracted from the map. The selection of this threshold range can be set by the vehicle manufacturer or the user, such as the area covered by a circle centered on the vehicle with a radius of 20 meters, 50 meters, 100 meters, etc. Of course, those skilled in the art can understand that other numerical values or shapes of the threshold range are also within the scope of the present invention.

[0046] By comparing real-time sensor data with the extracted map data, significantly incorrect or abnormal real-time sensor data can be corrected, thereby preventing large deviations in each sensor data set in the general statistical model format data due to sensor performance issues or environmental factors. As those skilled in the art can understand, to obtain accurate comparison results, generally the real-time sensor data and the map data are first converted to the same coordinate system (for example, various coordinates are uniformly converted to coordinates in the world coordinate system), and then the real-time sensor data and the corresponding map data are compared.

[0047] Generally speaking, the map data as prior information indicates static objects, such as road signs, lane lines, buildings, road edges, bridges, utility poles, elevated structures, traffic lights, or traffic signs, etc. Therefore, if the real-time sensor data indicates pedestrians or other vehicles around the vehicle (i.e., dynamic objects), there is no need to correct the real-time sensor data to effectively avoid accidents and incidents. In other words, if there is no record of a corresponding static object at the same position in the map data for the coordinates of the object indicated by the real-time sensor data, there is no need to correct the real-time sensor data. However, if there is a record of a corresponding static object at the same position in the map data for the coordinates of the object indicated by the real-time sensor data, and the data of the real-time sensor data is incomplete or inaccurate, then data representing the corresponding static object is extracted from the map data to correct the real-time sensor data.

[0048] According to one or more embodiments of the present invention, there can be various ways to correct real-time sensor data. Assuming that the general statistical model format data for this first moment (real-time) is {t 第一时刻 ,d 摄像头 ,d 毫米波雷达 ,d 激光雷达}, depending on the extracted map data, three types of sensor data sets can be corrected respectively so that d 摄像头 ,d 毫米波雷达 ,d 激光雷达 respectively contain sensor data sets describing correct static objects. Alternatively, considering the data volume, only one or both of d 摄像头 ,d 毫米波雷达 ,d 激光雷达 can be corrected, while making the data of the other two or one empty to save storage space.

[0049] For example, at a first moment (i.e., in real time), if according to the map data within the threshold range around the vehicle, there should be a traffic sign indicating a right turn on the road 10 meters ahead of the vehicle, but since most of the traffic sign is covered by sand scattered by a sand truck ahead, the sensor cannot sense the traffic sign completely, resulting in each sensor data set in the general statistical model format data for this first moment only recording data indicating the incomplete traffic sign. Thus, by comparing with the extracted map data, one or more of the sensor data sets in the general statistical model format data for this first moment can be updated so that the general statistical model format data for this first moment includes at least one type of sensor data describing the complete traffic sign.

[0050] Return Figure 2 , at step 250, the corrected general statistical model format data for the first moment is fused with the historical general statistical model format data to update the general statistical model format data for the first moment. In practice, the real-time sensor data obtained at a single moment alone cannot accurately depict the object. In particular, for continuous objects such as lane lines, the fusion of sensor data at multiple consecutive moments is required to describe the lane line.

[0051] According to an embodiment of the present invention, it is assumed that multiple types of sensors loaded on the vehicle are synchronized in time and output sensor data at the same time interval. According to different actual requirements, the time interval can be different, such as 0.1 second, 0.2 second, 0.5 second, 1 second, etc. Of course, time intervals of other lengths are also within the scope of the present invention. According to one or more embodiments of the present invention, the historical general statistical model format data can be formed at several consecutive moments before the first moment and stored in the vehicle's memory or cached for quick reading. As should be understood, the historical general statistical model format data has the same data format as the general statistical model format data for the first moment and is formed in the same manner as the general statistical model format data for the first moment at one or more moments before the first moment.

[0052] Figure 3 FIGS. 301 and 302 show schematic diagrams for the fusion of general statistical model format data according to one or more embodiments of the present invention. Briefly, FIG. 301 shows single fusion, while FIG. 302 shows iterative multiple fusions.

[0053] The schematic diagram 301 shows the fusion of the general statistical model format data for the first moment with the historical general statistical model format data including multiple general statistical model format data for multiple previous moments within a threshold time period before the first moment. That is, {t 第一时刻 ,d s1 ,d s2 …d sn} is fused with {t 第一时刻-1 ,d s1 ,d s2 …d sn}, {t 第一时刻-2 ,d s1 ,d s2 …d sn}……{t 第一时刻-tn ,d s1 ,d s2 …d sn} for the previous multiple consecutive moments to update {t 第一时刻 ,d s1 ,d s2 …d sn}. Among them, between two adjacent moments, for example, between t 第一时刻 and t 第一时刻-1 , between t 第一时刻-1 and t 第一时刻-2 , there is a predetermined time interval as described above, and the threshold time period elapsed from t 第一时刻-tn to t 第一时刻 can also be selected according to actual needs. For example, when the predetermined time interval is 0.1 second, the threshold time period can be selected as 1 second, and thus 10 (i.e., in this case, tn is 10) historical general statistical model format data within 1 second before the first moment are selected for fusion. For example, {t 第一时刻 ,d s1 ,d s2 …d sn} is fused with 10 historical general statistical model format data within the previous 1 second to update {t 第一时刻 ,d s1 ,d s2 …d sn} with the fused sensor data to obtain {t 第一时刻 ,d s1’ ,d s2’ …d sn’}, where each of the 10 historical general statistical model format data respectively corresponds to the general statistical model format data obtained at a time interval of 0.1 meter within 1 second before the first moment. Thus, it can be seen that the method of schematic diagram 301 is to {t 第一时刻 ,d s1 ,ds2 …d sn} is fused with the historical general statistical model format data once to update {t 第一时刻 ,d s1 ,d s2 …d sn}.

[0054] FIG. 302 is a schematic diagram showing the iterative fusion of multiple general statistical model format data for multiple previous moments within a threshold time period before the first moment. Continuing with the above example, assume that the threshold time period is 1 second and the predetermined time interval between two adjacent moments is 0.1 second. The general statistical model format data for the previous moment is iteratively fused with the general statistical model format data for the next moment to update the general statistical model format data for the next moment until the general statistical model format data for the first moment is updated, thereby obtaining {t 第一时刻 ,d s1’ ,d s2’ …d sn’}.

[0055] For example, first, {t 第一时刻-tn ,d s1 ,d s2 …d sn} is fused with {t 第一时刻-tn+1 ,d s1 ,d s2 …d sn} to update {t 第一时刻-tn+1 ,d s1 ,d s2 …d sn}, obtaining the updated {t 第一时刻-tn+1 ,d s1’ ,d s2’ …d sn’}. Then, {t 第一时刻-tn+1 ,d s1’ ,d s2’ …d sn’} is fused with {t 第一时刻-tn+2 ,d s1 ,d s2 …d sn} to update {t 第一时刻-tn+2 ,d s1 ,d s2 …d sn}, obtaining the updated {t 第一时刻-tn+2 ,d s1’ ,d s2’ …d sn’}. Then, {t 第一时刻-tn+2 ,d s1’ ,d s2’ …dsn’}, and {t 第一时刻-tn+3 , d s1 , d s2 … d sn} are fused to update {t 第一时刻-tn+3 , d s1 , d s2 … d sn}, obtaining the updated {t 第一时刻-tn+3 , d s1’ , d s2’ … d sn’}. And so on, until {t 第一时刻-1 , d s1’ , d s2’ … d sn’} and {t 第一时刻 , d s1 , d s2 … d sn} are fused to update {t 第一时刻 , d s1 , d s2 … d sn}, obtaining the updated {t 第一时刻 , d s1’ , d s2’ … d sn’}.

[0056] In order to make the fused data more accurate, the following mathematical methods can be adopted during the fusion process.

[0057] Figure 4 shows a flowchart of method 400 for fusing general statistical model format data for the first moment and historical general statistical model format data according to Figure 3 's embodiment. At step 410, historical general statistical model format data is obtained, and the historical general statistical model format data includes multiple general statistical model format data for multiple previous moments within a threshold time period before the first moment.

[0058] At step 420, the general statistical model format data for the first moment and the historical general statistical model format data are converted to the same coordinate system. For example, assume that the vehicle is the origin of the local coordinate system, the traveling direction of the vehicle is the x-axis of the local coordinate system, and the direction perpendicular to the traveling direction of the vehicle is the y-axis of the local coordinate system. Then, as the vehicle travels from time t 第一时刻-1 to t 第一时刻 , the vehicle travels a distance L in the traveling direction. It can be understood that the origin of the local coordinate system at t 第一时刻 has moved compared to the origin of the local coordinate system at t 第一时刻-1 by (L x , L y). Through coordinate transformation, the sensor data set in the collected historical general statistical model format data is transformed into t 第一时刻 In the local coordinate system, all sensor data sets used for fusion are in the same coordinate system. According to another embodiment of the present invention, the various coordinates used by the collected historical general statistical model format data and the general statistical model format data for the first moment can be uniformly converted into coordinates in the world coordinate system, so that all sensor data sets used for fusion are in the same coordinate system. Various coordinate conversion methods include, but are not limited to, translation and rotation of coordinates in two-dimensional space, translation and rotation of coordinates in three-dimensional space, and the like.

[0059] In step 430, according to Figure 3 Either of the two fusion methods 301 and 302 shown fuses the general statistical model format data for the first moment and the historical general statistical model format data of the sensor data set in the same coordinate system, so that the general statistical model format data for the first moment is updated to include the fused sensor data. As known to those skilled in the art, in order to obtain smooth and coherent data, the fusion process includes aggregation and denoising of the data set. For example, in the fusion method using 301, it is assumed that the threshold time period is 1 second and the predetermined time interval between two adjacent moments is 0.1 second. Set {t 第一时刻 ,d s1 ,d s2 …d sn} s1 ,d s2 …d sn Included sensor datasets and top 10 historical general statistical model format data in d s1 ,d s2 …d sn The sensor data sets included are aggregated, and then the duplicate data or abnormal data in the aggregated sensor data sets are removed or filtered to obtain the fused updated {t 第一时刻 ,d s1’ ,d s2’ …d sn’ For another example, in the fusion method of 302, the sensor data sets included in the general statistical model format data of two consecutive moments are also similarly aggregated and denoised, thereby updating the general statistical model format data for the latter moment until the general statistical model format data for the first moment is updated.

[0060] In one embodiment, a weighted average algorithm can also be used for fusion. For example, during aggregation, the historical general statistical model format data recorded closer to the first moment in time is given a higher weight, while the general statistical model format data recorded further away from the first moment in time is given a lower weight. Of course, other weighting methods can also be conceived.

[0061] Return Figure 2 , in step 260, the updated general statistical model format data for the first moment is compared with the implicit model to identify the object. As described above, the implicit model format is {pd s1 , pd s2 … pd sn , object}, which represents a set of different types of sensor data samples used to describe an object. By comparing {t 第一时刻 , d s1’ , d s2’ … d sn’} obtained in step 250 with one or more {pd s1 , pd s2 … pd sn , object}, the specific object described by {t 第一时刻 , d s1’ , d s2’ … d sn’} can be obtained. According to an embodiment of the present invention, assume that a vehicle uses three sensors, and {t 第一时刻 , d s1’ , d s2’ , d s3’} is compared with {pd s1 , pd s2 , pd s3 , object 1}. In this example, d s1’ is compared with pd s1 , d s2’ is compared with pd s2 , and d s3’ is compared with pd s3 respectively to determine whether {t 第一时刻 , d s1’ , d s2’ , d s3’} describes object 1. In specific practice, it is very likely that not all three comparison results are true among the three comparison results for the three sensors. In this regard, the vehicle manufacturer can pre-define the determination criteria. For example, if two of the three comparison results of the three sensor data sets are true, it is regarded as the overall comparison result being true, or all three comparison results of the three sensor data sets must be true to be regarded as the overall comparison result being true.

[0062] Alternatively, according to preset settings, the vehicle can automatically select a determination criterion based on the current climate environment, road environment, or the identified object. For example, as described above, different types of sensors have different advantages and disadvantages. Depending on the different adaptabilities of the sensors to environmental conditions, in an environment with poor visibility such as haze or heavy fog, the confidence level of the sensor data set output by the millimeter-wave radar can be specified as relatively high, while in an environment with better conditions, the confidence levels of the sensor data sets output by lidar and cameras can be specified as relatively high. In addition, depending on the ways in which different types of sensors obtain data, different confidence levels of sensor data sets can be set for different types of objects. For example, for some objects with three-dimensional characteristics such as buildings, the confidence level of the sensor data set output by the camera can be set lower than the confidence levels of the sensor data sets output by the laser sensor and the millimeter-wave radar sensor. However, for some planar objects such as lane lines and ground traffic signs, the confidence level of the sensor data set output by the camera can be set lower than the confidence levels of the sensor data sets output by the laser sensor and the millimeter-wave radar sensor. In this way, generally speaking, the overall determination result can be calculated by the following equation:

[0063] Overall = Confidence s1 *S1 + Confidence s2 *S2 + …… + Confidence sn *Sn.

[0064] Wherein, Confidence s1 + Confidence s2 + …… Confidence sn = 1, S1 is the comparison result between d s1’ and pd s1 , S2 is the comparison result between d s2’ and pd s2 , Sn is the comparison result between d sn’ and pd sn . Among them, S1, S2 …… Sn are 1 or 0. For example, S1 = 1 means that after comparing d s1’ with pd s1 , it is obtained that d s1’ identifies the object described by pd s1 , while S1 = 0 means that after comparing d s1’ with pd s1 , it is obtained that d s1’ does not identify the object described by pd s1 . The same is true for S2…Sn. Thus, the vehicle manufacturer or user can set that if the overall > a predetermined value (for example, 50%), it can be determined that {t 第一时刻 ,d s1’ ,d s2’ …d sn’} identifies {pd s1, pd s2 …pd sn , the object described by {the object}. Of course, various other different determination criteria can also be conceived.

[0065] In step 270, the identified object is combined with the extracted map data to form a real-time road model. As described above, depending on environmental factors, etc., the real-time sensor data collected by the sensor cannot comprehensively reflect the surrounding road conditions. Thus, the object identified from the updated general statistical model format data for the first moment can be stitched with the map data extracted in step 230 to form a real-time road model. In some embodiments, in step 260, the updated general statistical model format data for the first moment can be compared only with the implicit model describing dynamic objects (e.g., pedestrians, other vehicles, etc.) so that the real-time sensor data is only used to identify dynamic objects. Thus, the real-time dynamic objects identified by the sensor are combined with the static objects in the map data (e.g., road signs, lane lines, buildings, road edges, bridges, utility poles, elevated structures, traffic lights, or traffic signs, etc.) to form a real-time road model. Thereby, the computational burden on the in-vehicle processor is reduced.

[0066] Also, in step 260, after comparing the updated general statistical model format data for the first moment with the implicit model, it is also possible that the updated general statistical model format data for the first moment fails to successfully identify an object. In this case, the extracted map data can also be used as the real-time road model.

[0067] Thus, by using the method of the present invention, compared with obtaining sensor data from various different types of sensors separately and processing them separately, by putting different types of sensor data sets into a unified data model, the sensor data sets can be processed more quickly. At the same time, taking the map data as prior information also enables the combination of the real-time data of the sensor and the existing information of the map data, and a real-time road model can be constructed more accurately and quickly.

[0068] Figure 5 is a block diagram of an apparatus 500 for constructing a real-time road model according to an embodiment of the present invention. All functional blocks of the apparatus 500 (including each unit in the apparatus 500) can be implemented by hardware, software, or a combination of hardware and software. Those skilled in the art should understand that Figure 5 the functional blocks described can be combined into a single functional block or divided into multiple sub-functional blocks.

[0069] The apparatus 500 may include a sensor data acquisition module 510 configured to acquire a first set of sensor data output by different types of sensors mounted on a vehicle at a first moment. The apparatus 500 may further include a data feeding module 520 configured to feed the first set of sensor data into a general statistical model to form general statistical model format data for the first moment. The apparatus 500 may further include a map data extraction module 530 configured to extract map data within a threshold range around the location where the vehicle is located at the first moment from a map. The apparatus 500 may further include a data correction module 540 configured to correct one or more of a plurality of sensor data sets in the general statistical model format data for the first moment based on the map data. The apparatus 500 may further include a data fusion module 550 configured to fuse the general statistical model format data for the first moment with historical general statistical model format data to update the general statistical model format data for the first moment. The apparatus 500 may further include an object identification module 560 configured to compare the updated general statistical model format data for the first moment with an implicit model to identify an object. The apparatus 500 may further include a real-time road model formation module 570 configured to combine the identified object with the map data to form the real-time road model.

[0070] Figure 6 A block diagram of an exemplary computing device according to an embodiment of the present invention is shown, which is an example of a hardware device applicable to various aspects of the present invention.

[0071] Reference Figure 6 , a computing device 600 will now be described, which is an example of a hardware device applicable to various aspects of the present invention. The computing device 600 may be any machine configurable to perform processing and / or computing, and may be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal digital assistant, a smart phone, an in-vehicle computer, or any combination thereof. The foregoing various methods / devices / servers / client devices may be implemented in whole or at least in part by the computing device 600 or a similar device or system.

[0072] The computing device 600 may include components that can be connected or communicate via one or more interfaces and a bus 602. For example, the computing device 600 may include a bus 602, one or more processors 604, one or more input devices 606, and one or more output devices 608. The one or more processors 604 can be any type of processor and may include, but are not limited to, one or more general-purpose processors and / or one or more dedicated processors (e.g., specialized processing chips). The input device 606 can be any type of device capable of inputting information into the computing device and may include, but are not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote controller. The output device 608 can be any type of device capable of presenting information and may include, but are not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The computing device 600 may also include a non-transitory storage device 610 or be connected to the non-transitory storage device. The non-transitory storage device can be any storage device that is non-transitory and capable of implementing data storage, and the non-transitory storage device may include, but are not limited to, a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a magnetic tape, or any other magnetic medium, an optical disk, or any other optical medium, a ROM (read-only memory), a RAM (random access memory), a cache memory, and / or any storage chip or cartridge, and / or any other medium from which a computer can read data, instructions, and / or code. The non-transitory storage device 610 can be separated from the interface. The non-transitory storage device 610 may have data / instructions / code for implementing the above methods and steps. The computing device 600 may also include a communication device 612. The communication device 612 can be any type of device or system capable of implementing communication with internal devices and / or communication with a network and may include, but are not limited to, a modem, a network card, an infrared communication device, a wireless communication device, and / or a chipset, such as a Bluetooth device, an IEEE 1302.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or a similar device.

[0073] When the computing device 600 is used as an in-vehicle device, it can also be connected to external devices (e.g., a GPS receiver, sensors for sensing different environmental data (such as an acceleration sensor, a wheel speed sensor, a gyroscope, etc.)). In this way, the computing device 600 can receive, for example, positioning data and sensor data indicating the form condition of the vehicle. When the computing device 600 is used as an in-vehicle device, it can also be connected to other devices for controlling the driving and operation of the vehicle (e.g., an engine system, a windshield wiper, an anti-lock braking system, etc.).

[0074] In addition, the non-transitory storage device 610 may have map information and software components, so that the processor 604 can implement route guidance processing. In addition, the output device 606 may include a display for displaying maps, displaying positioning markers of the vehicle, and displaying images indicating the driving status of the vehicle. The output device 606 may also include a speaker or a headphone jack for audio guidance.

[0075] The bus 602 may include, but is not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus. In particular, for in-vehicle devices, the bus 602 may also include a Controller Area Network (CAN) bus or other architectures designed for automotive applications.

[0076] The computing device 600 may also include a working memory 614, which can be any type of working memory capable of storing instructions and / or data beneficial to the operation of the processor 604 and may include, but is not limited to, random access memory and / or read-only storage devices.

[0077] The software components may be located in the working memory 614, and these software components include, but are not limited to, an operating system 616, one or more application programs 618, drivers, and / or other data and code. The instructions for implementing the above methods and steps may be included in the one or more application programs 618, and the modules / units / components of the foregoing various devices / servers / client devices may be implemented by the processor 604 reading and executing the instructions of the one or more application programs 618.

[0078] It should also be recognized that changes can be made according to specific requirements. For example, custom hardware may also be used, and / or specific components may be implemented in hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. In addition, connections to other computing devices, such as network input / output devices, etc., may be adopted. For example, part or all of the disclosed methods and devices may be implemented by programming hardware (such as programmable logic circuits including Field Programmable Gate Arrays (FPGAs) and / or Programmable Logic Arrays (PLAs)) with assembly language or a hardware programming language (such as VERILOG, VHDL, C++) using the logic and algorithms according to the present invention.

[0079] Although aspects of the present invention have been described so far with reference to the accompanying drawings, the above methods, systems, and devices are merely examples, and the scope of the present invention is not limited to these aspects, but is defined only by the appended claims and their equivalents. Various components may be omitted or may also be replaced by equivalent components. Additionally, the steps may be implemented in an order different from the order described in the present invention. Furthermore, the various components may be combined in various ways. It is also important that, as technology develops, many of the components described may be replaced by equivalent components that emerge later.

Claims

1. A method for constructing a real-time road model, comprising: At a first moment, obtain a first set of sensor data output by a plurality of different types of sensors mounted on a vehicle; Feed the first set of sensor data into a general statistical model to form general statistical model format data for the first moment, the general statistical model format data including a plurality of sensor data sets, wherein each sensor data set in the plurality of sensor data sets is output by one of the different types of sensors; Extract map data within a threshold range around the location where the vehicle is located at the first moment from a map; Correct one or more of the plurality of sensor data sets in the general statistical model format data for the first moment based on the map data; Fuse the general statistical model format data for the first moment with historical general statistical model format data to update the general statistical model format data for the first moment; Compare the updated general statistical model format data for the first moment with an implicit model to identify an object, the implicit model including a plurality of sensor data sample sets for describing predefined objects, wherein each sensor data sample set in the plurality of sensor data sample sets includes a pre-acquired sensor data sample set for describing the predefined object output by one of the different types of sensors; And Combine the identified object with the map data to form the real-time road model.

2. The method according to claim 1, wherein Correcting one or more of the plurality of sensor data sets in the general statistical model format data for the first moment based on the map data further includes: Convert the plurality of sensor data sets in the general statistical model format data for the first moment and the map data into the same coordinate system; If there is no record of a corresponding object at the same position in the map data for the object indicated by the plurality of sensor data sets, there is no need to correct one or more of the plurality of sensor data sets in the general statistical model format data for the first moment; If there is a record of a corresponding object at the same position in the map data for the object indicated by the plurality of sensor data sets, and the data of one or more of the plurality of sensor data sets is incomplete or inaccurate, extract data representing the corresponding object from the map data to correct one or more of the plurality of sensor data sets.

3. The method according to claim 1, wherein Fusing the general statistical model format data for the first moment with historical general statistical model format data includes: fusing the general statistical model format data for the first moment with historical general statistical model format data including a plurality of general statistical model format data for a plurality of previous moments within a threshold time period before the first moment.

4. The method according to claim 1, wherein Fusing the general statistical model format data for the first moment with historical general statistical model format data includes: Iteratively perform, for a plurality of general statistical model format data for a plurality of previous moments within a threshold time period before the first moment: fuse the general statistical model format data for each moment with the general statistical model format data for the subsequent moment after a predetermined time interval to update the general statistical model format data for the subsequent moment, until the general statistical model format data for the first moment is updated.

5. The method according to claim 1, wherein The fusing further includes: causing both the historical general statistical model format data and the general statistical model format data for the first moment to be represented in the same coordinate system by any one of the following: converting the historical general statistical model format data into the local coordinate system with the position of the vehicle at the first moment as the origin of the local coordinate system, or uniformly converting various coordinates used in the historical general statistical model format data and the general statistical model format data for the first moment into coordinates in the world coordinate system.

6. The method according to claim 5, wherein The fusing further includes: correspondingly clustering sensor data sets output by the same sensor in the historical general statistical model format data and the general statistical model format data for the first moment, and removing duplicate data in each clustered sensor data set.

7. The method according to claim 1, wherein The object includes at least one of the following: road signs, lane lines, buildings, pedestrians, another vehicle, road edges, bridges, utility poles, elevated structures, or traffic signs.

8. The method according to claim 1, wherein The plurality of different types of sensors includes vision sensors and radar ranging sensors, the vision sensors include cameras, and the radar includes one or more of lidar, ultrasonic radar, and millimeter-wave radar.

9. The method according to claim 1, wherein The map includes a high-precision map, and comparing the updated general statistical model format data for the first moment with the implicit model to identify an object further includes: only comparing the updated general statistical model format data for the first moment with the implicit model describing dynamic objects to only identify dynamic objects, the dynamic objects including pedestrians and / or another vehicle; Combining the identified object with the map data to form the real-time road model further includes: combining the identified dynamic object with the map data to form the real-time road model.

10. An apparatus for constructing a real-time road model, comprising: A sensor data acquisition module configured to acquire a first set of sensor data output by different types of sensors mounted on a vehicle at a first moment; A data feeding module configured to feed the first set of sensor data into a general statistical model to form general statistical model format data for the first moment, the general statistical model format data including a plurality of sensor data sets, where each sensor data set in the plurality of sensor data sets is output by one of the different types of sensors; A map data extraction module configured to extract map data within a threshold range around the position where the vehicle is located at the first moment from a map; A data correction module, which is configured to correct one or more of a plurality of sensor data sets in the general statistical model format data for the first moment based on the map data; A data fusion module, which is configured to fuse the general statistical model format data for the first moment with the historical general statistical model format data to update the general statistical model format data for the first moment; An object identification module, which is configured to compare the updated general statistical model format data for the first moment with an implicit model to identify objects, the implicit model including a plurality of sensor data sample sets for describing predefined objects, wherein each sensor data sample set in the plurality of sensor data sample sets includes a pre-acquired sensor data sample set that describes the predefined object by one of the different types of sensors; And A real-time road model formation module, which is configured to combine the identified objects with the map data to form the real-time road model.

11. The apparatus according to claim 10, wherein The data correction module is further configured to: Convert a plurality of sensor data sets in the general statistical model format data for the first moment and the map data into the same coordinate system; If there is no record of a corresponding object at the same position in the map data for the coordinates of the objects indicated by the plurality of sensor data sets, there is no need to correct one or more of the plurality of sensor data sets in the general statistical model format data for the first moment; If there is a record of a corresponding object at the same position in the map data for the coordinates of the objects indicated by the plurality of sensor data sets, and the data of one or more of the plurality of sensor data sets is incomplete or inaccurate, extract the data representing the corresponding object from the map data to correct one or more of the plurality of sensor data sets.

12. The apparatus according to claim 10, wherein, The map includes a high-precision map, wherein the object identification module is further configured to: only compare the updated general statistical model format data for the first moment with the implicit model describing dynamic objects to only identify dynamic objects, the dynamic objects including pedestrians and / or another vehicle; Wherein the real-time road model formation module is further configured to: combine the identified dynamic objects with the map data to form the real-time road model.

13. The apparatus according to claim 10, wherein, The plurality of different types of sensors include vision sensors and radar ranging sensors, the vision sensors include cameras, and the radar includes one or more of lidar, ultrasonic radar, and millimeter-wave radar.

14. A vehicle, comprising: A plurality of different types of sensors; And The device according to any one of claims 10-13.

15. The vehicle according to claim 14, wherein, The plurality of different types of sensors include vision sensors and radar ranging sensors, the vision sensors include cameras, and the radar includes one or more of lidar, ultrasonic radar, and millimeter-wave radar.

Images